Author Topic: Saving a large file is *really* slow  (Read 5479 times)

jporkkahtc

  • Senior Community Member
  • Posts: 2620
  • Hero Points: 210
  • Text
Saving a large file is *really* slow
« on: September 20, 2017, 06:19:19 PM »
I created a plain text file,
size=218,264,095 bytes.
#lines=7,385,617

Saving in Beta3 takes 40 seconds!
Saving the same file in V21: Less than 1 second.

Happens even if I turn off backup history.
Happens with UTF-8 and DBCS encoding. (The data is plain ascii text).
Happens every time I save the file, or SaveAs to a different name.

From an external program I can see the size of the file being saved slowly growing from 0 to 218mb.

I don't see any significant difference in options between v21 and v22 - same FileOtions and  ApplicationOptions.



jporkkahtc

  • Senior Community Member
  • Posts: 2620
  • Hero Points: 210
  • Text
Re: Saving a large file is *really* slow
« Reply #1 on: September 20, 2017, 06:30:40 PM »
Looking with PROCMON, saving the file (with dbcs encoding).
Slick writes about 32kb at a time ... but not 32,768 bytes -- seemingly random lengths just under 32,768 bytes each time.

It oughta write 32,768 bytes each time, and keep it aligned.

Also, at the start of the save it appears to truncate the file to 0 bytes - OK, but before writing that much it would help to set the file length to the desired size for best disk space allocation.

V21 writes the same file 8,192 bytes at a time - aligned with DBCS, but offset with utf8 by 3 bytes - but either way it is fast.

This is writing to an SSD.

Dan

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 2896
  • Hero Points: 153
Re: Saving a large file is *really* slow
« Reply #2 on: September 20, 2017, 06:52:45 PM »
Just to be REALLY sure, while the file is open run "save +o" on the SlickEdit command line.

jporkkahtc

  • Senior Community Member
  • Posts: 2620
  • Hero Points: 210
  • Text
Re: Saving a large file is *really* slow
« Reply #3 on: September 20, 2017, 07:01:12 PM »
Same result.

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 6823
  • Hero Points: 526
Re: Saving a large file is *really* slow
« Reply #4 on: September 20, 2017, 07:29:43 PM »
I've read through this quickly. Have you tried a default config?

I just tried saving foo3 and on my machine with a clean config, the save is a fraction of a second.

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 6823
  • Hero Points: 526
Re: Saving a large file is *really* slow
« Reply #5 on: September 20, 2017, 07:47:15 PM »
Note: SlickEdit was on plain text mode for foo3

jporkkahtc

  • Senior Community Member
  • Posts: 2620
  • Hero Points: 210
  • Text
Re: Saving a large file is *really* slow
« Reply #6 on: September 20, 2017, 07:59:51 PM »
OK, clean configuration doesn't repro --- in this case Slick is doing 8KB IOs.

I tried importing all options except appearance, keyboard, debugging, languages.
It still did not repro.

I then imported all options, and now it does repro.

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 6823
  • Hero Points: 526
Re: Saving a large file is *really* slow
« Reply #7 on: September 20, 2017, 08:04:06 PM »
My best guess is a save option

jporkkahtc

  • Senior Community Member
  • Posts: 2620
  • Hero Points: 210
  • Text
Re: Saving a large file is *really* slow
« Reply #8 on: September 20, 2017, 08:08:11 PM »
Which?
Short of importing all settings, it doesn't repro.

I tried:
Repro: Importing all
Not repro: Importing all except appearance, keyboard, debugging, languages
Not repro: Importing only appearance, keyboard, debugging, languages



Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 6823
  • Hero Points: 526
Re: Saving a large file is *really* slow
« Reply #9 on: September 20, 2017, 08:09:54 PM »
Are you changing the line endings on save?  If so, it will have to write a spill file which takes a long time. You can increase the buffer cache size (default is 2 megabytes) to speed this up.

jporkkahtc

  • Senior Community Member
  • Posts: 2620
  • Hero Points: 210
  • Text
Re: Saving a large file is *really* slow
« Reply #10 on: September 20, 2017, 08:13:37 PM »
Nope.

PROCMON shows the only IO is Slick writing just slightly less than 32KB at a time.

From a clean config it actually alternates reading the original file and writing at 8kb at a time -- much faster somehow.

In my config
Buffer cache size (MB)"=8000 (default=2)
Load partially when files are larger than=10,001 (default is 8,000)
Wrap line length=8,000 (default is 4,000)
Maximum size to backup=75,000(default 20,000)

THese are the only difference from default config which looked relevant to me, yet setting these isn't enough to make it repro.

jporkkahtc

  • Senior Community Member
  • Posts: 2620
  • Hero Points: 210
  • Text
Re: Saving a large file is *really* slow
« Reply #11 on: September 20, 2017, 08:17:36 PM »
I uploaded my options export to support/porkka_slow_save.

Dan

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 2896
  • Hero Points: 153
Re: Saving a large file is *really* slow
« Reply #12 on: September 20, 2017, 08:22:13 PM »
I haven't looked at the options you exported, but strip trailing spaces would be a killer on a large file too.

jporkkahtc

  • Senior Community Member
  • Posts: 2620
  • Hero Points: 210
  • Text
Re: Saving a large file is *really* slow
« Reply #13 on: September 20, 2017, 08:40:49 PM »
nope, i don't have that set.

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 6823
  • Hero Points: 526
Re: Saving a large file is *really* slow
« Reply #14 on: September 20, 2017, 09:01:20 PM »
You are translating the line endings to Unix EOL on save (see language specific save options). The save code could be optimized but hasn't been yet (i.e only change the data when the eol characters actually change). I don't know if this is easy, hard, or too hard to be worth doing.