Max editable file size?

[Margaret]

The blurb on v18 says

Reading and saving files greater than 2GB is now supported. In theory the maximum limit is 2TB, but available system memory and hard drive speed are now the limiting factors. For substantially better performance, use an SSD.

Does that mean the editor reads it all into memory rather than paging the file to disc?

[Margaret]

Nobody knows?

[jporkkahtc]

Usually SlickEdit will not read the entire file into memory if it is very large.
You can see this when you open a large file, jump to the end and see that Slick doesn't know the line number.

Look in Tools->Options->Virtual Memory, and Tools->Options->FileOptions.

Some operations will force it to at least scan the whole file, even if it doesn't keep the whole thing in memory .... such as doing a search over the whole file.

[Margaret]

Thanks, but I'm really trying to get at whether that's still true with the new larger size limit.  According to the blurb I quoted, it doesn't seem to be ("In theory the maximum limit is 2TB, but available system memory and hard drive speed are now the limiting factors"), but it sounds silly that they'd discard the paging routines, so I can't make sense of it.

[Matthew]

We do not read the entire file into memory to support large files. The theoretical 2TB limit is based upon the maximum number of blocks we can keep track of. We start shutting off (or delaying) features when large files are opened. For example, if you load up a 5GB file, you won't see any line numbers for a while if you jump down to the end of the file, as the line counting has been deferred to speed up the loading. Where large files start to hit their limit is when it comes time to save an edit (which is disk intensive obviously), or when doing a search/replace that touches a large number of lines (which is memory intensive).  In practice, we have done most of our large file support testing using files in the 4GB to 8GB range.

[Margaret]

Thanks, that helps!  Now, once I discover whether I'll be able to continue to use XP, I can take a decision on whether to splash out for the latest version of Slick.

[guth]

I'm not sure it works correctly. Maybe I'm missing some settings, but for me it seems the file is loaded entirely regardless of the setting of partial load.

I just tested on Linux to do

dd if=/dev/zero of=temp.txt

for a while and got a 900 MB file with all bytes equal to zero. Trying to load this into SlickEdit gives a sigsev after heavy loading from disk for a while.

Why I'm trying this is that I had a 3 GB text file a month or two ago, but failed to load it because the computer started to trash before the complete file was loaded. There was more than 3 GB of available RAM, but vs process consumed much more than that, but still not loading the full file.

[Margaret]

That's definitely helpful to hear, since file-loading/paging shouldn't differ between operating systems.

[Clark]

SlickEdit has it's own internal virtual memory system. For example, my Windows machine has 16gb of memory. I've editing files larger than 100gb and used search/replace and save. Openning the file takes <1 second with the default options.

You do need to watch out for running out of disk space. This only occurs if you make a lot of edits to the file (like a search and replace which modifies almost the entire file). However, if you use a load option which forces the editor to load the entire file (like expand tabs ON), then you would have to worry about running out of disk space when opening the file. The default options are excellent though for editing extremely large files.

I highly recommend storing your large files on an SSD. If you are editing a large amount of data, you should also put your spill file on the SSD as well. This will make a huge difference in performance. In most cases, I found that increasing the editors buffer cache size does not help performance. I have a 500GB SSD and I love it!!! My spill file is not on my SSD but I'm not typically doing search and replace on huge files.

[guth]

Right. I tried to start vs with a new configuration directory, and now I can open very large files in an instance. However, as soon as I select GNU Emacs as emulation, these large files are now loaded into memory instead. Which setting is set by GNU Emacs emulation, that I want to unset?

[Clark]

The softwrap option causes the file to be preloaded. You can turn this off for just plain text (Tools>Options>Languages>Miscellaneous Languages>Fundamental>Word Wrap>Enable soft wrap).

The problem with softwrap is that it has performance problems with handling very long lines. When softwrap is on, SlickEdit needs to break the lines up into multiple lines (without lines breaks - default is 4k) so that you get good performance when the screen is displayed.

I should probably add an option for this and turn off softwrap by default for huge files (especially since performance currently depends on your emulation)

[Clark]

I have added an option so that softwrap is turned off by default for large files (files over 100 megabytes). This will help performance a ton for emulations like emacs which have softwrap on by default.

[jporkkahtc]

WRT options: The trick for the user is always how to find the right option!
Currently, when I try to load a really big file in Slick, slick takes a *long* time and scans the entire file (as verified by procmon.exe).

Why?
I have no idea, and Slick won't tell me why.
I checked some of the obvious (to me) options.
Presumably I'm missing something.

What I'm really complaining about here is discoverability and troubleshooting, not specifically dealing with really big files.

Seems to me that the most common complaint historically has been "Doing X in slick is too slow, how can I fix it?".
I would be great if Slick could do something like the Notifications log that shows a recent history of features used, but in this case just for slow operations.
Or, when something is being slow, a pop up that says
   Sorry this is taking a while.
   Slick is performing: ?
   [ Cancel ]  [ Change the relavent option ] [ Help ] [ Report Issue ]

Slick really should *always* open a popup to offer to cancel long running operations instead of Windows displaying Slick as non-responsive.
Then people don't need to know the magic macro cancel key combination.

[jporkkahtc]

When I try to load a really big file, slick is performing this kind of non-optimal I/O.
It scans the entire file, reading 32kb at a time, but offset by 2 bytes.
So instead of reading from 0, 32k, 64k, ....
It is reading from position 0 (len=4k), 2 (len=32k), 32k+2(len=32k), 64k + 2(len=32k), 96k+2(len=32k), ...

This pattern repeats as it scans the entire file.
At about the 2GB location in the file, Slick then starts writing to $SLK.2.

Slick is clearly scanning the file, and appears to copy its contents to %TEMP%\$slk.2
It seems that Slick at the very least should be performing aligned reads.

When I close this buffer (having not modified it), Slick is performing *TONS* of I/O to $slk.2.
It is scanning through this file and writing 8 bytes at a time.

[Clark]

With v18, we've made attempts to optimize the performance automatically but there are still gotchas. Ideally, the user shouldn't have to know but it's not there yet.

Editing files with incredible performance is a VERY difficult problem to solve. That's why there's no other editor I know which overall does a better job. There's still more work to be done.

Start with a default config (vs -sc tempconfig). Here are some performance gotchas I know of: proportional fonts, load entire file, count lines, Unicode files (even Utf-8), expand tabs, soft wrap.

There will never be a way to significantly optimize options like expand tabs, converting a file to Utf-8 for a Unicode file, or load entire file. With some difficult design changes it's possible for us to make softwrap, proportional fonts, and openning Utf-8 files instant. The count lines options needs a maximum file size.

Once you figure out what option you are using which is destroying your huge file performance, I may be able to give you a simple work around.