BUG: v20, Windows 7 x64. Fails to save edited file

[Margaret]

I'm trying to progressively comb out unwanted fields from a Very Large (>110GB) XML file.

The first edit pass went smoothly, deleting some 150M substrings.  But when I tried to save the result so that the journal file could be closed, I first got a "Saving" status message, but after a very few seconds it changed to a "(location n.m%) Searching..." message, which suggests either that someone reused a progress message but forgot to edit it to say "Saving" rather than "Searching", or a buffer overflowed and made a mess.

It took about a minute to advance the counter by 3%, so it was about 30 minutes later that it reached 92.3% and failed with a write-error popup.  I wasn't keeping track as closely the first time this happened (last week), but my memory says it was about at that same almost-done point when it failed with the same write error.

The disks are good and have plenty free space, so I suspect that the funny progress message was indeed A Clue.  The length of time involved also seems to be a clue, since a scratch file had to have been written to disc during the edit since the file was 4X too large to fit in memory, so "saving" the file would amount to renaming the original to foo.old, the scratch file to foo, closing that journal file and starting a new one, and deleting foo.old.  Which wouldn't have even begun to take 30 minutes.

[Clark]

Reproduced the incorrect message issue. It always displays the wrong message (trivial bug..easy to fix). We will fix that.

Don't know why you're having problems saving though. Spill File I/O can take a while especially if this isn't on an SSD.

[Clark]

You can reduce spill file I/O but increasing the cache size (Tools>Options>Virtual Memory>Buffer cache size).

If the changes you are making are pretty sparse (significant number of bytes between changes -- like 16k), this will help a lot.

If you are not making many changes, or making a ton of changes, it doesn't help much to increase the buffer cache size.

[Margaret]

Reproduced the incorrect message issue. It always displays the wrong message (trivial bug..easy to fix). We will fix that.

Good that it was just an unedited message.

Don't know why you're having problems saving though. Spill File I/O can take a while especially if this isn't on an SSD.

I don't know why it's failing either, but I'd surely like to!

The time it takes is hardly ever a problem (does it take so long because you're only journaling the edits until the save?).  The failure right at the end  of the save and its uninformative accompanying message is the only thing that bothers me.  I'm not having any write failures elsewhere.  There've been one or two momentary hiccups trying to read from or write to the server, but nothing serious nor limited to any one source.  This problem seems to be slick (or maybe v20) -specific.

[Dan]

What happens if you type "save +o" on the SlickEdit command line?

[Margaret]

What happens if you type "save +o" on the SlickEdit command line?

I don't know.  What conditions do you have in mind for the experiment?

[Dan]

Just open the large file.

[Clark]

Using +o isn't supposed to make a difference unless you've changed the default backup options. When using the default backup options, backup is turned off for files this large. Can't remember the limit but it's definitely less than 1 gigabyte.

I tested a bunch of large file saves and didn't reproduce any problems. I used all default settings.

[Clark]

Is this file on a remote server?

[Margaret]

Is this file on a remote server?

Yes (well, not very remote -- only about 7 feet away  ) but I was trying to edit a local copy.

I think I need to give up on W7 and evaluate a version that runs under XP.  I can't even get a save-as of an unedited file to work -- it consistently hangs.  It also doesn't clean up its scratch files.  During my struggle to edit that big file, it created 2 $slk files that choked my ~330GB C: partition and didn't clean them up after.  And it discards without notice my choice of spill file path.  It probably does other things, too, that I just haven't discovered yet.

[Margaret]

Just open the large file.

The open was okay, but when it failed the save (at about 12% in, possibly because it had consumed the entire 210GB+ free space on C: for its $slk files), it trashed my file.

[Clark]

Is this file on a remote server?

Yes (well, not very remote -- only about 7 feet away  ) but I was trying to edit a local copy.

I think I need to give up on W7 and evaluate a version that runs under XP.  I can't even get a save-as of an unedited file to work -- it consistently hangs.  It also doesn't clean up its scratch files.  During my struggle to edit that big file, it created 2 $slk files that choked my ~330GB C: partition and didn't clean them up after.  And it discards without notice my choice of spill file path.  It probably does other things, too, that I just haven't discovered yet.

My question was to find out if this was a local or network file. Networks tend to be more likely to have I/O failures. The only way to know if that's the problem is to try editing a huge local file. If that always works, then the problem is your network.

[Margaret]

My question was to find out if this was a local or network file.

I know - that was just  me trying to make a small, sad joke.  The failures were all with a local copy, resident on a directly-attached disk.   Since it took several days to produce the file by filtering an even larger one, I didn't want to risk having anything happen to it.  So I made a local copy and did all my experimentation on it, leaving the original in peace on the server.

[Clark]

Typically save related bugs are reproducible but sometimes you need to recreate the edits that were done. If you can figure out something, reproducible, we will look into it right away. Save bugs get the highest priority.

With a local disk, I wouldn't expect there to be I/O problems that could blamed on the OS. I've seen more issues on spinning disks than SSDs. Usually SSDs either work or pretty much just die. Spinning disks can get bad sectors and go bad slowly.

[Margaret]

It's possible that file ops might want to be revisited.

Still trying to figure out where these problems are coming from, I hypothesised that perhaps the scratch disc I'd been using might have undiagnosed problems.  So I copied the file into free space on another disc.

Then I loaded the file from that new location into the editor and, without making any changes, save-as'd it to free space on still a third drive.  The last time I tried a simple save-as, it unexpectedly crashed.  But I wasn't watching and so didn't know why.

This time I watched, via sysmon, as v20 consumed all 32G of system memory and, over more than an hour's time, tried to create a $slk file.  Unfortunately, since that file was much bigger than the original, it consumed all the free space on the designated scratch-file disc and eventually crashed before completion. 

There was no reason for it to try to create that file -- it had to have known that I hadn't done any edits!  It was just a straight copy, with v20 in the role of the shell.  Which is one of the reasons I think re-visitation might be in order.

And the size of that file surprised me -- I'd supposed the $slk files were scratch files, but evidently they're intermediate files of some kind, with a lot of Knuthian sugar added.   Which re-surprised me, since evidently the sugar is added everywhere rather than sparsely.   Is the size multiplier known?  So far I've  not had enough scratch space available to allow that file to be fully created -- the editor has crashed every time!

There's also a problem where v20 (maybe it was fixed for v21) messes up the screen and appears to go unresponsive as far as W7 is concerned.  I'm not sure what prompts that behavior, but it's not helpful.

(This is reproducible  )