Deletion of regex selected lines takes WAY too much time and cripples Win7

[davew]

Initially, I had a large file (161447 lines) in which I wanted to remove certain lines with a regex as follows:
:v/^.{16}0x[0-9a-f]{16}\s+\S+\s+.*(?:string1|string2)/d

After waiting 5 minutes, I did Ctrl-Alt-Shift-F2. During this time, I couldn't even click on icons on the Task Bar and the clock on the task bar was not updating.

- I tried the same thing in the gvim editor and it succeeded instantaneously.
- I tried reducing the number of lines in VS to 1000 and it still took 5 seconds.
- I tried just using the regex (^.{16}0x[0-9a-f]{16}\s+\S+\s+.*(?:string1|string2)) to highlight lines in the full file (161447 lines) and it took 1 second, so it's not the regex that's going slow.
- If I removed the 'd' at the end of the regex but still used :v (i.e. :v/^.{16}0x[0-9a-f]{16}\s+\S+\s+.*(?:string1|string2)/), it popped up a window after 18 seconds (still slow, but at least not excruciatingly).

Again gvim will run the whole command with the 'd' in < 1 second. 93275 lines are deleted by gvim.

[Clark]

Sounds like something in the vim emulation needs some optimization.

We will take a look.

As a work around, you could bypass vim emulation and use a regex replace in the GUI which matches the entire line (need \R at the end) and replace with nothing. We'll see what we can do to speed up this vim emulation operation.

[davew]

Thanks for the workaround suggestion Clark. I ran into an apparent bug when trying it that I thought I would mention.

If you have a file with the following lines:
line 1
line 2
line 3
line 4

And then execute the following regex:
%s/^((?!test.)*$\R//

It should remove all the lines since no line has the string "test" in it. However, line 2 is not removed.

I tried this in gvim using its syntax for negative lookahead and it worked:
%s/^\(\(test\)@!.\)*$\n//

Interestingly, if I add 'g' at the end of the regex I use in SlickEdit, it works.
%s/^((?!test.)*$\R//g

Also, when running my actual regex on a large file, there were many more lines other than line 2 that were not deleted even though they were highlighted when search with the regex. Again, if I added the 'g' at the end, it worked.

[Clark]

Sounds like a different VIM emulation issue. We will look into this too.

By using the GUI, I meant (Search>Replace)

[davew]

Yes, using the GUI works with ^((?!.{16}0x[0-9a-f]{16}\s+\S+\s+.*(?:string1|string2)).)*$\R.
This implies that the GUI effectively adds the 'g' at the end.
However, there still seems to be a bug in the command line %s/xxx// functionality.

[Clark]

Sounds like a bug. VIM emulation has a ton of code which is only used in VIM emulation. It tries to use built-ins but there's still a ton of VIM only wrapper code.

The GUI can replace one by one or do a replace all.

There's also a non-VIM emulation "c" command and it works fine (c/^.*\R//L).  I verified that :%s has an issue (misses second line).

[Clark]

Thanks for the workaround suggestion Clark. I ran into an apparent bug when trying it that I thought I would mention.

If you have a file with the following lines:
line 1
line 2
line 3
line 4

And then execute the following regex:
%s/^((?!test.)*$\R//

It should remove all the lines since no line has the string "test" in it. However, line 2 is not removed.

I tried this in gvim using its syntax for negative lookahead and it worked:
%s/^\(\(test\)@!.\)*$\n//

Interestingly, if I add 'g' at the end of the regex I use in SlickEdit, it works.
%s/^((?!test.)*$\R//g

Also, when running my actual regex on a large file, there were many more lines other than line 2 that were not deleted even though they were highlighted when search with the regex. Again, if I added the 'g' at the end, it worked.

Currently, SlickEdit's VIM emulation substitute command doesn't support a multi-line search and replace at all. It worked when you used the 'g' but only because the range was the entire file (a bit of luck). No way to fix for v20 but we'll try to overcome this limitation for v21.

[davew]

Thanks Clark. Hopefully the :v/xxx/d will be fixed as well.

[Clark]

Both are on the list speed is a big deal

[davew]

I'm really glad to hear that speed is a big deal because when I talk to co-workers about SE, I always have to tell them that it's a great editor, but it is slow at searching. Hopefully, this will get much better in the future.

[Clark]

Took a quick look at what was going on with g!/whatever/d.

Biggest issue is that the SlickEdit vim delete command copies the text to the system clipboard (not just internal memory)!!! We can tweak that so that it doesn't do that when in the global command.

There's also a bad issue with it restarting it's search from the top of file (top of range) if the number of lines in the file changed when the command was executed. This is an N Squared problem. Sounds like the worse of the two problems but it's squaring a somewhat small number.

This is all fixable.

FYI: The multi-file search/replace speed has been greatly optimized for v21  (16x faster on my machine). We finally got the right engineering in place so we could make this change. It's actually faster than sgrep.exe because it's single threaded. For now, if color coding is specified, it has to revert to the old performance but that isn't needed that often. Eventually, we will make that work fast as well.

The options dialog search has also been optimized for v21 since it wasn't fast enough.

v21 will also have a tweak for a single file search and replace which makes it almost twice as fast if there are tons of replaces and the search string length doesn't match the replace string.

Any time you find a speed problem, post something on the forum. We always keep a list of performance issues we would like to address but user requests get priority when possible.

[davew]

Sounds great Clark!
When's v21 scheduled to be released?

[Clark]

v21 is still a ways off. Definitely before the of the year.