Linux: Find in Files is not threading in beta 2?

[rowbearto]

I am doing "Find in Files" on a large directory tree. It is taking a very long time. I'm using top -H and I see only 2 threads of vs_exe, one using 10% of CPU and another user about 8%. I was expecting to see multiple threads of parallel searching, but don't see it.

In my Tools->Options->Editing->Search the "find in files threads" is set to 8, but there are only 2 threads of SE running. I don't see any other threads using as much CPU. I thought maybe there is another executable that SE calls to do search but I don't see that in my thread list in top.

I'm using Linux x64/beta2.

[Clark]

If you use the "Line current context" option, the search will be single threaded.

More specifically, there are two processes (two instances of SlickEdit). One generating the output, and the other reading the output.

[rowbearto]

Have previous versions of SE been single threaded for "List current context"? Or is this something new in v26?

Why can't "List current context" also be multi-threaded?

I always have this enabled by default - so useful. It would have to be a conscious decision to turn this off.

If it can't be multi-threaded would be nice to have some kind of warning about it (something like an 'i' to click) to let me know that it will be single threaded.

But would be SO GREAT if "list current context" could also be multi-threaded.

Indeed when I turn off "List current context" I see 8 threads of sgrep and the search is tremendously faster.

[Clark]

Too much of the code used by List current context isn’t multi-threaded. Maybe some day but it’s a monumental under taking.

[rowbearto]

Maybe it would be useful to add to the config to provide a threshold if more than "X" files are to be searched then don't use "List Current Context" and multi-thread instead?

This way I can always have "List Current Context" checked and then for large filesystem searches it can be disabled to allow faster multi-threading. I wouldn't have to remember to turn it off.

[rowbearto]

Couldn't you use sgrep for "list current context" and then when SE is displaying the output if "List Current Context" is checked then the output thread would insert the context.

Then you could have a "list current context" implementation that would not be such a monumental undertaking?

[Clark]

This isn’t very possible. keep in mind that completely different processes are being executed.  There’s no way to switch processes later. also the listing of files is handled in the additional process.

[rowbearto]

Why use different processes? What about right from the start with 'list current context' you use the same processes as used for 'sgrep'?

Then when the output thread is getting the results from sgrep it can add the current context if 'list current context' is checked.

EDIT: OK I think you were answering about using a threshold of # of files, yes I could see that you'd need to switch process for that. But if you used sgrep for 'list current context' searching then you wouldn't need to switch processes.

[Clark]

Case 1 sgrep is used. Case 2 vs is used. These processes are insanely different. Can’t use sgrep because it zero code for doing parsing and context tagging. That’s massive and requires the editor. Like I said.  Not realistically doable.

[rowbearto]

I think you could still use sgrep to find the text only (not the context), then use the editor to find the context based on what sgrep found. When sgrep finds the text it tells the editor/output thread the file#/line# of the found text. Then the editor could look up the context. Why wouldn't that work?

[Clark]

The performance is problem is caused by opening each file in the editor so the file can be parsed and context information determined.  This must be done in a separate process/thread and not in the original SlickEdit instance. Maybe if there aren't that many hits, this would be faster. Until all this code can be redone, there's no point.

[rowbearto]

In the search I just did with 36000+ files there were only about 10 files that had the search string. So in this specific case there would have only been about 10 files that would have needed the editor for context vs needing it for 36000+ files.

I would say that the large majority of my search cases there are only a handful of hits when searching a huge number of files, so this could be a substantial improvement.

[rowbearto]

Expanding my last example even with only 10 hits out of 36000 files you could still have just 1 thread (don't rework the code to multi-thread) using the editor to find the context and it would still be so much faster working on 10 files instead of 36000, even with single thread.

So you wouldn't need to rework the parsing/context code for multi-thread, so I would think that could be less work.

[Clark]

This type of solution is within reach but will require a lot of work.  Need to build parts of sgrep into vs.

[Clark]

Adding threading support for "List current context" search option when searching multiple files for beta 3. Significantly faster if less than half the files have a match.

Some of sgrep was already built into vs. It was needed for multi-file search and replace and for foreground searching.