Why is 'Find in Files' so slow on SlickEdit 14.0.1.2?

[shadowfax]

Hi,
I downloaded a trial version of SlickEdit Core a couple of days ago. I wonder why is it that 'Find In Files' operations are agonizingly slow, and hang SlickEdit for the first few seconds? When I search for a term in a group of files, SlickEdit hangs, displays "Matching Files..." as its status, and then is completely unresponsive for about 5-10 seconds. Then, it starts to search for term, and reports results - all taking about 15 seconds. I am on a MacBook Pro with 4GB RAM, and don't see a valid reason why the search should be so slow.

Anyone here has had any such experience with SlickEdit search?

[ScottW, VP of Dev]

Generally, our Find in Files is one of the fastest available. Are you searching files that are local to your machine, or are they on a network? If you repeat the search, is it just as slow?

Do you have "Foreground search" checked? If so, that's why SlickEdit is unresponsive during the search. It's devoting all of its time to the search and doesn't allow you to type.

We're checking to see if we can reproduce this on the Mac.

[chrisant]

@shadowfax -- You said the total time is 15 seconds.
How long is it non-responsive before you see it start to search?
How long does the search take once it begins?
How many files does it search?

It sounds to me as if something is taking a long time while spawning the second process that does the background searching.

[shadowfax]

Hi!
Thank you for your responses.

Scott,
All the files on are on my local filesystem. However, these files were checked out from a subversion repository, and has a few SVN metadata files in each of the directory. The total number of source files is about 4000(4k). I do have 'Foreground search' checked, but that does not seem to affect the outcome. It was turned off by default, yet search was slow even then. I turned it on to see if it could speed up my searches. And yes, if I repeat the search, its just as slow each time.

Chris,
The total unresponsiveness is for almost the entire duration of time. The time is variable, but its always in 10s of seconds. Once the actual search starts, its pretty fast - less than a second. The search space is about 4000 files. I wonder what is it about my system that's tripping up file matching. Does SlickEdit rebuild its file database(and/or search index) each time I search for a term in the files of the project? If so, is there a way to turn it off?

[ScottW, VP of Dev]

Can you post a screen shot of the Find and Replace dialog with it filled in for the search you are describing? See the one I have attached as an example.

Be sure to uncheck, "Foreground search". It will make SlickEdit unresponsive and isn't much (if any) faster (based on the latest intel from one of the developers). It looks faster because the file being searched is updated more frequently.

SlickEdit does build a file list each time you do a search. That's what's going on when the "Matching files" message is being displayed. It's just surprising that this is taking so long on your system. In my tests, that message goes by so quickly I hardly get to read it.   

I did a search of about 4200 files using the options in the attached screen shot. It took about 30 seconds to complete on my Windows machine.

[ScottW, VP of Dev]

I just noticed that you said, "SlickEdit Core" in your original post. Are you using our Eclipse plug-in or our standalone editor?

[Ryan]

How do you have SlickEdit Core on a Mac?

There is no Background Search option on SlickEdit Core, because the way that is implemented is by spawning a second vs process in order to do the search.  There is no vs process to spawn with the Eclipse plug-in.  So 'Find in Files' always does a Foreground search...and yea, as has been said here, you can't really do any editing until the Foreground search finishes.

- Ryan

[shadowfax]

@Scott,
Here's a screenshot of the Find In Files search box on my system. I unchecked the Foreground option, but didnt notice a big improvement in performance.

@Ryan,
I am using the standalone version of SlickEdit and not the Eclipse plugin. Isn't the standalone version called 'SlickEdit Core'? I think I am confused by the naming - anyway, I hope its clear that I am not using the plugin on my system.

[ScottW, VP of Dev]

A couple things to try next:
1) Instead of using <Workspace>, can you point the search to a top-level directory? We're wondering if it is taking a long time to get the list of files from the workspace. How did you set up your projects? Did you use "Add Tree" or "Add Wildcard"?

2) The file types filter is definitely slowing things down. That is a big, generic filter that we created with the commonly used file extensions for a C or C++ project. If you use a smaller set of extensions (like *.cpp and *.hpp), then just use those extensions in the File types filter field *(be sure to use semicolons to delimit the extensions). The fewer you use, the faster the "Matching files" part will take. I also got a significant slowdown using the stock filter you showed.

[shadowfax]

Thanks! That helped. Search is a lot, lot quicker now than it was with the earlier settings.  And, I had used 'Add Tree' to add files to the project.

[ScottW, VP of Dev]

Did you try those separately? I'm hoping the thing that made the difference was the change to the file filters. If not using <Workspace> made a difference, then we'll have to see why there's so much overhead with that--particularly since you used "Add Tree". We thought that, maybe, the wildcards might be getting re-searched if you had used "Add Wildcard".

[chrisant]

I've had to make a potentially related optimization in a reg-ex engine before, so I'll try to describe an optimization in case it could be of value for SE:

When evaluating *, a reg-ex engine typically keeps track of where to back up to if the stuff following the * fails to match.  But it is possible to back up more efficiently, to avoid unnecesssary re-processing -- the optimization can potentially change a mismatch from O(N^2) to O(N) performance.

Since there are many wildcards to evaluate, it could be that the total cost is coming out to roughly O(M*(N^2)*K) where M is the number of files in the workspace, N is the length of the file name, and K is the number of wildcard expressions.  If the back up is not already optimized, then optimizing it might reduce the cost to roughly O(M*N*K) which would scale better.

Attached is an sample of the optimization -- see Flatfile.cpp line 319 (in the cmpsort function).  Flatfile.cpp is part of the source to my EnhProj macro+DLL (see here).  (I tried to paste just the function, for convenience, but the forum doesn't like some keyword in the text and returns HTTP errors, so I just attached the whole file).