Smart open slow and blocking when workspace has huge number of files

[rowbearto]

In v26 on Linux x64 I've noticed that when I have a workspace with a huge number of files (over 34,000 files, multiple projects, 2 of them have roughly 17K files) that smart open can get very slow.

When I try to open a file under my home directory such as ~/temp/error.log, it often freezes in the middle of typing this in the smart open edit box and I have to wait multiple seconds or 10s of seconds. I think it is because the smart open functionality is searching through the large amount of possible files in the workspace and blocks.

I don't remember behavior like this in v25 and earlier releases.

Would be nice to have this fixed.

[Clark]

Smart open isn't threaded and never has been. It uses similar code to the Files tool window but has to search more lists (files in current directory, buffers, etc.).

Does your project have any wildcards? Just wondering what type of test case I'll need to create.

[rowbearto]

I don't have any wildcards.

Maybe you can try with a huge project like the linux kernel? And don't use wildcards.

I don't remember this stalling in old releases, I think I would have remembered it. So I suspect some change in v26.

[Clark]

I created a project with 67,000 files (Qt 5 source) and tested with a debug build to slow my machine down. No luck reproducing this. I'm going to need something more specific. Maybe you can download an open source project and reproduce this. This might be specific to something in your project.

I do get some 1-1.5 second delays.

[rowbearto]

Look for slow_smart_open.tar.xz on support.

See the README file.

I also used qt5 but ended up with 128,000 source files after recursively pulling all submodules.

The delays that I see with this are comparable to what I have with my 37,000 file project.

I included a video demonstration, my user.cfg.xml, my .vpj file, and a README.

[Clark]

v25 does seems to be quite a bit faster. Not sure why. Definitely worth tracking down.

[Clark]

Fixed for v26.0.1

SlickEdit has a file regex class which does all the file pattern matching. For v26, it was enhanced with options to support full regular expressions (not all that accessible yet). The limited performance testing we did showed that it was faster (not sure how). With this type of wildcard match and some others I tested it's almost 3x slower for a release build. Fix was to use the old algorithm when new features not needed.

[rowbearto]

Thanks so much Clark!

[rowbearto]

Perhaps another optimization you could do is that if I start my entry in smart open with a tilde character (for my or other's home directories), then you don't search the project? Maybe that could only work in Linux.

[Clark]

The problem here is that all cases were too slow. Even simpler test cases like test.cpp. Solution needs to be to make it faster. To handle a project with >=300,000 files, would probably require threading some of this code.