Author Topic: Find in files with many files and subdirectories search freezes SE  (Read 139 times)

rowbearto

  • Senior Community Member
  • Posts: 1075
  • Hero Points: 62
Find in files with many files and subdirectories search freezes SE
« on: September 13, 2018, 09:11:58 pm »
In beta4 when I do a "Find in Files" that needs to search over a large number of subdirectories and files, SE seems to freeze up until the search is practically over. I think I've noticed this for all the earlier v23 betas as well.

In previous releases I thought I remember that SE will populate results as it finds them into the search results window, and SE won't freeze so that I can continue working while a long search is ongoing.

My multi-file search is over a network on an NFS mount and our network is pretty fast, but it is not the speed of an SSD.

I'm using Slickedit v23.0.0.5 on Linux x64, CentOs 7.2,
« Last Edit: September 14, 2018, 01:25:50 am by rowbearto »

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 4393
  • Hero Points: 328
Re: Find in files with many files and subdirectories search freezes SE
« Reply #1 on: September 14, 2018, 01:24:31 pm »
Some generation of the file list is done on the main thread. Some of the file listing is threaded. Searching starts before thread file listing finishes.

You could try turning off "List current context" and see if that makes a difference. This feature takes a lot more overhead.

« Last Edit: September 14, 2018, 01:28:21 pm by Clark »

Lee

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 1175
  • Hero Points: 112
Re: Find in files with many files and subdirectories search freezes SE
« Reply #2 on: September 14, 2018, 01:31:13 pm »
Are any of the files in the search path open in SlickEdit?  There are cases where if the file is open in SlickEdit , the foreground process will search the open file instead.  If the file is large, or the search expression is complex enough, then that could cause something like that.  You could try closing all open buffers and repeat the search to see if there is a difference.

rowbearto

  • Senior Community Member
  • Posts: 1075
  • Hero Points: 62
Re: Find in files with many files and subdirectories search freezes SE
« Reply #3 on: September 14, 2018, 01:47:31 pm »
Turning off "list current context" is not helping. The issue still happens when I have no buffers open in SlickEdit.

I do have the following characterization though:

My search is over a directory tree with 34081 files in it. My original search only wants the files that match filespec '*.rules', there are only 28 files that match this pattern out of 34081 files. The search over *.rules with only 28 files out of 34081 there is what freezes SE for a very long time. If I search all files, '*', it freezes for a much shorter time and finds one file quickly, then there are 3 threads of sgrep running in the background for a while until the search is done.

So I think Clark is probably on the right track in that the file listing in the main thread is freezing up SE in this case.

rowbearto

  • Senior Community Member
  • Posts: 1075
  • Hero Points: 62
Re: Find in files with many files and subdirectories search freezes SE
« Reply #4 on: September 14, 2018, 01:51:42 pm »
I also get a similar long freeze if I just search for "Files" ("Files" tab in BigFind, not "Find in Files") that match the *.rules pattern. It freezes for a while until first few are found, then unfreezes while I see one thread of sgrep running in the background.

Lee

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 1175
  • Hero Points: 112
Re: Find in files with many files and subdirectories search freezes SE
« Reply #5 on: September 14, 2018, 02:00:37 pm »
Files search is also now done in background process and is single-threaded, that checks out.  sgrep should be handling all the directory and file pattern matching, and in Files mode doesn't even bother with checking document type.  SlickEdit only should be waiting for results from pipe process, which is on a timer.  Could you try doing a Slick-C profile during search?  See if that can turn up anything.

rowbearto

  • Senior Community Member
  • Posts: 1075
  • Hero Points: 62
Re: Find in files with many files and subdirectories search freezes SE
« Reply #6 on: September 14, 2018, 02:02:44 pm »
I can do a profile, but I have not done one before. Are there instructions somewhere that I can follow to do that?

Lee

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 1175
  • Hero Points: 112
Re: Find in files with many files and subdirectories search freezes SE
« Reply #7 on: September 14, 2018, 02:18:16 pm »
Sure, Macro > Slick-C Profiler > Begin Profiling... and then Macro > Slick-C Profiler > Finish Profiling... when you are done capturing what you need.   The results will popup with the immediate results, then you can  Macro > Slick-C Profiler > Save... to save it to file.  Just post the saved tsv file and I can take a look the results see if there is any macro calls that spike.  Could be something entirely outside of interpreter, but may help us narrow it down.

rowbearto

  • Senior Community Member
  • Posts: 1075
  • Hero Points: 62
Re: Find in files with many files and subdirectories search freezes SE
« Reply #8 on: September 14, 2018, 02:20:41 pm »
I did it for "Find in Files" and also for "Files" - both profiles attached.

Lee

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 1175
  • Hero Points: 112
Re: Find in files with many files and subdirectories search freezes SE
« Reply #9 on: September 14, 2018, 02:26:56 pm »
Also, could you post the top-line of the Search Results for either search?

Ex:
Find all "blabblah", Subfolders, Zip files, "E:\work\files\", "*.c;*.cxx;*.cpp;*.c++;*.h;*.hpp;*.hxx;*.h++;*.inl;*.cs;*.e" 

rowbearto

  • Senior Community Member
  • Posts: 1075
  • Hero Points: 62
Re: Find in files with many files and subdirectories search freezes SE
« Reply #10 on: September 14, 2018, 02:29:33 pm »
Find in Files: Find all "PROGS", Subfolders, "/home/rbresali/v/git/rbresali-neapi-int/src/pss/node", "*.rules", Exclude: ".git/.svn/;libbm_testing/;testing;boost/;*.o;*.a;*.so;*.so.*;*.vtg;*.contrib;*.O;*.cs.sim;.cmake.state;.git/;*.pyc;*.tgz;*.mvl;*.wrl;*.P;*.ptl;*.d;*.ex;<Binary Files>"

Files: Find files "/home/rbresali/v/git/rblasse-TlsComms_v2/src/pss/node", "*.rules", Exclude: ".git/.svn/;libbm_testing/;testing;boost/;*.o;*.a;*.so;*.so.*;*.vtg;*.contrib;*.O;*.cs.sim;.cmake.state;.git/;*.pyc;*.tgz;*.mvl;*.wrl;*.P;*.ptl;*.d;*.ex;<Binary Files>"

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 4393
  • Hero Points: 328
Re: Find in files with many files and subdirectories search freezes SE
« Reply #11 on: September 14, 2018, 03:27:27 pm »
We reproduced a problem. This doesn't happen on Windows. On Unix, it appears that when the sgrep process is listing files, the SlickEdit process is blocked somehow.

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 4393
  • Hero Points: 328
Re: Find in files with many files and subdirectories search freezes SE
« Reply #12 on: September 14, 2018, 06:08:25 pm »
Fixed for beta 5. There was an extra layer of read buffering that was blocking when there's no data to read. Easy fix.

rowbearto

  • Senior Community Member
  • Posts: 1075
  • Hero Points: 62
Re: Find in files with many files and subdirectories search freezes SE
« Reply #13 on: September 14, 2018, 06:16:09 pm »
Thanks Clark and Lee!