Opening projects with about 90k files significantly slower on v27?

[florihupf]

  Hi there,

I have a project that took about 20s to open on v26 (spinny ball on MacOS) after everything loaded before I can start typing.

This time has been increased significantly with the new v27: It seems it is close to two minutes now.

The workspace has about 4750 projects with a total of about 90k files.

Is there an easy fix? Or something I can help profile this?

Thank you!

[Dennis]

Are you using wildcard projects ?  How many files do you have actually open?  That is, how many files are restored when you open the project?  Look at the Files tool window / Buffers. 

You can profile where it is spending time by closing the workspace, then going to Macro > Start Slick-C Profiling, then re-opening the workspace, then going to Macro > Slick-C Profiler > Save... to save the results of the profiling session.

[Clark]

Try closing the Project tool window. This will have a large effect in the amount of time it takes to initially open your workspace. There isn't supposed to be a difference in performance between 26 ad 27. I'll check. The Project tool window isn't that useful for very large projects like this. You can open files from your project using the Open tool window or Files tool window. 

[Clark]

Does your SlickEdit workspace (.vpw) file have 4750 SlickEdit project files (.vpj) files in it? I'm guessing these are build components but not actual .vpj files.

I tested a wildcard project with ~67500 files in it (workspace with only one .vpj file in it). It definitely doesn't take more time in v27 than v26 to open this large test wildcard project (about 22 seconds for each on Intel mac mini). When I close the Projects tool window and restart the delay was reduced to 0.

I also tested the same project as a static (non-wildcard) file list. No need to close the project tool window when you use a static file list. Delay is only 1-2 seconds. This is because static file lists folders and files are cached better. There is a wildcard cache along with quite a few optimizations but it's still not enough to match static folders and files.

[florihupf]

Thanks Dennis and Clark for all the pointers.  Will do some digging and report back.

[florihupf]

I went ahead and profiled my project open.  I can see it takes a long time, but I don't see why that is.  Does this ring a bell to anyone?  (See attachment).

[Dennis]

If this was a project migrated from an earlier release of SlickEdit, try removing the workspace ".vpwhist" file (or renaming it just to set it aside).  Also worth noting before deleting it just how large it was.  From the profiling I see it spent a significant amount of time in a single call to "_ini_find_section()" which is only possible if the history file was very large (or on a terribly slow disk).  Clark made some optimizations to the history file and how we save and restore the file cache.

[florihupf]

Thanks for the pointer.

I found a 740kb .vpwhistu file.  Let me see if it helps removing it.

Also, yes Clark, the project has 4750 .vpj files.  I generate them using a script to break down things into smallish projects.

[florihupf]

Removing the .vpwhistu files brought down project open to 18s.

However, immediately closing and reopening the project brings it back to 3min.

Hmmm...

[florihupf]

Ah, looks like the culprit is

_workspace_opened_rte(...)

Returning 0 right away makes things super fast and it seems it is all in support for Java RTE, which I don't care about as long as the custom RTE work.

All happy now.

[Clark]

Can you reduce the number of projects? I created a workspace with 4300 projects and I'm seeing a number of performance issues. I suspect I'm seeing worse performance than you because my machine isn't as fast as yours and some of my tests are with debug builds. I'd recommend reducing the number of projects to under 500 hundred.

I will look into improving this but it's looking like a lot of code has performance problems with this many projects.

[florihupf]

Thanks for being able to repro.  And yeah, maybe I can find a way to trim things.  It is just sooooo convenient to be able to switch back and forth in a super fine grained manner.

I will give it a whirl, because in the end, I don't need to context switch _that_ much.
 

[Clark]

For 27.0.1 I've added a cache for the ActiveConfig. For me, this reduced the time in _workspace_opened_rte to about 1.3 seconds in a debug build. This cache also fixes the bad performance problem with using the Files tool window to list Workspace files (Project>Open Files from Workspace...). I also added timeouts for a couple places where it just wasn't necessary to list everything if it's going to take forever (file completion and listing all projects in the menu).

[florihupf]

Thanks for the follow up and improvements Clark, can't wait to try them out :-)

[florihupf]

 

Hi Clark,

I just installed 27.0.1 AND IT IS SOOOOOOOO MUCH FASTER!

Thank you!