Slow updating projects filelist.

[perssontm]

Hello,

I have a workspace with a bunch of projects, and its really slow updating the projects filelists. I have used the add wildcard feature to select some specific filetypes that Im interested in. Is there some way to speed this up? There are a lot of small files in the projects, but perhaps theres some trick to make it faster?

[sdayts]

I have also noticed this.  This prevents me from using "Add wildcard" feature.  In all fairness, there are close to 16000 files in the list.

[perssontm]

Yes, really annoying. I somewhere around that, today I splitted the workspace in several projects to keep the number of files down per project, but its still to slow to not be annoying. If it was possible to update a folder within a project, it could go faster, since you would need to update less files, but thats just a suggestion.

[perssontm]

I decided to try to get this fixed, since its really annoying. So I started slickedit with strace and watched the output.
From the start I didnt have any workspaces or anything open, which I did expect should give me any output at all from strace, but I got constants calls like this, quite fast:
gettimeofday({1205356274, 717581}, {4294967236, 0}) = 0

ioctl(3, FIONREAD,

• )                 = 0

gettimeofday({1205356274, 774746}, {4294967236, 0}) = 0
gettimeofday({1205356274, 863118}, {4294967236, 0}) = 0
gettimeofday({1205356274, 913724}, {4294967236, 0}) = 0
gettimeofday({1205356274, 917318}, {4294967236, 0}) = 0
ioctl(3, FIONREAD,

• )                 = 0

gettimeofday({1205356274, 917609}, {4294967236, 0}) = 0
select(4, [3], NULL, NULL, {0, 5000})   = 0 (Timeout)
ioctl(3, FIONREAD,

• )                 = 0

ioctl(3, FIONREAD,

• )                 = 0

poll([{fd=3, events=POLLIN}], 1, 0)     = 0
ioctl(3, FIONREAD,

• )                 = 0

gettimeofday({1205356274, 923411}, {4294967236, 0}) = 0
gettimeofday({1205356274, 923555}, {4294967236, 0}) = 0
ioctl(3, FIONREAD,

• )                 = 0

gettimeofday({1205356274, 923845}, {4294967236, 0}) = 0
select(4, [3], NULL, NULL, {0, 5000})   = 0 (Timeout)
ioctl(3, FIONREAD,

• )                 = 0

ioctl(3, FIONREAD,

• )                 = 0

I then opened a workspace, and expanded the first level of a projects tree on the left, and got this output:

2, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl64(12, F_SETFD, FD_CLOEXEC)        = 0
getdents(12, /* 6 entries */, 4096)     = 120
stat64("/home/eric/dev/web.persson.tm/opt/www/persson.tm/sites/misc/samples/xmlrpc/php/.svn/tmp/wcprops", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat64("/home/eric/dev/web.persson.tm/opt/www/persson.tm/sites/misc/samples/xmlrpc/php/.svn/tmp/.", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat64("/home/eric/dev/web.persson.tm/opt/www/persson.tm/sites/misc/samples/xmlrpc/php/.svn/tmp/..", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat64("/home/eric/dev/web.persson.tm/opt/www/persson.tm/sites/misc/samples/xmlrpc/php/.svn/tmp/props", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat64("/home/eric/dev/web.persson.tm/opt/www/persson.tm/sites/misc/samples/xmlrpc/php/.svn/tmp/prop-base", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat64("/home/eric/dev/web.persson.tm/opt/www/persson.tm/sites/misc/samples/xmlrpc/php/.svn/tmp/text-base", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
getdents(12, /* 0 entries */, 4096)     = 0
close(12)                               = 0
gettimeofday({1205356127, 946302}, {4294967236, 0}) = 0
stat64("/home/eric/dev/web.persson.tm/opt/www/persson.tm/sites/misc/samples/xmlrpc/php/.svn/tmp", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
gettimeofday({1205356127, 946472}, {4294967236, 0}) = 0
stat64("/home/eric/dev/web.persson.tm/opt/www/persson.tm/sites/misc/samples/xmlrpc/php/.svn/tmp/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=675, ...}) = 0
readlink("/home/eric/dev/web.persson.tm/opt/www/persson.tm/sites/misc/samples/xmlrpc/php/.svn/tmp/wcprops", 0xaf836410, 512) = -1 EINVAL (Invalid argument)
stat64("/home/eric/dev/web.persson.tm/opt/www/persson.tm/sites/misc/samples/xmlrpc/php/.svn/tmp/wcprops", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
gettimeofday({1205356127, 946949}, {4294967236, 0}) = 0
open("/home/eric/dev/web.persson.tm/opt/www/persson.tm/sites/misc/samples/xmlrpc/php/.svn/tmp/wcprops/", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|0x80000) = 12
fstat64(12, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl64(12, F_SETFD, FD_CLOEXEC)        = 0
getdents(12, /* 2 entries */, 4096)     = 32
stat64("/home/eric/dev/web.persson.tm/opt/www/persson.tm/sites/misc/samples/xmlrpc/php/.svn/tmp/wcprops/.", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat64("/home/eric/dev/web.persson.tm/opt/www/persson.tm/sites/misc/samples/xmlrpc/php/.svn/tmp/wcprops/..", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
getdents(12, /* 0 entries */, 4096)     = 0
close(12)                               = 0
gettimeofday({1205356127, 947562}, {4294967236, 0}) = 0
stat64("/home/eric/dev/web.persson.tm/opt/www/persson.tm/sites/misc/samples/xmlrpc/php/.svn/tmp/wcprops", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
gettimeofday({1205356127, 947736}, {4294967236, 0}) = 0
stat64("/home/eric/dev/web.persson.tm/opt/www/persson.tm/sites/misc/samples/xmlrpc/php/.svn/tmp/wcprops/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=675, ...})


Since I dont have lot of experience reading strace output, I can mostly guess. And it seems like its doing a filestat on each file in the path of the project(I have several include with wildcards). It takes a lot of time, since there are about 32000 files below the projects root. I'm also amazed by the repeted calls to /etc/localtime, it seems like that should be kept in memory instead of beeing checked all over for each time, but the kernel should do that anyway, but yet again, guessing here.

I'm using slickedit 12.0.3 on Linux with the latest cumulative hotfixes, downloaded and installed today.

Is there anything else I can provide, or should I just assume that this is how slickedit works, or can I change the config somehow?

/eric

[Dennis]

Try this with your large network source tree.

    find . -name "*.cpp" -print | sort >/tmp/list.txt

I'm interested in the comparitive times.  I don't expect us to beat 'find', but in the case that a 'find' even takes a long time, how can you possibly expect a wildcard project to perform well?  Obviously, you have to keep in mind that SlickEdit has a whole lot more work to do to get the file list and then dump it into the Project window and make sure the tag file is up-to-date.

I think that to an extent, wildcard projects are a "you get what you asked for" sort of scenario.  We do have a plan for a hybrid wildcard project file specification that will perform better at project open, but will require manual resync's, but that is a bit of an undertaking, so it is being prioritized for a future release.

[perssontm]

Did this,
time find . -name "*.php" -name  -print | sort >/tmp/list.txt

real    0m3.725s
user    0m0.184s
sys     0m0.347s

In one of my folder where theres a lot of files, went fast. The interesting thing is that this is not a network drive, but my local drive. In slickedit, the "same" process takes between 1-3 minutes, which is a really annoying break when youre in some kind of flow.

Is there some things I can deactivate to get it to spend less time on this?

[Dennis]

Try closing the Project tool window.
Also try closing the Files tool window (if you are using the "workspace" view).

Also, can you confirm, did that find locate all the files that are in your workspace, or were you just testing one project's worth?

[chrisant]

I, too, work in multiple projects with over 16,000 files each.  You're right, Dennis, the scan is going to be slow.  But the user doesn't necessarily have to be blocked during the scan.  I was (un?)lucky enough to get to implement project support in the editor I was using before I switched to Slick.  I cheated.  Here was the hybrid approach I used, and it worked pretty well:

- The project definition showed the list of wildcards.
- The wildcards were used to build a file list in memory, but the generated file list also got saved/loaded with the project.
- Resync was automatic, in the background (on a background thread, but it could be emulated in the foreground thread during idle timeslices).
- In all of the dialogs that referred to the file list, there was a key binding and/or button to forcibly trigger a new background resync.
- While a resync was in progress, the old file list was still in effect until the resync was finished.
- Also while a resync was in progress, anywhere the UI showed the file list it also showed a count of files found so far (in a statusbar at the bottom of the file list UI).

If resyncs are done too infrequently or too passively, the file list is out of date.  If resyncs are done too frequently or too aggressively, the performance drain on the system can become an issue of its own.  The challenge is to balance the two, but in keeping with the configurable theme of Slick, perhaps it would be sufficient to expose a lot of knobs and levers so people can tune the performance to their liking, similar to the tuning strategy for background tagging.  With a key/button to trigger a resync (embedded in every prompt/dialog/window that could care...), it is probably reasonably safe to skew towards resyncing less frequently and less aggressively.  Also, the basic strategy can be enhanced, for example if the project is closed while a resync is in progress it could save off the path that it is currently scanning, and then resume from there next time the project is opened.  That should ensure uniform coverage over all parts of the project tree, regardless of how long it takes to perform a resync (or how many times it gets interrupted and restarted, etc).

[perssontm]

Wouldnt it be possible to write a slick-c extension that only resynced one folder within a project?

And why does it stat the files, once it have done it, is it that it does the entire scan each time its opened/closed due to the wildcard option?

Will try to get some comparison between a find and the slick-way.