How to add thread number to vsmktags?

[onlyflyer]

Hi,
I have a big workspace which have more than 100,000 files and our development server has lots of cores.

I find vsmktags only can occupy ~150% CPU. It takes more than one hour to retag the whole workspace.
I am wandering if we can increase the thread number to speed up the retag work.

Thanks,
Hui

[Graeme]

Did you run vsmktags.exe from the command line?  It has threaded tagging enabled by default but I don't know how many threads it uses.

In slickedit, go to the help, select the Index tab and type in "threads"  - then look at "use background tagging threads".  In slickedit tools-> options, type "thread" in the search box then go to "background tagging".  You can specify the number of threads to use for tagging and the amount of memory to use.  You have to restart slick to get some of these changes to take effect.

[onlyflyer]

I did all what you mentioned. Enable all background tagging knob and set the thread number to the max value(.
But CPU usage is not high, it is 120%~150%.We have many idle cores on server, we hope to make vsmktags much faster.

Thanks!

[Graeme]

I think it shouldn't take an hour.  Maybe it's disk bound.  Are the source files and tag files both on the server?  You can change the location of the tag file via "workspace properties" on the project menu.  In tools -> options -> application options -> virtual memory, try increasing the tag file cache size.

[mjdl]

I think it shouldn't take an hour.  Maybe it's disk bound.  Are the source files and tag files both on the server?

Am I wrong in speculating (technically naive, of course!) that whether or not the tagging process is started by the editor or from the command line, it will still execute on the client workstation and thus be limited by the I/O of the client/server link? In other words, the server CPU is only doing network/disk I/O in this scenario, and there would have to be a really high-bandwidth I/O to saturate the CPU, and in fact the server OS may scale down the CPU demands of the I/O process for other reasons.

Wouldn't somehow making the tagging process execute on the server be a better approach, since then the only limiting factor is the number of cores and local disk I/O? Perhaps via a daily scheduled command-line job on the server, so that the client workstation Slickedit tagging process would effectively only need to keep track of which files needed retagging in the course of work. Of course, adjusting the Slickedit client to cache in memory all the tag files on the server would be necessary.

As I say, I'm not exactly very well informed about all the pieces of the puzzle...

[Dennis]

In terms of I/O, the largest bottleneck is generally where the tag file is written.  As Graeme suggests, adjust Workspace Properties to put the workspace tag file on a fast SSD or if you have enough memory, a RAM disk, and you'll see a big improvement.

Also, if you have enough memory, try adjusting Tools > Options > Background Tagging > Background Tagging Threads > Maximum number of active tagging jobs and Maximum amount of background tagging memory usage (MB).  Give both of them an order of magnitude increase (10,000 and 256 MB), and you'll also see some gains.

The other thing I'd like to point out, and I see this a lot, is that if those 100,000 files are 10 copies of different versions of a set of 10,000 files, then, well, don't do that, it's just not good practice to toss everything into the blender.  It just makes things harder for the tagging engine without providing you with any real benefit.  Create a separate workspace for each version of the files.