speeding up auto-completion and other tag related help

[greggman]

continued from the unhappier and unhappier with v11 thread

Since v10, the auto-completion type of features in slickedit have gotten so slow that they often make me want to turn them off. I understand the tagging issue is a hard problem. It certainly seems to be a loading/caching problem but as an example just for testing I went into file that has a rather large C++ class. Inside one of the methods I typed "this->"

The moment I typed the "s" in "this" v11 froze for 3-4 seconds. Completely froze.

I then opened the same file in Visual Studio 2003 and in the same spot typed "this->".  VC++ never froze and it only took what felt like about 0.1 seconds for it to bring up the completions.

Note that both of these tests were after a fresh reboot so there was no caching invovled as far as I know.

Also note that that 3-4 seconds is rather short. I've often had to wait 10+ seconds which seems like forever when all I'm trying to do is type the next character and don't really want the completions.

I'm sure there's got to be some solution to speed up the tag stuff for v12. If it is only a loading/caching problem then maybe an option to preload tags when opening a project would help.

As for reproducing them for test, this is a hard problem because short of sending you my entire code base and all the libraries I have tagged their isn't a lot I can do. I can say I've got windows headers, Maya headers, Sony PSP headers, Boost headers and my own libraries as a minimum tagged. But, they are also tagged in VC++ as well. Of course VC++ doesn't appear to keep any global tags, it just does it per project but as far as the database being big, well, my Google Desktop installation has 1 million items scanned and never takes longer than 0.5 seconds to give me a search result. I'm sure that's an apples to oranges comparison but there's got to be some creative ideas to speed up slickedit.

It's also not just about loading/caching because it happens to me often during the day and I only load slickedit once. I suppose slickedit could be releasing what it has loaded or maybe everytime it retags in the background it discards it's cache?

[srouleau]

For what it's worth, I get the same sort of behavior as well, but mainly when I press Ctlr-.  It will often just sit there for a good 15 seconds, during which it won't repaint, and appear completely dead.  I don't recall V10 doing this, or at least not as often.

I don't have 180,000 files in my project; rather it's more like ~23,000.  I can't recall off-hand if it lags the same on smaller projects, I'll try paying more attention to it.

As greggman stated, it's not like I can send the source code over.  But, perhaps I can send the tag files, and .sta?  Would that help in any way, without compromising my client's source base?  (Pretty sure you can't do much with _just_ the tag files, reverse-engineering-wise, but hey, gotta ask)

Steph

[ScottW, VP of Dev]

I checked, and having the tag files and state file without the source isn't any help. Bummer. I think what really helps are detailed scenarios about when the problem occurs. What action precipitated the delay (pressing Ctrl+., typing a dot or -> on a symbol, etc.), what symbol was being looked up, is it in your code or in a library, what's the nature of the symbol (function, class, templated class, etc.) how long you've been coding (so we can determine if this is a cache hit problem), the length of the delay, etc.

Wow! That's a lot of questions. 

We should be able to use this with our knowledge of the inner workings to figure out a few things. I don't know how Visual Studio handles this, but it could be that's one of the reasons they take 30 seconds or more (on my machine) to open a workspace, to provide time to load tags or equivalent data.

Thanks for the help trying to figure this out. Please believe me this issue has our full attention!

--Scott

[srouleau]

BTW Scott, I haven't forgotten -- but you've jinxed it.  It hasn't occured so far this week, and I've been working on a couple different projects, some with 200-300 files, others with 23,000+... 

I'll keep you posted.

Steph.

[ScottW, VP of Dev]

Hey! Maybe we've come up with a new product support approach. Customers report a problem. I use my magical jinx powers and the problem goes away. Think of the time we'd save on actual bug fixes. Then I could teach seminars to other product managers about how to use the power of jinxing to resolve issues. Ah, for a moment there I was in a very, very happy place.    If only problems were so easily resolved.

Of course, this intermittent nature is one of the reasons this problem still exists. You all know the difficulty fixing bugs you can't reproduce reliably.  Just let us know when you come up with something. We're working on some ideas of our own, and I have a project that I can get to hang now and then so I'm trying to pin this down for the developers.

[srouleau]

Oh!

Just the fact that you can get it to reproduce from time to time internally is music to my ea.. erm eyes.  Thought it was just me and Greggman going nuts. 

BTW, nope, never seen intermittent bugs.  Nope, not once.

Steph.

[hs2]

I've noticed a few other huge but maybe hidden performance impacts, which are unrelated to Slick.

- there is enough PHYSICAL memory available to make Slick happy
- no other app. has gone crazy in the background and is life-locking your computer eating up your CPU(s)
- nothing Slick related is located on a network share (and you are not aware of it)
- virus real-time protection is properly configured / off

Excuse me - it's quite obvious. Just some bad experiences ...

BTW: Is there an increased system load (as a sign that Slick is really busy servicing the tag-request) when you're forced to have another coffee ? Or is it blocked on / waiting for something else most of the time ?
Could be an add. hint for the S-Team...

HS2

[srouleau]

Okay -- just got it to happen, but not sure it'll help.

I was on a line with

if( stuff->method() == constant ||
    stuff->method() == other_constant )

I placed my cursor on the first ->method (at the 'm' to be precise), pressed Ctrl-. (I just wanted to see the prototype, to know what it returned), and had to wait a good solid 15 seconds (subjective 1 hour, of course)

When it came back, it gave me 24.5 pages-worth of matches (ie: I could press page-down in the tag dialog match 25 times, with the last one being roughly half full).  This is a base class in the project I'm working on, with a lot of overrides as you can see.  I pressed ESC, did it again, and of course it came up instantly.

The project has 24972 files.

Stephane

[Phil Barila]

It's not just Stephane and Gregg, I see similar things with a Visual Studio 2003 workspace with 17 projects and less than 500 source files.  Using Process Explorer from sysinternals, I can see that SlickEdit isn't doing any IO, and isn't driving the CPU to the rail.  In fact, the CPU usage is ridiculously small.  I didn't expect it to do any IO, since I have 3 GB of memory in the system, and have the tag cache set to 256MB.  Since I saw it appear to hang when doing a Ctrl+/ (Go to reference...), I expected it to be CPU bound, but that's not what I saw.

The AV scanner isn't doing anything, either.  I've seen the AV scanner hang an app, this is not the same.  When the AV scanner hangs an app, the CPU usage of the scanner goes high.

One possible contributor, I have tag files for the Windows Server 2003 SP1 Platform SDK, the Windows Server 2003 SP1 DDK, the KMDF DDK add-on, and wxWidgets 2.6.3, and Cygwin gcc, all with references.  I also have Visual Studio 2005, Visual Studio 2003, and Cygwin-3.4.4-i686-pc-cygwin compiler tag files, which do not have references.  Since there is already a Cygwin compiler tag, I've just removed my own cygwin gcc tag file.

[srouleau]

Happened again...

I'm back on the 20K+ files project.  I did a tag lookup, jumped to it, then press Ctrl-/ to get to all the references to it.  The 'references' window popped up, I pressed 'ESC' to have it go back down, pressed Ctrl-L to get to the first reference, and it froze for a good 10-15 seconds, no repaint, nothing.  I had time to start task manager and see that it was using up 25% of my CPUs (dual core + HT, so "4" cores).  Memory was at ~150MB, but not climbing that much.

Hmm, 150MB may be a bit small, I'll try increasing that.

Steph.

[srouleau]

Further follow-up.

1) On that computer, I'm using 10.0.3, not 11.0.2.
2) I increased the tag buffer to 512MB (i've got 8GB physical on that box), same result.  In fact, so far I can get it to crawl everytime I try to do Ctrl-/ on that particular symbol.  I'd thought that once it'd slowed down once, doing the same lookup right away would be fast but nope, same behavior.  I suppose this may be a Good Thing, if you guys want me to try anything else on it.

(as a note to myself, in case the reply comes weeks later on, the symbol itself is 'drawCheck()' in the DX9 layer :-D)

Steph.

[ScottW, VP of Dev]

Great information! Thanks for the contributions.

Addressing srouleau's issue:

Ctrl+/ is a somewhat different case than regular tag lookup. By it's nature, it's finding a list of references, which means it will take longer the more references that are found. You can control the responsiveness of this feature by changing one key setting.

By default Ctrl+/ builds the list of references and lists each occurrence within the file of the specified symbol. The tagging database knows which files the specified symbol is in (or potentially in), so that part is fast. But it must scan each of those files to find the precise location and pull out the displayed information in the tree list.  I suspect this is where you are seeing the slow down.

To improve performance, select Tools > Options > Tagging Options and put a check in "Find references incrementally (faster)". Now SlickEdit will build the file list in the references tree view, but it will only expand the first file. You can then scan the list of files containing references and expand the ones you are interested in.

While you are on that option screen, you may want to put a check in '"Go To Reference" only lists references'. Otherwise, Ctrl+/ will jump to the first reference in the list, which isn't statistically likely to be the one you're interested in.

As for your CPU utilization, 25% sounds about right. This operation is not multithreaded, so at best it could only completely use one core. Believe me, this is an area we are actively discussing. Unfortunately, this is one area where having such a long history works against you.

I think a similar scenario is at work on the case you posted when doing Ctrl+. and got 24.5 pages of matches. Again, we know what file the tag is located in, but the exact line is something that we scan for. That has to be about 200-300 files that need to be scanned, so this is going to be time-consuming. Have you seen this take a long time when it only finds a single hit (jumps directly to the match) or when the list is less than one screen?

Keep feeding us this information so we can determine what we can do to better optimize this for performance.

--Scott

[srouleau]

To improve performance, select Tools > Options > Tagging Options and put a check in "Find references incrementally (faster)". Now SlickEdit will build the file list in the references tree view, but it will only expand the first file. You can then scan the list of files containing references and expand the ones you are interested in.

Close, but no cigar -- this was already selected. 

While you are on that option screen, you may want to put a check in '"Go To Reference" only lists references'. Otherwise, Ctrl+/ will jump to the first reference in the list, which isn't statistically likely to be the one you're interested in.

Hmm, I'll try that.  I also see "Update references and call tree on single click", which sounds, well, something you'd do with a mouse, which I rarely use :-D

As for your CPU utilization, 25% sounds about right. This operation is not multithreaded, so at best it could only completely use one core. Believe me, this is an area we are actively discussing. Unfortunately, this is one area where having such a long history works against you.

Oh, I believe you!  I'm supposed to find ways of threading this project I'm working on, which has 15+ years of history behind it.  Huh huh...

I think a similar scenario is at work on the case you posted when doing Ctrl+. and got 24.5 pages of matches. Again, we know what file the tag is located in, but the exact line is something that we scan for. That has to be about 200-300 files that need to be scanned, so this is going to be time-consuming. Have you seen this take a long time when it only finds a single hit (jumps directly to the match) or when the list is less than one screen?

Unfortunately, it's not going to be this easy.  For this particular case, I'm dealing with a symbol that's found in a single .cpp file, referenced about 90 times from within that single file.  It's a local function -- no templates, no class, just a function.

Steph.

[greggman]

This isn't one of the worse cases but it is repeatable on my machine and maybe it's related. Here are the steps to recreate it

* Create a new file "foo.cpp"
* enter the following code

Code: [Select]

int foo (int bar)
{
return bar * 2; // some comment just the make at least one line kind of long not that it really matters
}



* Select all those lines and including the blank lines after
* Copy (Ctrl-C)
* use the "argument" command. In my setup it's Ctrl-R
* type 800 and then press Paste (Ctrl-V)
* Go to line 400
* type "#i" as though you were going to type #if but don't type the f.
* press space (actually typing anything that is not an "f" will cause the problem)

On my machine slickedit will freeze for 2 seconds. Press backspace and slickedit is frozen for another 2 seconds.

Another interesting artifact. Put the cursor in the first column and just try to cursor past the #i (ie, press the up and down cursor keys).  As I pass slickedit freezes for 1-2 seconds.

hmmmm, maybe this belongs in a different topic as actually it appears to not be tag related, it's matching related but if entering parens doesn't have this problem.

[Clark]

I just tried the above ("#i<space>") on my machine and it is instantaneous.  The resulting file was 2005 lines and line 400 was a blank line.  The difference could be the tag files you have.  Other than that, I'm a bit baffled.  Our SlickEdit workspace has about 2500 files in it (not huge but not tiny either).