New user hitting lots of issues with SlickEdit

[brunokc]

Hi,

I'm a new SlickEdit user, coming from the aging (but still working) Source Insight and I'm trying to get help with some SlickEdit issues. I contacted support but got no answer, and the issues are just getting worse.

The more serious one is a crash at start up. Tagsdb.dll is acting up:

Code: [Select]

0:011> .lastevent
Last event: 1810.2f30: Access violation - code c0000005 (first chance)
  debugger time: Mon Jul 30 13:06:52.776 2012 (UTC - 7:00)
0:011> kn
 # Child-SP          RetAddr           Call Site
00 00000000`0c02f560 00000000`0f825798 tagsdb!tag_restart_async_tagging+0x35679
01 00000000`0c02f570 00000000`0f828913 tagsdb!tag_restart_async_tagging+0x35b98
02 00000000`0c02f5d0 00000000`0f83709a tagsdb!tag_restart_async_tagging+0x38d13
03 00000000`0c02f610 00000000`0f823d10 tagsdb!tag_restart_async_tagging+0x4749a
04 00000000`0c02f710 00000000`0f7e1bff tagsdb!tag_restart_async_tagging+0x34110
05 00000000`0c02f750 00000000`0f7bbe97 tagsdb!slickedit::SETagInformation::propagateTagFlags+0x2ff
06 00000000`0c02f930 00000000`0f7eb8ec tagsdb!SETagInsertFileEnd+0x1d7
07 00000000`0c02f9f0 00000000`0f7f0677 tagsdb!slickedit::SEListTagsTarget::insertAsynchronousTagsInDatabase+0x55c
08 00000000`0c02fb30 00000000`0f7f50c2 tagsdb!tag_restart_async_tagging+0xa77
09 00000000`0c02fd00 00000000`7781652d tagsdb!tag_restart_async_tagging+0x54c2
0a 00000000`0c02fd30 00000000`77cac521 kernel32!BaseThreadInitThunk+0xd
0b 00000000`0c02fd60 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

Before that I was getting a lot of horrible multi-minute UI lock ups (SlickEdit becomes unresponsive). Clicking back onto SE when the UI became responsive would only cause it to go become unresponsive again. That went through for several hours, apparently together with the tagging of files in the background.

Are SlickEdit employees monitoring these forums? Can anybody help me?

Edit: forgot to add that this is happening on SE 17.0.1.0 x64 running on Windows 7.

Thanks,
Bruno.

[Phil Barila]

How did you "contacted support"?  Did you open a support ticket?
Do you get these hangs with a default configuration?  Do you get the crash with a default config?

[brunokc]

Thank you for the reply, Phil. Yes, I opened a support ticket.

I believe I have what you call a default configuration. I installed SE, created a workspace and project and let it tag all files. At first things were not that bad, but at some point I started seeing this in the status bar at the bottom of the SE window: "<file>: Database cannot exceed 2GB" or something to that effect.

I decided to break the project into 2 with the smaller one being used for SDK files. It would still complain about the 2GB limit because it was still tagging the SDK files on the original tag file. Since I couldn't figure out how to remove those, I nuked the tag db (chose the "Rescan Src files" option in the tag screen). That's when the hangs became worse. It was almost impossible to use SE while it was re-tagging files "in the background" as it becomes unresponsive very frequently (something like: hang for 1-3min, become responsive for 10sec, I click something in SE and it hangs again, rinse and repeat).

Eventually the tag db grew to 2GB again. :-(

[chrisant]

I don't think the "Rescan Src files" option nukes the tag db.  I think the only way to reliably nuke a tag file is to close SE and delete the corresponding .vtg file.


Wow about the 2GB file.  I work in some pretty massive codebases, but I've never had a tag file larger than about 600MB.  I do try to manage how many files are in the workspace, and I haven't let a workspace contain more than about 300,000 files before.

Are you at liberty to share the nature of the configuration that's leading to such massive tag files?  What code base and SDK(s) etc are involved?  About how many files are involved in the code base, and about how many in the SDK?

Do you have the SDK files in the compiler tag file, or are they added directly to the Project tag file?

[Dennis]

When you say you "tagged all the files", I assume that means you recursively added "*.*" to the project.  How about if you trim that down to just the source files, like "*.cpp;*.h", for example.

Also, I'm guessing that if you have a 2GB tag file you are probably tagging a lot of stuff that is irrevelant, or in many ways, worse yet, duplicate versions of the same code. 

What programming language are you working with?  Is there embedded code (such as assembly embedded in C)? 

You can delete your workspace history by removing vrestore.slk from your SlickEdit configuration directory, or since you are basically working in a default configuration, just go ahead and delete the entire configuration directory and then start from scratch. 

Hone your project down to what you really need to be working on.  You can create several different workspaces and easily switch between them if you need to work on different versions of the code or different branches of the source tree.

Finally, could you create a mini-dump the next time you get the crash?  It is very easy to do by simply attaching to SlickEdit with Visual Studio.

[brunokc]

Just a note to say that SlickEdit support contacted me and asked me to delete vslick.sta and vrestore.slk and restart SE. While that helped a bit, as soon as I open my workspace and it restarts tagging in the background, I get the crash again.

I'm working on answering your questions next. Will reply shortly.

Thanks.

[brunokc]

I don't think the "Rescan Src files" option nukes the tag db.  I think the only way to reliably nuke a tag file is to close SE and delete the corresponding .vtg file.

I seem to remember seeing the file size going down after that. I'll try to blow away the file as well.

Wow about the 2GB file.  I work in some pretty massive codebases, but I've never had a tag file larger than about 600MB.  I do try to manage how many files are in the workspace, and I haven't let a workspace contain more than about 300,000 files before.

Are you at liberty to share the nature of the configuration that's leading to such massive tag files?  What code base and SDK(s) etc are involved?  About how many files are involved in the code base, and about how many in the SDK?

Do you have the SDK files in the compiler tag file, or are they added directly to the Project tag file?

I guess I can talk about number of files and things like that.

Think of the "SDK" as being the Microsoft Platform SDK + DDK plus some internal header files. It has around 14K source files (mostly .h). I created a tag file with just the "SDK" content and its size is about 435MB.

The project itself is formed by a bunch of smaller sub-project. Right now I have it all as one project under one workspace, with a total of about 52K files. Its tag file is hitting the 2GB limit.

As you can see, all in all it's not a whole lot of files. I don't recall messing with anything tag related in the options menu, so I think I'm using the default config for that. Since I'm blocked on this right now, I'm going to go ahead and delete all the tag files for this workspace and see if that helps. If it does, I'll recreate the tag files, this time being careful to not have the project tag file include the SDK files (I was having problems with that before I experienced the crash).

Thanks.

[brunokc]

When you say you "tagged all the files", I assume that means you recursively added "*.*" to the project.  How about if you trim that down to just the source files, like "*.cpp;*.h", for example.

I used "*.c; *.cpp; *.cxx; *.h; *.hpp; *.hxx; *.idl; *.inc; *.mk; *.txt". There are not a whole lot of inc, mk and txt files in the projects, but I'm now wondering what SE is doing with those as far as tagging is concerned.

Also, I'm guessing that if you have a 2GB tag file you are probably tagging a lot of stuff that is irrevelant, or in many ways, worse yet, duplicate versions of the same code. 

I think I'm still duplicating the SDK files into the project tag file. I'm looking into this right now (I'm rebuilding both tag files).

What programming language are you working with?  Is there embedded code (such as assembly embedded in C)? 

This is mostly C++ with some plain C involved as well. There's a bit of assembly embedded in C, yes.

You can delete your workspace history by removing vrestore.slk from your SlickEdit configuration directory, or since you are basically working in a default configuration, just go ahead and delete the entire configuration directory and then start from scratch. 

I'll try to remove the tag files individually first and will try the "blow everything" approach next. I was hoping SlickEdit would be interested in more information about the crash though so they could fix it.

Hone your project down to what you really need to be working on.  You can create several different workspaces and easily switch between them if you need to work on different versions of the code or different branches of the source tree.

There are some cross references between the various sub-projects and I was hopping I could get symbol look up and autocompletion working throughout the whole source tree. Will I get that if I break things down into smaller projects/workspaces?

For large projects what do people usually do: create multiple projects inside the same workspace; or create various workspaces with a single project inside each one; or perhaps something in between?

Finally, could you create a mini-dump the next time you get the crash?  It is very easy to do by simply attaching to SlickEdit with Visual Studio.

I can currently re-create the crash at will. My problem with creating a mini-dump is what it will contain. I guessing it will probably contain information about our projects and I'm pretty sure my employer would not be happy about that. Is there anything else I can provide?

Thanks.

[Phil Barila]

You'll get better results if you add the SDK and DDK as separate compilers.  You'll probably need to add each of them as a new compiler and point SE at the include dir(s).  See "Tools > Options > Languages > Application Languages > C/C++ > Compiler Properties".

[Dennis]

With respect to creating a mini dump, that only contains an accurate stack trace, no heap data or local variables.

Yes, we are interested in debugging the crash, but also interested in getting you into a state where you are able to effectively use the product.  Based on what you said about the number of files you have, your tag file should not be nearly that big.  My guess is you have a lot of duplicate code tagged in your workspace.

[brunokc]

I'll look into getting you a mini-dump. Assuming I can get you one, what's the best way to send that over to you? Attach to the thread? Use skydrive/dropbox/etc?

I've re-created my workspace, this time without the SDK files (I clicked the "add tree" button, recursive, added those extensions I listed above, checked the "Add as Wildcard" checkbox and added the SDK folder as an exclusion). Background tagging is going slowly but surely. It still says 56K+ files to go, and Tools -> Tag Files says there are 12K files in the tag file -- is it still picking up the SDK files? We'll see after it's done.

At this point, with 56K+ files left to be tagged, the tag file is >680MB in size already.

I should add that the UI still locks up very easily while tagging is running. The status bar at the bottom of the screen goes blank when I try to interact with SE, but the UI quickly becomes unresponsive. Looking at the process state under windbg, it looks like SE is performing non-message-pumping waits on the UI thread:

Code: [Select]

0:000> kn
 # Child-SP          RetAddr           Call Site
00 00000000`00186958 000007fe`fe3e10dc ntdll!ZwWaitForSingleObject+0xa
01 00000000`00186960 00000000`0f775900 KERNELBASE!WaitForSingleObjectEx+0x79
02 00000000`00186a00 00000000`0f7767da tagsdb!tag_restart_async_tagging+0x5d00
03 00000000`00186a30 00000000`0f7ae76d tagsdb!tag_restart_async_tagging+0x6bda
04 00000000`00186a60 00000000`0f74926d tagsdb!tag_restart_async_tagging+0x3eb6d
05 00000000`00186b10 00000000`0f7494db tagsdb!vsTagOpenDB+0x1cd
06 00000000`00186b40 00000000`0fdb99d4 tagsdb!tag_open_db+0xb
07 00000000`00186b70 00000000`0fdba99f vsapi!vsNameDllAddr+0x1644
08 00000000`0018b0a0 00000000`0fdb4015 vsapi!vsNameDllAddr+0x260f
09 00000000`0018b0f0 00000000`0fdb4134 vsapi!vsError+0xe5
0a 00000000`0018b130 00000000`0fd53af4 vsapi!vsError+0x204
0b 00000000`0018b180 00000000`0fc1d9c6 vsapi!vsCallPtr+0x194
0c 00000000`0018b600 00000000`66864af0 vsapi!vsQTSlickCExecuteKeyEvent+0x1696
0d 00000000`0018b640 00000000`64b0ea03 QtCore4!QObject::event+0x3f0
0e 00000000`0018b840 00000000`64ab50ca QtGui4!QWidget::event+0xf93
0f 00000000`0018ba30 00000000`64ab4d68 QtGui4!QApplicationPrivate::notify_helper+0xda
10 00000000`0018ba60 00000000`6684f512 QtGui4!QApplication::notify+0x1ad8
11 00000000`0018bdd0 00000000`6685010a QtCore4!QCoreApplication::notifyInternal+0x72
12 00000000`0018be30 00000000`668812ed QtCore4!QCoreApplicationPrivate::sendPostedEvents+0x28a
13 00000000`0018bec0 00000000`77719bd1 QtCore4!winGetMessage+0x32d
14 00000000`0018bf60 00000000`777198da USER32!UserCallWinProcCheckWow+0x1ad
15 00000000`0018c020 00000000`6688214d USER32!DispatchMessageWorker+0x3b5
16 00000000`0018c0a0 00000000`64b2e385 QtCore4!QEventDispatcherWin32::processEvents+0x3ed
17 00000000`0018f3d0 00000000`6684d825 QtGui4!qt_getRegisteredWndClass+0x465
18 00000000`0018f400 00000000`6684fa10 QtCore4!QEventLoop::exec+0x215
19 00000000`0018f480 00000001`3f2826cb QtCore4!QCoreApplication::exec+0xe0
1a 00000000`0018f4d0 00000001`3f2884ea vs!slickedit::SEListTagsTarget::setFileEncoding+0xee2b
1b 00000000`0018f780 00000001`3f2c3f64 vs!vspsGetEnv+0xeba
1c 00000000`0018f8f0 00000000`7781652d vs!vspsGetEnv+0x3c934
1d 00000000`0018f9a0 00000000`77cac521 kernel32!BaseThreadInitThunk+0xd
1e 00000000`0018f9d0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

Thanks.

[Dennis]

You can attach a mini-dump by zipping it up and attaching it to this thread.

[brunokc]

Here are the mini-dumps. One for the crash and one for the UI hang.

Thanks,
Bruno.

[Dennis]

I have reviewed both mini-dumps.

I may have a solution for the stall problem.  I'm going to look into that for our 17.0.2 release.

I suspect that the crash is on account of a corrupt tag database.  If you killed SlickEdit when it was stalled, the tag database was probably left in an invalid state.  If you ever kill slickedit or it crashes while building a tag database, it is good practice to delete the corrupt tag database before restarting SlickEdit.

[brunokc]

I may have a solution for the stall problem.  I'm going to look into that for our 17.0.2 release.

Good to hear that, Dennis. Thank you for looking into it. Let me know if you'd like me to experiment with any potential fix you may have in the future around this issue.

I suspect that the crash is on account of a corrupt tag database.  If you killed SlickEdit when it was stalled, the tag database was probably left in an invalid state.  If you ever kill slickedit or it crashes while building a tag database, it is good practice to delete the corrupt tag database before restarting SlickEdit.

That's very likely in my case. I did kill SE a couple times as I was waiting for it to cancel tagging and close. I'll make a note of this for future repro attempts. Right now I've created a new workspace/project and added just 2 of the folders I'm most actively working on right now and SE is behaving much better.

I should note that I found out there's a folder with a large number of huge header files that was most likely causing the size of the tag file to hit the 2GB limit and causing all sorts of UI hangs and SE slowdowns. Check my post here for more information: http://community.slickedit.com/index.php/topic,8552.msg36539.html#msg36539.

Thanks again!
Bruno.