High CPU utilization under linux. High memory utilization and crash seen in VM.

[cvavruska]

After I launch slickedit 22.0.1.0 I am seeing really high CPU utilizations (95-100%). I have no buffers open, no workspace open.  This is constant. I have run the run-time profiler and attached the tsv. My home directory is nfs mounted and I am running a really old kernel (CentOS 6.1 2.6.32) due to our IT dept not wanting to upgrade due to clearcase. I guess they think if it ain't broke don't upgrade it.
On the same machine I have a VirtualBox VM (CentOS 7.4) with 8G allocated to it running 22.0.2.1 (clearcase running but not used). It has the same NFS mounted home dir. I am not seeing the high CPU utilization but if I load up a workspace with ~8300 files the tagging says it completes pretty quickly but then it continues with high CPU and memory utilization until it finally crashes.

Any help would be appreciated.

[patrick]

For your main system, try starting SlickEdit with the "-sul" option on the command line, and see if that makes a difference for you. You can try it on the VM as well, but since it crashed there is a risk that the tag files were left in an incomplete state, so I'd remove the .vtg files before restarting it.

For the crash on the VM, did it produce a core file?  The ulimit may be set so there isn't a core dump, so in that case, you'd need to set the core ulimit to unlimited and then reproduce the crash again to get a core file. Where the core dump ends up also depends a bit on system configuration.  The usual default used to be the working directory of the process that crashed, but if your system has a crash reporter like apport set up, then the dump can end up as a file or directory in /var/crash.  Administrators can also set a custom location/name for core dumps in /etc/sysctl.conf of /etc/sysctl.d with the "kernel.core_pattern" variable, so checking for that can help locate the dump.

If you have a core file, and have gdb installed, the fastest thing you can get to us is stack trace of the crash.  As an example, if your core was "/tmp/core.234" and SlickEdit was installed at /opt/se22 then you would run "gdb /opt/se22/bin/vs_exe /tmp/core.234". At the gdb prompt, run "thread apply all bt", and copy and paste all of that output into a reply.  If you don't have gdb, we can PM you with instructions on how to upload the core to us.

[cvavruska]

the -sul didn't help the high utilization. I am guessing the utilization on the VM is based on what is eventually causing the abort.

Here is the backtrace on the second issue. Looks like it is running out of memory based on the coresize of 8.1G

Program terminated with signal 6, Aborted.
#0  0x00007fd9f4c081f7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install expat-2.1.0-10.el7_3.x86_64 fontconfig-2.10.95-11.el7.x86_64 freetype-2.4.11-15.el7.x86_64 glib2-2.50.3-3.el7.x86_64 glibc-2.17-196.el7_4.2.x86_64 libICE-1.0.9-9.el7.x86_64 libSM-1.2.2-2.el7.x86_64 libX11-1.6.5-1.el7.x86_64 libXau-1.0.8-2.1.el7.x86_64 libXcursor-1.1.14-8.el7.x86_64 libXext-1.3.3-3.el7.x86_64 libXfixes-5.0.3-1.el7.x86_64 libXi-1.7.9-1.el7.x86_64 libXinerama-1.1.3-2.1.el7.x86_64 libXrandr-1.5.1-2.el7.x86_64 libXrender-0.9.10-1.el7.x86_64 libXt-1.1.5-3.el7.x86_64 libuuid-2.23.2-43.el7_4.2.x86_64 libxcb-1.12-1.el7.x86_64 pcre-8.32-17.el7.x86_64
(gdb) bt
#0  0x00007fd9f4c081f7 in raise () from /lib64/libc.so.6
#1  0x00007fd9f4c09a28 in abort () from /lib64/libc.so.6
#2  0x0000000001ce6b4d in __gnu_cxx::__verbose_terminate_handler() ()
#3  0x0000000001ce5426 in __cxxabiv1::__terminate(void (*)()) ()
#4  0x0000000001ce5471 in std::terminate() ()
#5  0x0000000001ce5578 in __cxa_throw ()
#6  0x00000000019e6dc9 in cmThrow(long long) ()
#7  0x00000000019edd1f in cmStringT<char, 1, 30>::createNewRefBuf(char const*, long long, long long, bool) ()
#8  0x00000000019ee023 in cmStringT<char, 1, 30>::replaceRefBuf(char const*, long long, long long) ()
#9  0x00000000019f08fd in cmStringT<char, 1, 30>::_append(char const*, long long, int) ()
#10 0x0000000001107f28 in CPPDefine::expandDefine(cmStringT<char, 1, 30>&, cmArray<cmStringT<char, 1, 30> > const&) const ()
#11 0x00000000009ea68e in CPPLexer::maybeExpandMacro(cmStringT<char, 1, 30> const&, cmArray<CPPDefine> const&, bool) ()
#12 0x00000000009ec488 in CPPLexer::maybeExpandMacro(SETokenType) ()
#13 0x000000000145659a in TBPLexer::getNextToken() ()
#14 0x00000000010d791e in SETaggingLexer::getNextToken() ()
#15 0x0000000001476399 in TBPTokenSource::nextTokenObj() ()
#16 0x00000000009eec55 in CPPLexer::nextTokenObj() ()
#17 0x00000000009e4e39 in CPPLexer::appendIDThenNextToken(slickedit::SEString*) ()
#18 0x0000000000a1d106 in CPPParser::skipUntilRightBrace(bool, slickedit::SEString const&, slickedit::SEString const&, slickedit::SEString*)
    ()
#19 0x0000000000a3cfae in CPPParser::parseIdentifierDeclaration(SETagType, slickedit::SEString const&, slickedit::SEString const&, SETagFlags, int, slickedit::SEString const&, slickedit::SEString const&, int, int, bool, bool, slickedit::SEString*, slickedit::SEString const&, int*, int, CPPTagInterface*, int, bool) ()
#20 0x0000000000a4e617 in CPPParser::parseGlobalDeclarations(slickedit::SEString const&) ()
#21 0x0000000000a556b1 in CPPParser::parseProgram() ()
#22 0x00000000009c7174 in cpp_list_tags_callback(slickedit::SEListTagsTarget&) ()
#23 0x00000000014bfb50 in slickedit::SEListTagsTarget::parseFileForTags() ()
#24 0x00000000014dc6ae in slickedit::SEListTagsThread::runAsTagger() ()
#25 0x0000000001abb727 in cmThread::ThreadStartRoutine(void*) ()
#26 0x00007fd9fccefe25 in start_thread () from /lib64/libpthread.so.0
#27 0x00007fd9f4ccb34d in clone () from /lib64/libc.so.6

[patrick]

Hmm.  That crash is reminiscent of a bug that was fixed for 22.0.2, so I'm wondering if this is a related issue that wasn't caught.  Give this a try.  Restart SlickEdit on the VM, bring up the options dialog via Tools -> Options, navigate to Languages -> Application Languages -> C/C++ -> C/C++ Parsing Options.   Turn off the "Dynamically expand local #defines" setting.  Exit out of SlickEdit, delete any tag files, start it back up, and let it tag and see if it still crashes. 

For the high CPU, is perftools available on that system?  If so, I can probably put together a "perf record" and "perf report" command lines for you to run that could give us an idea of where it's spinning.

[cvavruska]

Turning off the "Dynamically expand local #defines" seems to have fixed the crash issue in the vm

I do have perftools installed on my system. My thought is it must have something to do with either the old 2.6 kernel, clearcase or both.

[patrick]

For the VM crash, that may be a parsing bug with macros.  Did we luck up today, and that project you have open is some sort of open source project?

For the utilization problem, start up SlickEdit, and go to a terminal, and get the PID for vs_exe.  And then run "sudo perf record -g -p PID_FOR_VS_EXE".  Wait for 4 minutes, and then Ctrl-C the perf command.  That will generate a "perf.data" file.  You can exit SlickEdit now.

The easiest thing to do with this is to dump it as a text histogram that can just be posted.

• "chown" the perf.data file that got collected to your own user id.  To avoid running report as root.
• Run "perf report -g".  This will bring up a curses interface to view the data.
• Press 'E' .  (must be capital E).  This expands the call trees out.
• Press 'P'. (also capital) This dumps the calltree out to perf.hist.N, where N is probably 0
• The perf.hist.N file is just a text file, you can post it here.

[cvavruska]

Nope, private project. Did you really think it would be that easy?

Attached is the perf data for the high utilization issue.

[patrick]

Ok.  Can you try starting it like this, letting it settle down and see if you still have high cpu utilization:

Code: [Select]

QT_NO_GLIB=1 /path/to/install/bin/vs_exe +new

I can sometimes get a trace similar to yours on one of my systems, but it's very unreliable - doesn't always happen, and when it does, if I look at it funny, it will go back to normal cpu utilization.

[cvavruska]

That seems to have fixed the high utilization. It is now running at a respectable level.

[patrick]

Thanks.  I've got a couple of possibilities to look at now.  I'll let you know once I have something for you to try.