SlickEdit Community

SlickEdit Product Discussion => SlickEdit® => Topic started by: IkerAriz on November 06, 2017, 04:28:51 pm

Title: v22 memory leak?
Post by: IkerAriz on November 06, 2017, 04:28:51 pm
Just upgraded to v22 and memory consumption seems to have gone up from v21. Details:
I used "top" to keep an eye on se's RES during those three hours and watched it steadily grow from 190m to >700m. I stopped watching closely after that, but growth continues to well over 1GB over the course of the day (I just restart when I notice it that high).

Is there any way to help pin down where SE is using the memory?

Regards,
Iker
Title: Re: v22 memory leak?
Post by: guth on November 06, 2017, 10:26:33 pm
I have a similar experience. I had vs open, under Windows, for, maybe, two days. A Python project but mainly editing an HTML file with javascript. Did a lot of moving focus to the web browser and back. Noticed that vs grew a couple Mb every time alt-tabbing back. Attached image shows process sizes of two vs processes. One reached over 4 Gb with a handful of smallish source code files.
Title: Re: v22 memory leak?
Post by: JeffB on November 07, 2017, 06:46:13 pm
I'm pretty sure I see a leak as well (Linux).  I had one instance at >5GB today.  Killed it and it came back with half that.  I had one up for several days last week and it eventually went >10GB.  Restarted it and it came back with 1-2GB (can't remember).
Title: Re: v22 memory leak?
Post by: rowbearto on November 08, 2017, 03:41:28 pm
I think I'm seeing this too in Linux. A fresh SE restart and reloading my workspace uses 475K (with no editor windows open). But this morning after running over a day I was at 2.5GByte (but did have many editor windows open).

I'm doing a very long build now that fills up the process buffer, and memory is constantly growing, probably expected due to memory for the process buffer? Perhaps when I was at 2.5GByte my process buffer had too much?
Title: Re: v22 memory leak?
Post by: patrick on November 08, 2017, 04:50:53 pm
We're looking at some valgrind output, and we have found one leak.  Still checking to see if that's everything, but it's not a speedy process. It's like running the editor in molasses when I've got everything turned on for full leak checking.  Will update with more information once I have it.
Title: Re: v22 memory leak?
Post by: Clark on November 15, 2017, 04:20:30 pm
One of these will fix this:

For 64-bit windows version do this:

* Backup your vsapi.dll
* Replace it with this:
    http://support.slickedit.com/Outbound/2200/vsapi_22000009.dll (http://support.slickedit.com/Outbound/2200/vsapi_22000009.dll)

For 64-bit Linux version do this:

* Backup your vs_exe
* Replace it with this:
    http://support.slickedit.com/Outbound/2200/vs_exe_22000009_linux64.bin (http://support.slickedit.com/Outbound/2200/vs_exe_22000009_linux64.bin)

For Mac version do this:

* Backup SlickEditPro2017 installation. A simple rename will do this trick.
* Download and install this:
    http://support.slickedit.com/Outbound/2200/se_22000009_mac_2017_11_15.dmg (http://support.slickedit.com/Outbound/2200/se_22000009_mac_2017_11_15.dmg)
Title: Re: v22 memory leak?
Post by: IkerAriz on November 16, 2017, 08:05:49 pm
Thanks Clark - that definitely helped! However, it appears a smaller leak may remain (at least under ubuntu 16). Using the same setup as my earlier report "RES" memory grew from an initial 190 MB to 370MB over the course of ~6 hours. I'll keep an eye on it and report if it grows further.

Iker
Title: Re: v22 memory leak?
Post by: patrick on November 16, 2017, 08:24:27 pm
All other things being equal, it will grow a bit at first as the context tagging and other caches get populated through use, and then should level off.  If there's a remaining leak, it would be something that we didn't exercise when running instrumented, and we'll probably have to get some details from you about what languages are in your project, etc...   Thanks for the update.
Title: Re: v22 memory leak?
Post by: IkerAriz on November 16, 2017, 08:57:33 pm
Here's some info about my project:
I've attached my current tagging options as well.

Regards,
Iker
Title: Re: v22 memory leak?
Post by: IkerAriz on November 16, 2017, 09:00:22 pm
Follow up.

In the time between my reply to Clark and now memory consumption has grown to 522 MB.

Iker
Title: Re: v22 memory leak?
Post by: Dennis on November 16, 2017, 09:51:42 pm
We do a lot of caching, so it is normal for memory to grow to an extent.  If memory usage exceeds 2G in a day with a project your size, then perhaps there still is a lingering problem.
Title: Re: v22 memory leak?
Post by: JeffB on November 17, 2017, 06:22:55 pm
This seems to work for me as best I can tell.  I haven't seen an instance go >3GB yet.
Title: Re: v22 memory leak?
Post by: jporkkahtc on November 17, 2017, 07:55:05 pm
WRT Caching: I did a FindFile to look for *.X files in a collection of ZIP archives.
Then from Explorer I tried deleting some of these files, but delete failed because Slick was holding them open (even though it had no buffers open for them).

Using PROCEXP I could see Slick holding handles to these files, and releasing them one at a time.
So I happened to see some Slick caching in action - it seems to hold *.ZIP files open for some short time, then closes them.
Title: Re: v22 memory leak?
Post by: JeffB on November 17, 2017, 08:49:14 pm
Spoke to soon.  The instance at 3GB a little while ago is now close to 5GB.  I can load files and do references and every now and then it'll jump a few hundred MB, but I don't see a pattern.

This is a large project ~200k files.  Tag file is ~2.5GB (which is also the Tag file cache maximum)
Title: Re: v22 memory leak?
Post by: Dennis on November 17, 2017, 08:54:45 pm
Does that 5GB include memory allocated to memory mapped files?   Tag files are memory mapped, unless you have them configured otherwise.  With a tag file cache of 2.5G, it is conceivable that you got to 5G rather organically just by filling up the tag file cache, in addition to memory allocated to other memory mapped tag files and other caching and normal memory usage.
Title: Re: v22 memory leak?
Post by: JeffB on November 17, 2017, 10:01:45 pm
I assume it does not.  FYI....The Virtual Size is just over 6GB.

I'll leave it up for a while and see if it keeps growing.
Title: Re: v22 memory leak?
Post by: JeffB on November 22, 2017, 05:54:57 pm
FYI...After sitting for a few days with only minor usage, the resident memory is 6.35GB and Virtual is 7.5GB.
Title: Re: v22 memory leak?
Post by: IkerAriz on November 27, 2017, 07:18:00 pm
Additional follow up...

Total RES memory for my VS process is now a bit over 1GB and continues to grow, albeit slowly. A quick peek at the "/proc/pid/smaps" file suggests most of that is heap (see output below).

Regards,
Iker

Code: [Select]
02f18000-3b924000 rwxp 00000000 00:00 0                                  [heap]
Size:             927792 kB
Rss:              927084 kB
Pss:              927084 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:    927084 kB
Referenced:       920808 kB
Anonymous:        927084 kB
AnonHugePages:    864256 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr ex mr mw me ac sd
Title: Re: v22 memory leak?
Post by: patrick on December 06, 2017, 07:03:20 pm
Thanks for the updates.  We found the leak that probably accounts for your continued resident memory growth, and it's been fixed for the point release.

For our current tests, the leak was slow enough where it got lost in the noise of memory we keep in caches.  But editing large source files, or files that are really tag dense (like XML files) can make the leak fast enough to observe just watching the OS reports on memory usage.  Just running for a long time with normal usage will leak as well, just not as quickly.
Title: Re: v22 memory leak?
Post by: IkerAriz on December 07, 2017, 09:48:12 pm
Great to hear! Thanks for the follow up.

Regards,
Iker

Title: Re: v22 memory leak?
Post by: IkerAriz on January 22, 2018, 04:43:50 pm
Hi Patrick,

I switched to version 22.0.1 and while the memory leak seems to have slowed down it's still present (eg, 1GB+ heap after a few days). It also appears that as memory use grows editing becomes more janky/less responsive: cursor movement and scrolling stagger, noticeable delays in autocomplete, etc.

Are there any diagnostics I can enable to help track the issue?

Regards,
Iker
Title: Re: v22 memory leak?
Post by: JeffB on January 22, 2018, 05:38:08 pm
FYI...
The leak seems to be gone for me.  I have noticed unusual delays and extreme lapses in response, but I'm not sure if that's related only to Python files or not.

Jeff
Title: Re: v22 memory leak?
Post by: patrick on January 23, 2018, 03:19:28 pm
@IkerAriz, probably the best option is to turn on Slick-C profiling when it starts getting janky, do whatever operations are showing delays for you for a few minutes, and then turn it off and send us the profile.  If there is a relation between the memory use and the jankiness, then that could point us in the right direction. You can access the profiler controls in the Macro -> Slick-C Profile submenu.  In the meantime, I'll re-run some of the memory tests with a project setup like you described, to see if I can spot anything I've missed.

I see from one of your previous posts that background tagging is disabled, so there are a few things that can block the main thread with that.  After the profiling, it narrow things further to see if that makes a difference. 
Title: Re: v22 memory leak?
Post by: IkerAriz on January 30, 2018, 06:34:56 pm
Hi Patrick,

I enabled background tagging and UI performance didn't get better (AFAICT) but memory consumption did appear to grow faster. Attached is a slick-c profile (BG tagging enabled) that captures a few examples of significant latency.

Thanks,
Iker
Title: Re: v22 memory leak?
Post by: patrick on January 30, 2018, 06:51:25 pm
Thanks, looking at the profile. 

Quick question: do you have C# files in your project, or were you editing any? 
Title: Re: v22 memory leak?
Post by: IkerAriz on January 30, 2018, 07:36:08 pm
My project has 4 C# files but I was not editing them (the files are part of the project but they're practically never touched). At the time I produced the profile I was editing a C++ file.
Title: Re: v22 memory leak?
Post by: Dennis on January 30, 2018, 08:34:57 pm
Looking over your profiling results also.  My suggestion to eliminate the hiccups.  Document > C/C++ Options > Auto-Complete > Auto-list compatible values > OFF (uncheck).  Also turn off Document > C/C++ Options > Context Tagging > Auto-list compatible parameters.

While nice to have, these two features require a lot of calculation, especially with templates, which I can see from your profiling data that you have a lot of.

You can manually force SlickEdit to list compatible values by hitting Ctrl+,

Title: Re: v22 memory leak?
Post by: IkerAriz on February 02, 2018, 04:56:04 pm
Thanks Dennis. Disabling those two features improved responsiveness. However, the memory issue remains with heap consumption regularly exceeding 1GB.

BTW, I noticed something that may be of interest. I recently added another C++ project to my workspace with ~1200 files and made it inactive (ie, my first project with ~3800 files is the active one). I quit VSE and then went through the following steps:
When I make the new project "active" and repeat the steps above the big memory jump happens in step 4.

Iker
Title: Re: v22 memory leak?
Post by: IkerAriz on February 15, 2018, 12:46:54 pm
After disabling "Auto-list compatible values", "Auto-list compatible parameters" and "C/C++->auto-complete" typing latency is still significant; in particular, when hitting an opening parentheses, a member access period or arrow.

Attached is another profile that captures latency with this configuration. Note that heap memory for this instance of VSE is nearly 1.4GB (after a couple of days).

Title: Re: v22 memory leak?
Post by: Dennis on February 19, 2018, 08:07:06 pm
Are you using Boost, or heavily using the STL in your code?
Title: Re: v22 memory leak?
Post by: IkerAriz on February 20, 2018, 02:02:11 pm
No boost. But the project does make liberal use of the more common STL types (string, vector, list, map, etc).
Title: Re: v22 memory leak?
Post by: IkerAriz on March 01, 2018, 12:22:22 am
Any luck with the latest profile? Is there other info I can provide to help with the debugging?

Iker
Title: Re: v22 memory leak?
Post by: Dennis on March 01, 2018, 04:25:27 pm
I was not able to gather much from that profile.  I can say that STL context tagging is tough.  If you ever try to manually trace down what actually happens just to do something like: vector v [ x ] -> <list symbols>, you would have a long day of work before you untangled it all, if you were able to at all.  It is just a maze of typedefs and allocator templates, nothing goes directly from point A to point B.  And now we also have to do type inference with that.  We have some shortcuts in there to make some things faster, but perhaps we need to do even more.  We have another outstanding complaint that tagging is slow with Visual Studio 2017 headers.  This happens occasionally, compiler vendors change their STL implementation, and it seems to always adds more complexity.

One thing you can do to make things easier, is to make sure that you do not have duplicate STL implementations tagged in your workspace.  Set up and use your compiler configuration tag file, and do not tag another set of compiler headers in addition.  It is hard enough for tagging to untangle this mess, and even harder if there are multiple STL implementations to disambiguate.
Title: Re: v22 memory leak?
Post by: IkerAriz on March 07, 2018, 10:05:49 pm
Here are my configuration changes thus far:
Editing feels faster, if a bit more spartan. There's also noticeable delay when hitting "alt-," to manually trigger function argument help (though this is less of a problem in practice). Memory consumption, surprisingly, appears to have worsened. My reduced workspace and and trimmed tag files are both considerably smaller than they were at the start of this thread yet heap memory on startup (with no files open) is ~230 MB compared to the earlier 175 MB. Heap consumption after startup grows continuously as before.

Thanks,
Iker

Title: Re: v22 memory leak?
Post by: IkerAriz on May 11, 2018, 02:26:11 pm
Any news on this front?

Iker
Title: Re: v22 memory leak?
Post by: Dennis on May 14, 2018, 02:32:10 pm
@IkerAriz:  What is your tag file cache set to?  You can, I do not entirely recommend it, but you can set it as low as 8M for a maximum tag file cache size.  You can also turn off the option to use memory mapped file I/O to further reduce the tag file caching footprint.  The memory footprint numbers you are reporting are in my opinion pretty normal amounts.  With most computers having at 8G plus of memory, we are trying to improve performance by caching in a lot of areas in the editor, so it is normal for the memory footprint to grow over time.  However, the tag file maximum is one way that you can absolutely reduce the footprint, if you are willing to sacrifice a certain amount of performance.
Title: Re: v22 memory leak?
Post by: IkerAriz on May 15, 2018, 01:48:39 pm
Dennis,

Both "tag file cache size" and "tag file cache maximum" are set to 64 MB.

My concern is not that SE consumes a lot of memory per se but that I have so little control over, or insight into, how it's used for my benefit. I've disabled several tag-related features (as you recommended), removed many files from my project, trimmed tagged includes (STL), and reduced cache sizes to no avail. Memory use slowly but surely grows past 1 GB and beyond while editing a fairly small set of files (that's 1 GB of private, dirty memory). If all of my project's source code amounts to less than 100 MB what is that memory being used for? And why does it appear to grow without stopping? (*). 

Regards,
Iker

(*) Perhaps it does stop eventually but I restart SE once it hits ~1.5 GB.
Title: Re: v22 memory leak?
Post by: IkerAriz on May 22, 2018, 01:18:37 pm
Does SE provide any diagnostic options/flags that can shed light on memory use?

Iker
Title: Re: v22 memory leak?
Post by: IkerAriz on June 09, 2018, 08:11:39 pm
To try to reproduce the behavior I reported earlier I created a smaller project using source from the tinyxml project (see attached) and ran through a few tests. Here's how it went:

The memory growth here is smaller than in my larger project but appears similar in nature. Attached is a screenshot as well.

I'm running on Xubuntu 16.04. My SE tagging configuration is the same as I reported previously.

Regards,
Iker

Title: Re: v22 memory leak?
Post by: Dennis on June 11, 2018, 02:30:46 pm
I'll run through your steps to reproduce later this week and see what comes up.  I'm going to run with a default configuration to start out.  If I cannot reproduce the leak with that I am going to need to have your user.cfg.xml.
Title: Re: v22 memory leak?
Post by: IkerAriz on June 13, 2018, 02:29:05 pm
Attached is my user.cfg.xml file.

Thanks,
Iker
Title: Re: v22 memory leak?
Post by: Dennis on June 19, 2018, 12:43:14 am
OK.  I've spent about a half week on this.  Good news and bad news.

1 - Good news)  I've identified, found and fixed a few memory leaks.  These fixes will be incorporated into the next release.

2 - Bad news - or Jedi mind trick)  These are not the leaks you are looking for.  It is unlikely that what I found is what brought our SlickEdit memory footprint up to 2G.  They were small leaks.

3 - resizing and drawing)  Not leaks.  We have a fairly significant line drawing cache.  You are merely filling up the cache.  The test project has a bit over 6000 lines of code, the cache holds 20000 lines before it fills and purges.  This is just using memory, not leaking it.

4 - delete and undo)  I did not observe a significant memory increase here, and the memory leak tracking software did not identify anything as leaked as a result of this test.

5 - searching) same story, yes, first time use of a feature can uptick memory somewhat, when the form is loaded the first time, etc, but no definitive leaks identified.

6 - segmenting)  We have a our own memory allocator for speed.  Like any allocator, it can fall victim to some segmenting depending on usage patterns.  Some of the memory growth you see could be segmenting, well, it will appear as growth initially, but once there is enough memory in use, the growth will level off entirely.  Note, this can happen with the system malloc() too, so there is nothing particularly wrong with our allocator, it's just faster than malloc.

7 - scrolling) see (3)

8 - caches) we have a lot of caches.  Tagging especially has a lot of caching going on.  It is one of the few things that gives you some limited control over the caches, yet, even there, there are some caches that will just grow as needed.  You can't make a setting for everything.  I have suggested ways to cut back the tagging cache earlier in this thread.

9 - your config)  You were loading a mini-map tool that I did not have access to.  Hence, this did not factor into any of my testing.

Hope this helps.  I'm still digging around for leaks, as I work.  Maybe something will surface eventually.
Title: Re: v22 memory leak?
Post by: IkerAriz on June 19, 2018, 01:10:23 pm
Thanks for the follow up Dennis.

Any way to collect additional memory info during my tests? Eg, are there any diagnostic commands that can print the SE cache sizes and and/or allocator stats? (perhaps via a custom build?)

Thanks again,
Iker

P.S. BTW, the minimap command just toggles the zoom (I didn't use it during my tests). Here's the code:

Code: [Select]
int minimap_on = 0;
_command void minimap_toggle () name_info(',') {
   if (minimap_on) {
       wfont_zoom('9');
       minimap_on = 0;
   }
   else {
      wfont_zoom('4');
      minimap_on = 1;
   }
}

Title: Re: v22 memory leak?
Post by: Dennis on June 19, 2018, 02:42:51 pm
Sorry, there's nothing built in that would provide any really helpful information.  We do have some allocator stats that we occasionally use in debug and unit test builds internally, but they would just provide some stats, nothing of help for tracing down where the memory is going primarily.
Title: Re: v22 memory leak?
Post by: IkerAriz on June 19, 2018, 07:26:17 pm
Any other options I can pursue to help find the problem?

Iker
Title: Re: v22 memory leak?
Post by: Dennis on June 19, 2018, 10:20:33 pm
I found another memory leak today.  Between Patrick and myself, we have cleaned up quite a few for v23.  None of them were showstoppers, but hopefully v23 will run a bit more trim than v22 for you.  I would encourage you to participate in the beta for v23 when that time rolls around.  That's about all you can do for now.
Title: Re: v22 memory leak?
Post by: Clark on June 19, 2018, 11:08:56 pm

Code: [Select]
int minimap_on = 0;
_command void minimap_toggle () name_info(',') {
   if (minimap_on) {
       wfont_zoom('9');
       minimap_on = 0;
   }
   else {
      wfont_zoom('4');
      minimap_on = 1;
   }
}

Heads up. There will very likely be a minimap_toggle command on the view menu in v23. If you define a command with the same name (like above) the original minimap_toggle command will be replaced with yours. You should change the name of yours.
Title: Re: v22 memory leak?
Post by: IkerAriz on June 20, 2018, 02:18:57 pm
Thanks again. Looking forward to the v23 beta.

Iker

P.S. Any chance v23 can include some cache and/or allocator diagnostics to help users provide more detailed reports for issues like these?