Author Topic: v22 memory leak?  (Read 17786 times)

IkerAriz

  • Senior Community Member
  • Posts: 159
  • Hero Points: 6
Re: v22 memory leak?
« Reply #30 on: February 20, 2018, 02:02:11 PM »
No boost. But the project does make liberal use of the more common STL types (string, vector, list, map, etc).

IkerAriz

  • Senior Community Member
  • Posts: 159
  • Hero Points: 6
Re: v22 memory leak?
« Reply #31 on: March 01, 2018, 12:22:22 AM »
Any luck with the latest profile? Is there other info I can provide to help with the debugging?

Iker

Dennis

  • Senior Community Member
  • Posts: 3954
  • Hero Points: 515
Re: v22 memory leak?
« Reply #32 on: March 01, 2018, 04:25:27 PM »
I was not able to gather much from that profile.  I can say that STL context tagging is tough.  If you ever try to manually trace down what actually happens just to do something like: vector v [ x ] -> <list symbols>, you would have a long day of work before you untangled it all, if you were able to at all.  It is just a maze of typedefs and allocator templates, nothing goes directly from point A to point B.  And now we also have to do type inference with that.  We have some shortcuts in there to make some things faster, but perhaps we need to do even more.  We have another outstanding complaint that tagging is slow with Visual Studio 2017 headers.  This happens occasionally, compiler vendors change their STL implementation, and it seems to always adds more complexity.

One thing you can do to make things easier, is to make sure that you do not have duplicate STL implementations tagged in your workspace.  Set up and use your compiler configuration tag file, and do not tag another set of compiler headers in addition.  It is hard enough for tagging to untangle this mess, and even harder if there are multiple STL implementations to disambiguate.

IkerAriz

  • Senior Community Member
  • Posts: 159
  • Hero Points: 6
Re: v22 memory leak?
« Reply #33 on: March 07, 2018, 10:05:49 PM »
Here are my configuration changes thus far:
  • Disabled the options as recommended below (auto compat values, auto compat params, etc)
  • Removed all STL from the compiler tag files
  • Trimmed the compiler tag file to include only C files under /usr/include; ie, top level files, python C api headers, the gcc C headers (the "sys" and "asm" dirs under "x86_64-linux-gnu") plus a few others.   
  • Removed two C++ projects from my workspace (eigen and kaldi); workspace now has only one project (my own)
Editing feels faster, if a bit more spartan. There's also noticeable delay when hitting "alt-," to manually trigger function argument help (though this is less of a problem in practice). Memory consumption, surprisingly, appears to have worsened. My reduced workspace and and trimmed tag files are both considerably smaller than they were at the start of this thread yet heap memory on startup (with no files open) is ~230 MB compared to the earlier 175 MB. Heap consumption after startup grows continuously as before.

Thanks,
Iker


IkerAriz

  • Senior Community Member
  • Posts: 159
  • Hero Points: 6
Re: v22 memory leak?
« Reply #34 on: May 11, 2018, 02:26:11 PM »
Any news on this front?

Iker

Dennis

  • Senior Community Member
  • Posts: 3954
  • Hero Points: 515
Re: v22 memory leak?
« Reply #35 on: May 14, 2018, 02:32:10 PM »
@IkerAriz:  What is your tag file cache set to?  You can, I do not entirely recommend it, but you can set it as low as 8M for a maximum tag file cache size.  You can also turn off the option to use memory mapped file I/O to further reduce the tag file caching footprint.  The memory footprint numbers you are reporting are in my opinion pretty normal amounts.  With most computers having at 8G plus of memory, we are trying to improve performance by caching in a lot of areas in the editor, so it is normal for the memory footprint to grow over time.  However, the tag file maximum is one way that you can absolutely reduce the footprint, if you are willing to sacrifice a certain amount of performance.

IkerAriz

  • Senior Community Member
  • Posts: 159
  • Hero Points: 6
Re: v22 memory leak?
« Reply #36 on: May 15, 2018, 01:48:39 PM »
Dennis,

Both "tag file cache size" and "tag file cache maximum" are set to 64 MB.

My concern is not that SE consumes a lot of memory per se but that I have so little control over, or insight into, how it's used for my benefit. I've disabled several tag-related features (as you recommended), removed many files from my project, trimmed tagged includes (STL), and reduced cache sizes to no avail. Memory use slowly but surely grows past 1 GB and beyond while editing a fairly small set of files (that's 1 GB of private, dirty memory). If all of my project's source code amounts to less than 100 MB what is that memory being used for? And why does it appear to grow without stopping? (*). 

Regards,
Iker

(*) Perhaps it does stop eventually but I restart SE once it hits ~1.5 GB.

IkerAriz

  • Senior Community Member
  • Posts: 159
  • Hero Points: 6
Re: v22 memory leak?
« Reply #37 on: May 22, 2018, 01:18:37 PM »
Does SE provide any diagnostic options/flags that can shed light on memory use?

Iker

IkerAriz

  • Senior Community Member
  • Posts: 159
  • Hero Points: 6
Re: v22 memory leak?
« Reply #38 on: June 09, 2018, 08:11:39 PM »
To try to reproduce the behavior I reported earlier I created a smaller project using source from the tinyxml project (see attached) and ran through a few tests. Here's how it went:
  • Open SE with "leak" project. Project "folder view" mode is "directory view". Open tabs for all 6 files in project (see screenshot). Memory ~107 MB.
  • Click from one tab to another several times in effort to get SE to populate its caches. Mem at ~110 MB.
  • Bring tab for "tinyxmlparser.cpp" to the front.
  • Resize the window using lower right handle. Mem grew to 113.94 MB.
  • Leave SE alone for ~2 hours (walked away from computer).
  • Check memory on return. Memory unchanged at 113.94 MB.
  • Go back to resizing window with lower right handle (same tab is still showing). By "resize" I mean left click on the lower-right resize handle, and continuously drag to make the window smaller and bigger.
  • Resizing of the SE window reliably increases SE's RAM usage. The size of the increase appears to be proportional to the amount of resize activity. Eg, if a resize for a second or two the increases can be as small as 50K. If I drag around for several seconds I see jumps as high as 8000K.
  • Try simply moving the window, without resizing it. Ie, right client on title bar and drag window around. No change in RAM.
  • Move an unrelated window in fron of SE and drag it around. No change in RAM.
  • Leave SE window size fixed and go back to just cycling through the six open SE tabs (click on first, then second, third, and so on). Many cycles increase RAM by 40-100K, but not all. Note: I'm not clicking on the editor buffers at all - just clicking on the tabs.
  • Close all 6 project files with Ctrl-W then reopen them all by double clicking them in the projects window. Repeat many times. Most but not all close-open cycles (of the 6 files)  results in an increase of 100-1500K.
  • Click on "tinyxmlparser.cpp" tab. Grab scroll bar and scroll to the top of the file and back to the bottom. Repeat many times. Most up-downs increase RAM by 200-300K.
  • Click on "tinyxmlparser.cpp" tab. Hit Ctrl-F and search for "TiXmlDeclaration" with option "list all occurrences". I get 4 hits. Repeat exact same search many times. Most of these increase RAM from 4K to 300K. Some leave RAM unchanged.
  • Go back to resizing test. Memory continues to grow as before.
  • Memory at ~160MB.
  • Exit SE. Go through steps above once more. Same results.
  • Do one more test... Go to "tinyxmlparser.cpp" then delete one char and hit undo (Ctrl-Z). Repeat delete/undo 10-15 times, check memory, repeat again, check memory, and so on. Each cycle increases memory 500-1000K.

The memory growth here is smaller than in my larger project but appears similar in nature. Attached is a screenshot as well.

I'm running on Xubuntu 16.04. My SE tagging configuration is the same as I reported previously.

Regards,
Iker


Dennis

  • Senior Community Member
  • Posts: 3954
  • Hero Points: 515
Re: v22 memory leak?
« Reply #39 on: June 11, 2018, 02:30:46 PM »
I'll run through your steps to reproduce later this week and see what comes up.  I'm going to run with a default configuration to start out.  If I cannot reproduce the leak with that I am going to need to have your user.cfg.xml.

IkerAriz

  • Senior Community Member
  • Posts: 159
  • Hero Points: 6
Re: v22 memory leak?
« Reply #40 on: June 13, 2018, 02:29:05 PM »
Attached is my user.cfg.xml file.

Thanks,
Iker

Dennis

  • Senior Community Member
  • Posts: 3954
  • Hero Points: 515
Re: v22 memory leak?
« Reply #41 on: June 19, 2018, 12:43:14 AM »
OK.  I've spent about a half week on this.  Good news and bad news.

1 - Good news)  I've identified, found and fixed a few memory leaks.  These fixes will be incorporated into the next release.

2 - Bad news - or Jedi mind trick)  These are not the leaks you are looking for.  It is unlikely that what I found is what brought our SlickEdit memory footprint up to 2G.  They were small leaks.

3 - resizing and drawing)  Not leaks.  We have a fairly significant line drawing cache.  You are merely filling up the cache.  The test project has a bit over 6000 lines of code, the cache holds 20000 lines before it fills and purges.  This is just using memory, not leaking it.

4 - delete and undo)  I did not observe a significant memory increase here, and the memory leak tracking software did not identify anything as leaked as a result of this test.

5 - searching) same story, yes, first time use of a feature can uptick memory somewhat, when the form is loaded the first time, etc, but no definitive leaks identified.

6 - segmenting)  We have a our own memory allocator for speed.  Like any allocator, it can fall victim to some segmenting depending on usage patterns.  Some of the memory growth you see could be segmenting, well, it will appear as growth initially, but once there is enough memory in use, the growth will level off entirely.  Note, this can happen with the system malloc() too, so there is nothing particularly wrong with our allocator, it's just faster than malloc.

7 - scrolling) see (3)

8 - caches) we have a lot of caches.  Tagging especially has a lot of caching going on.  It is one of the few things that gives you some limited control over the caches, yet, even there, there are some caches that will just grow as needed.  You can't make a setting for everything.  I have suggested ways to cut back the tagging cache earlier in this thread.

9 - your config)  You were loading a mini-map tool that I did not have access to.  Hence, this did not factor into any of my testing.

Hope this helps.  I'm still digging around for leaks, as I work.  Maybe something will surface eventually.

IkerAriz

  • Senior Community Member
  • Posts: 159
  • Hero Points: 6
Re: v22 memory leak?
« Reply #42 on: June 19, 2018, 01:10:23 PM »
Thanks for the follow up Dennis.

Any way to collect additional memory info during my tests? Eg, are there any diagnostic commands that can print the SE cache sizes and and/or allocator stats? (perhaps via a custom build?)

Thanks again,
Iker

P.S. BTW, the minimap command just toggles the zoom (I didn't use it during my tests). Here's the code:

Code: [Select]
int minimap_on = 0;
_command void minimap_toggle () name_info(',') {
   if (minimap_on) {
       wfont_zoom('9');
       minimap_on = 0;
   }
   else {
      wfont_zoom('4');
      minimap_on = 1;
   }
}


Dennis

  • Senior Community Member
  • Posts: 3954
  • Hero Points: 515
Re: v22 memory leak?
« Reply #43 on: June 19, 2018, 02:42:51 PM »
Sorry, there's nothing built in that would provide any really helpful information.  We do have some allocator stats that we occasionally use in debug and unit test builds internally, but they would just provide some stats, nothing of help for tracing down where the memory is going primarily.

IkerAriz

  • Senior Community Member
  • Posts: 159
  • Hero Points: 6
Re: v22 memory leak?
« Reply #44 on: June 19, 2018, 07:26:17 PM »
Any other options I can pursue to help find the problem?

Iker