Author Topic: Non linear performance vs number of files open  (Read 2760 times)

Ian Ameline

  • Community Member
  • Posts: 70
  • Hero Points: 7
Non linear performance vs number of files open
« on: December 17, 2018, 09:43:41 PM »
It appears that performance does not scale linearly with the number of files open -- If I have a large number of files open (>50) performance of quitting a file, or loading a file becomes quite slow If I have 20 files open, the performance is good -- no noticeable delays. And app startup is also quite slow when I have 50 files open. I haven't tested if it is n log n, or n squared (it feels much closer to the latter) In any event, The total amount of data loaded is on the order of a few megabytes of text (50 C++ text files of under 100k each) -- nothing so much that any linear algorithms should be noticeable in the time they take. -- it's at the point where closing a file or opening a new file file takes over a second -- it should be under 50ms even if I have 500 files open)

Switching current file remains fast -- it's adding or removing a file from the set of opened files that is slow and that scales very badly.

I have a *very* fast SSD. (modern apple NVMe -- over 2 gig/second, > 300k iops) Plenty of RAM. CPU is fast - over 3Ghz skylake. How could closing a file take billions of cycles?

I've noticed this performance issue for many years. It's still there in VsPro 2018

-- Ian.

Dennis

  • Senior Community Member
  • Posts: 3954
  • Hero Points: 515
Re: Non linear performance vs number of files open
« Reply #1 on: December 17, 2018, 11:00:54 PM »
Try this to narrow down the problem:

1) Macro > Slick-C Profiler > Start Profiling...
2) Reproduce the problem several times (close files, open some new ones)
3) Macro > Slick-C Profiler > Save...  (/tmp/profile-file-closing.tsv)

The post the profiling data here.  This may help us track down what is different about your setup which is causing the slow-down.

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 6823
  • Hero Points: 526
Re: Non linear performance vs number of files open
« Reply #2 on: December 18, 2018, 01:35:28 AM »
This operation on macOS seems to be much slower than Windows (>4x slower). I'm almost certain that the performance issue is in the GUI (Qt) so Slick-C profiling mostly likely won't help. Post what you find anyway.

Tab creation is essentially what is slowing down performance. If I set "Files per window" to "Multiple files share window" (Tools>Options>Editing>Editor Windows>Files per window), then opening 420 files takes about 2 seconds. I happen to use the "Multiple files share window" setting but not for speed. I use it because it's what I'm used to.

Ian Ameline

  • Community Member
  • Posts: 70
  • Hero Points: 7
Re: Non linear performance vs number of files open
« Reply #3 on: December 18, 2018, 08:25:43 PM »
I'll try profiling that way and with Instruments and see what I find.

Ian Ameline

  • Community Member
  • Posts: 70
  • Hero Points: 7
Re: Non linear performance vs number of files open
« Reply #4 on: December 23, 2018, 03:06:36 PM »
It seems like qTabBar is pretty slow here.

When navigating the tab bar that shopws all the files (using the left and right buttons at the far right of the tab bar), I got it to spend 84% of the time in gui_mark_used_menu_ids

-- Ian.
 

Ian Ameline

  • Community Member
  • Posts: 70
  • Hero Points: 7
Re: Non linear performance vs number of files open
« Reply #5 on: December 23, 2018, 03:18:37 PM »
Dropped 100 files on slickedit -- 72.4% of the time is in gui_mark_used_menu_ids

For call stack, and within that function, see attachments;
Looks like it has a loop, and it's being called from within a loop (calling function loops), so that may be the n^2.
« Last Edit: December 23, 2018, 03:28:20 PM by Ian Ameline »

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 6823
  • Hero Points: 526
Re: Non linear performance vs number of files open
« Reply #6 on: December 24, 2018, 10:41:49 PM »
It seems like qTabBar is pretty slow here.

When navigating the tab bar that shopws all the files (using the left and right buttons at the far right of the tab bar), I got it to spend 84% of the time in gui_mark_used_menu_ids

-- Ian.
I opened 420 files. That takes a long time. When I click and hold the left/right arrows to the right of the QTabBar, the scrolling is reasonably quick on my old slow mac mini. The active tab doesn't change when I do this. I'm wondering if I'm doing something different than you.

Ian Ameline

  • Community Member
  • Posts: 70
  • Hero Points: 7
Re: Non linear performance vs number of files open
« Reply #7 on: December 25, 2018, 06:37:52 PM »
Prefs may be different -- I'll zip mine and send them to you -- do you have a drop-box? (I think they're a bit large for email). I'm also happy to share screen (zoom works best -- it has the lowest overhead screen sharing of anything I've seen), and show what I'm doing. (I can do a recording, if that helps).

-- Ian.

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 6823
  • Hero Points: 526
Re: Non linear performance vs number of files open
« Reply #8 on: December 26, 2018, 04:56:43 AM »
Just post your user.cfg.xml. I don’t think I’ll need anything else.

Dennis

  • Senior Community Member
  • Posts: 3954
  • Hero Points: 515
Re: Non linear performance vs number of files open
« Reply #9 on: January 08, 2019, 03:35:30 PM »
I have a fix which will help some when users have large numbers of files open.  It doesn't speed up opening new files by much of a factor, but it is a major help for close-all, and in general cuts down on some overhead that can amount to O(n) for a single file operation or O(n^2) for multiple file operations.  I will put this into the next hot fix for SlickEdit 2018.

Your profiling information seems to indicate to me that you might be using the old (deprecated) File Tabs tool bar, rather than using the built-in document tabs.  That toolbar does create a menu, and menu creation does tend to be slower on macOS.  Can you confirm that you are using File Tabs?  If so, why?
« Last Edit: January 08, 2019, 03:42:00 PM by Dennis »