Author Topic: B4: Setting error markers and next error taking very long time  (Read 589 times)

rowbearto

  • Senior Community Member
  • Posts: 1481
  • Hero Points: 112
B4: Setting error markers and next error taking very long time
« on: September 27, 2019, 04:06:09 pm »
Look for errormarkerslongtime.tar.gz on support.

I have a build log in there and my user.cfg.xml.

When I do a build of "cat build_last.log", it is taking a very very long time to set error markers. Also next-error is taking a long time.

I'm on Linux x64, CentOS 7.2

rowbearto

  • Senior Community Member
  • Posts: 1481
  • Hero Points: 112
Re: B4: Setting error markers and next error taking very long time
« Reply #1 on: September 27, 2019, 04:17:12 pm »
I exited and restarted SlickEdit and now it doesn't take so long anymore.

I was adding to the error parsing exclusions and rebuilding (cat build_last.log) to check that the excluding was working.

I think the last thing I did before the long time was editing the "No parallel" exclusion. It was previously named something like "Recurse.mk" and it used to be: "Recurse.mk\:?+\: No Parallel" and then I removed the "Recurse" at the start of the string and renamed it to "No Parallel". I think that is when the error parsing started to slow down, but not 100% sure.

rowbearto

  • Senior Community Member
  • Posts: 1481
  • Hero Points: 112
Re: B4: Setting error markers and next error taking very long time
« Reply #2 on: September 27, 2019, 04:22:54 pm »
Just started happening again after I added a new error exclusion and did another "build":

I added to exclusions:

Name: entering_dir
Expression: \.mk\::d+\:?+\: Entering directory
Test case: /gitworkspace/rbresali/cmagentvalg/lc/build/rules/Image.mk:7: make[4]: Entering directory `/gitworkspace/rbresali-cmagentvalg_190927.110559/lc

I validated the test case, then when I try a rebuild it takes a long time to set error markers.

But if I close and restart SE, it doesn't take so long anymore.

Also before restarting, after taking a long time to set the markers my exclusion was not working and it found the error. But after restart it skipped the error and properly excluded.

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 4922
  • Hero Points: 402
Re: B4: Setting error markers and next error taking very long time
« Reply #3 on: September 27, 2019, 08:39:47 pm »
I'm seeing slow performance before and after adding the 35th exclude regex. That's a lot of excludes.

I think you may have just reached a performance limitation.

Before adding new error regex: ~30 seconds

After adding new error regex: ~34 seconds

After restarting: ~35 seconds

Now I'm getting slower times (like 40 seconds) without the 35 regex exclusion after a restart. This VM is on a network so there could be more traffic now. 

rowbearto

  • Senior Community Member
  • Posts: 1481
  • Hero Points: 112
Re: B4: Setting error markers and next error taking very long time
« Reply #4 on: September 27, 2019, 08:44:34 pm »
When I restart it is very quick, I don't even notice the time so it may be 1 second or so.

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 4922
  • Hero Points: 402
Re: B4: Setting error markers and next error taking very long time
« Reply #5 on: September 28, 2019, 01:04:29 am »
I created a project and made the build command type out your sample file. That way, it tries to set error markers. It's always slow for me.

rowbearto

  • Senior Community Member
  • Posts: 1481
  • Hero Points: 112
Re: B4: Setting error markers and next error taking very long time
« Reply #6 on: September 28, 2019, 03:01:18 am »
Can we qualify the definition of the time interval that we are measuring?

I'm referring to when the last print of the build comes out until I get the prompt.

Actually I just tried it after running SE for a while without mucking with error parsing, and this time interval was pretty slow, likely in the 30s of seconds.

But then I quit and restarted SlickEdit and it was very fast. If you use my user.cfg.xml file, you will see that when I start SE I have no files open and no workspace open. So after restart I open the workspace and do the build right away. At this time the time interval is very fast.

Then I switched to a different workspace, then back to my original one. Seemed fast.

Then I opened a python file (just happened to be the last thing I worked on) and did the build. Then it became very slow.

But if I then go to no tabs, no buffers, no files open again and do the build and it is slow.

rowbearto

  • Senior Community Member
  • Posts: 1481
  • Hero Points: 112
Re: B4: Setting error markers and next error taking very long time
« Reply #7 on: September 28, 2019, 03:07:05 am »
It seems to have something to do with opening the first file. Maybe you also want my user.cfg.xml.

Now I tried:

Fresh restart, no buffers, no workspace, load workspace, do build: Fast
Use Smart Open to navigate to a directory with a python file I was working on, open that, do build: Slow

rowbearto

  • Senior Community Member
  • Posts: 1481
  • Hero Points: 112
Re: B4: Setting error markers and next error taking very long time
« Reply #8 on: September 28, 2019, 03:10:13 am »
My user.cfg.xml changed a little since I last uploaded. Maybe to reproduce my most recent results, you need it?

You can find it on support as user.cfg.xml.2019.09.27

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 4922
  • Hero Points: 402
Re: B4: Setting error markers and next error taking very long time
« Reply #9 on: September 28, 2019, 07:09:25 pm »
Before adding anything, I ran my test build command with your latest user.cfg.xml and it took 39 seconds to finish setting the error markers.

Note: This timing has nothing to do with what the message line displays. I'm waiting until a pending click and cursor movement occurs (before that SlickEdit is just locked up).
« Last Edit: September 28, 2019, 07:22:05 pm by Clark »

rowbearto

  • Senior Community Member
  • Posts: 1481
  • Hero Points: 112
Re: B4: Setting error markers and next error taking very long time
« Reply #10 on: September 28, 2019, 07:27:10 pm »
Ok. Anyway if you could look into why it takes so long. Would be so much better if it was faster. I think it is possible as it goes fast for me right after a restart.

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 4922
  • Hero Points: 402
Re: B4: Setting error markers and next error taking very long time
« Reply #11 on: September 28, 2019, 09:39:21 pm »
I'll look but I suspect this is just too many lines and too many excludes. The crux of the problem is that there are way too many miss hits. For each miss hit, it has to run 35 regular expressions only to figure out that it needs to skip it.

The better solution would be to write a regex which doesn't get so many false positives. If that's possible, there's no point in me trying to optimize this because you'll be able to blow away whatever I do.
« Last Edit: September 28, 2019, 10:41:50 pm by Clark »

rowbearto

  • Senior Community Member
  • Posts: 1481
  • Hero Points: 112
Re: B4: Setting error markers and next error taking very long time
« Reply #12 on: September 28, 2019, 11:11:10 pm »
Well the fact that it is very fast for me right after a reset seems to indicate even with all the 35 regular expression exclusions it can still be fast. Unfortunately you don't reproduce this.

I will look into changing my hit and miss regular expressions but I think it will be a very difficult exercise. Are there any tools to help me debug this?

rowbearto

  • Senior Community Member
  • Posts: 1481
  • Hero Points: 112
Re: B4: Setting error markers and next error taking very long time
« Reply #13 on: September 28, 2019, 11:14:06 pm »
I just unchecked the exclusions and it is still taking a very long time to do the error parsing.

Is there any way to find out which one of the "hit" regular expressions each error fired on?

rowbearto

  • Senior Community Member
  • Posts: 1481
  • Hero Points: 112
Re: B4: Setting error markers and next error taking very long time
« Reply #14 on: September 28, 2019, 11:20:11 pm »
Some important information from an experiment. Bottom line is after the restart, enabling the exclusions makes this FASTER. And if I disable the exclusions, even after a restart for me, it makes it much SLOWER. I think it is because the slowness is in setting all the error markers, more detail below.

1) With exclusions unchecked, after a restart and do a build it takes a very long time and is status bar says "Setting error markers". I see 695 of them in the "Messages" window (your number may be slightly different, I'm using a newer build long).

2) But when I enable the exclusions, it is very very fast and there are no errors listed in the "Messages" window.

So at least for me in the case of restart, having the 35 exclusions is speeding things up, not slowing them down!

The slowness may be in setting the error markers, which are not there when the exclusions are enabled.