SlickEdit Community

Archived Beta Discussions => SlickEdit 201x Beta Discussions => SlickEdit 2019 v24 Beta Discussion => Topic started by: rowbearto on September 27, 2019, 04:06:09 PM

Title: B4: Setting error markers and next error taking very long time
Post by: rowbearto on September 27, 2019, 04:06:09 PM
Look for errormarkerslongtime.tar.gz on support.

I have a build log in there and my user.cfg.xml.

When I do a build of "cat build_last.log", it is taking a very very long time to set error markers. Also next-error is taking a long time.

I'm on Linux x64, CentOS 7.2
Title: Re: B4: Setting error markers and next error taking very long time
Post by: rowbearto on September 27, 2019, 04:17:12 PM
I exited and restarted SlickEdit and now it doesn't take so long anymore.

I was adding to the error parsing exclusions and rebuilding (cat build_last.log) to check that the excluding was working.

I think the last thing I did before the long time was editing the "No parallel" exclusion. It was previously named something like "Recurse.mk" and it used to be: "Recurse.mk\:?+\: No Parallel" and then I removed the "Recurse" at the start of the string and renamed it to "No Parallel". I think that is when the error parsing started to slow down, but not 100% sure.
Title: Re: B4: Setting error markers and next error taking very long time
Post by: rowbearto on September 27, 2019, 04:22:54 PM
Just started happening again after I added a new error exclusion and did another "build":

I added to exclusions:

Name: entering_dir
Expression: \.mk\::d+\:?+\: Entering directory
Test case: /gitworkspace/rbresali/cmagentvalg/lc/build/rules/Image.mk:7: make[4]: Entering directory `/gitworkspace/rbresali-cmagentvalg_190927.110559/lc

I validated the test case, then when I try a rebuild it takes a long time to set error markers.

But if I close and restart SE, it doesn't take so long anymore.

Also before restarting, after taking a long time to set the markers my exclusion was not working and it found the error. But after restart it skipped the error and properly excluded.
Title: Re: B4: Setting error markers and next error taking very long time
Post by: Clark on September 27, 2019, 08:39:47 PM
I'm seeing slow performance before and after adding the 35th exclude regex. That's a lot of excludes.

I think you may have just reached a performance limitation.

Before adding new error regex: ~30 seconds

After adding new error regex: ~34 seconds

After restarting: ~35 seconds

Now I'm getting slower times (like 40 seconds) without the 35 regex exclusion after a restart. This VM is on a network so there could be more traffic now. 
Title: Re: B4: Setting error markers and next error taking very long time
Post by: rowbearto on September 27, 2019, 08:44:34 PM
When I restart it is very quick, I don't even notice the time so it may be 1 second or so.
Title: Re: B4: Setting error markers and next error taking very long time
Post by: Clark on September 28, 2019, 01:04:29 AM
I created a project and made the build command type out your sample file. That way, it tries to set error markers. It's always slow for me.
Title: Re: B4: Setting error markers and next error taking very long time
Post by: rowbearto on September 28, 2019, 03:01:18 AM
Can we qualify the definition of the time interval that we are measuring?

I'm referring to when the last print of the build comes out until I get the prompt.

Actually I just tried it after running SE for a while without mucking with error parsing, and this time interval was pretty slow, likely in the 30s of seconds.

But then I quit and restarted SlickEdit and it was very fast. If you use my user.cfg.xml file, you will see that when I start SE I have no files open and no workspace open. So after restart I open the workspace and do the build right away. At this time the time interval is very fast.

Then I switched to a different workspace, then back to my original one. Seemed fast.

Then I opened a python file (just happened to be the last thing I worked on) and did the build. Then it became very slow.

But if I then go to no tabs, no buffers, no files open again and do the build and it is slow.
Title: Re: B4: Setting error markers and next error taking very long time
Post by: rowbearto on September 28, 2019, 03:07:05 AM
It seems to have something to do with opening the first file. Maybe you also want my user.cfg.xml.

Now I tried:

Fresh restart, no buffers, no workspace, load workspace, do build: Fast
Use Smart Open to navigate to a directory with a python file I was working on, open that, do build: Slow
Title: Re: B4: Setting error markers and next error taking very long time
Post by: rowbearto on September 28, 2019, 03:10:13 AM
My user.cfg.xml changed a little since I last uploaded. Maybe to reproduce my most recent results, you need it?

You can find it on support as user.cfg.xml.2019.09.27
Title: Re: B4: Setting error markers and next error taking very long time
Post by: Clark on September 28, 2019, 07:09:25 PM
Before adding anything, I ran my test build command with your latest user.cfg.xml and it took 39 seconds to finish setting the error markers.

Note: This timing has nothing to do with what the message line displays. I'm waiting until a pending click and cursor movement occurs (before that SlickEdit is just locked up).
Title: Re: B4: Setting error markers and next error taking very long time
Post by: rowbearto on September 28, 2019, 07:27:10 PM
Ok. Anyway if you could look into why it takes so long. Would be so much better if it was faster. I think it is possible as it goes fast for me right after a restart.
Title: Re: B4: Setting error markers and next error taking very long time
Post by: Clark on September 28, 2019, 09:39:21 PM
I'll look but I suspect this is just too many lines and too many excludes. The crux of the problem is that there are way too many miss hits. For each miss hit, it has to run 35 regular expressions only to figure out that it needs to skip it.

The better solution would be to write a regex which doesn't get so many false positives. If that's possible, there's no point in me trying to optimize this because you'll be able to blow away whatever I do.
Title: Re: B4: Setting error markers and next error taking very long time
Post by: rowbearto on September 28, 2019, 11:11:10 PM
Well the fact that it is very fast for me right after a reset seems to indicate even with all the 35 regular expression exclusions it can still be fast. Unfortunately you don't reproduce this.

I will look into changing my hit and miss regular expressions but I think it will be a very difficult exercise. Are there any tools to help me debug this?
Title: Re: B4: Setting error markers and next error taking very long time
Post by: rowbearto on September 28, 2019, 11:14:06 PM
I just unchecked the exclusions and it is still taking a very long time to do the error parsing.

Is there any way to find out which one of the "hit" regular expressions each error fired on?
Title: Re: B4: Setting error markers and next error taking very long time
Post by: rowbearto on September 28, 2019, 11:20:11 PM
Some important information from an experiment. Bottom line is after the restart, enabling the exclusions makes this FASTER. And if I disable the exclusions, even after a restart for me, it makes it much SLOWER. I think it is because the slowness is in setting all the error markers, more detail below.

1) With exclusions unchecked, after a restart and do a build it takes a very long time and is status bar says "Setting error markers". I see 695 of them in the "Messages" window (your number may be slightly different, I'm using a newer build long).

2) But when I enable the exclusions, it is very very fast and there are no errors listed in the "Messages" window.

So at least for me in the case of restart, having the 35 exclusions is speeding things up, not slowing them down!

The slowness may be in setting the error markers, which are not there when the exclusions are enabled.
Title: Re: B4: Setting error markers and next error taking very long time
Post by: rowbearto on September 29, 2019, 01:30:15 AM
I found that one of the default error parsing expressions provided in the clean config, and also in my config, was triggering the 695 false positives in my build.

I also think that the reason I had 35 exclusions, built up over many many years, is because the default error parsing captured so many things that whenever I wanted to exclude anything I would quickly add it to exclusions, and over time I got to 35+ exclusions.

I never took the time until today to reexamine why I was getting so many false positives and adjust how error parsing is done. It was faster in the past to just add another exclusion.

I still think that SlickEdit has an issue because even when I turn off the exclusions, with 695 error markers there is quite a long pause before SE is usable again. But after performing my refinement of error parsing, it is no longer an issue for me personally anymore.

The problematic default error parsing is in the "default" Category and is the "def1". By turning this off, I got rid of my 695 false positives. However turning this off also excluded finding gcc errors and warnings that I would like to see.

SlickEdit may want to reconsider the default error parsing rules to not catch so many false positives.

So I made a much more restrictive error parser that doesn't have all the false positives, and now I only need 1 exclusion.

For gcc formatted errors I now use this:
Code: [Select]
^{#0?+}\:{#1:i}\:({#2:i}\::b|:b){#3(error\:|warning\:|undefined:breference|note\:)?+}$
It will match what gcc outputs in sample messages such as:

Code: [Select]
file:linenum:col: (error:|warning:|undefined reference|note:) message
or without the column

Code: [Select]
file:linenum: (error:|warning:|undefined reference|note:) message
By being this restrictive, I no longer need many of my exclusions for when timestamps were output to the build log, which had colon characters that created a false positive.

I added one restriction to exclude messages coming from our makefiles, which are overdue to be fixed, but it is not my responsibility. So I added 1 exclusion:

Code: [Select]
^(?+\.mk|Makefile)\::i\: warning\:
Which will exclude things like:

Code: [Select]
Makefile:linenum: warning: message
anyfile.mk:line: warning: message


Title: Re: B4: Setting error markers and next error taking very long time
Post by: Clark on September 29, 2019, 01:14:30 PM
For me, when I remove the excludes, this finishes very fast.  <2 seconds.

Because of this, I started to see if it is the number of excludes or a specific exclude. For me, I can leave all the excludes enabled except for the "csharp" exclude and performance is always good. There's something funky going on with the "csharp" exclude. It looks ok. I even tried tweaking it.

Then I disabled all excludes except for the "csharp" exclude. Once I do that, the performance is horrible. Seems like something weird is going. I'll have to investigate what's going on with it.

My results are always consistent. Doesn't matter whether I restart.
Title: Re: B4: Setting error markers and next error taking very long time
Post by: rowbearto on September 29, 2019, 03:35:36 PM
I just moved from beta4 to 24.0.0.7. I went back to my old user.cfg.xml to try out 24.0.0.7. Now it is very slow with the restart, even with excludes enabled. Perhaps the difference in our outcomes all along was that I was still on B4 and you were using a more recent version?

And with 24.0.0.7 when I disable the excludes it is very fast.

So now with 24.0.0.7 I'm seeing what you see.

Perhaps if you back to beta4 you can see what I was reporting previously.

Regarding the csharp exclude, I just added that a few days ago and it also lines up with when I first started seeing this issue.
Title: Re: B4: Setting error markers and next error taking very long time
Post by: Clark on September 29, 2019, 03:54:21 PM
I'm pretty sure I found the problem.

Edit error.e and change line #2423 to the following:

Code: [Select]
      if (folder_array[i].m_enabled && pos(langRE, folder_name, 1, "ir") == 1) {

The problem is that the logic is backwards.

Reload error.e and restart. That should fix the problem. This same issue exists in v23.
Title: Re: B4: Setting error markers and next error taking very long time
Post by: rowbearto on September 29, 2019, 04:05:59 PM
Well after applying your error.e fix, 24.0.0.7 is going back to the behavior I had before. I think you fixed one issue but there is still a remaining issue.

After a fresh restart, the build is very fast, like I was observing before with B4. So this is fixed.

Then I loaded a python file into a buffer and did a 2nd build, it is very slow when I do that. Saying "setting error markers..." and then many (location x%) Searching..."
Title: Re: B4: Setting error markers and next error taking very long time
Post by: rowbearto on September 29, 2019, 04:12:25 PM
Then, after the 2nd build loading the python file, I disable excludes, a 3rd build is still very slow. But I only see in the status bar "Setting error markers" for a long time. I don't see the "(location x%) Searching..." in the status bar in this case.
Title: Re: B4: Setting error markers and next error taking very long time
Post by: Clark on September 29, 2019, 04:41:46 PM
I'm trying to reproduce bad performance. Not getting it right now. Editing a Python file too. There aren't any language specific regular expressions (that's where I found the bug). It shouldn't matter what type of file you edit because there are no language specific regular expressions.
Title: Re: B4: Setting error markers and next error taking very long time
Post by: rowbearto on September 29, 2019, 06:08:57 PM
I uploaded errormarkerslongtime2.tar.gz to support.

Look for the README file in there. I included some stuff in there to make your system more like mine, so maybe you will be able to reproduce.

Another suggestion is to try to use the Virtual Machine you got from me.
Title: Re: B4: Setting error markers and next error taking very long time
Post by: Clark on September 29, 2019, 07:23:06 PM
Before I do that. Here's a new error.e.  Load it and restart SlickEdit.

Once you see that the build is slow, put the focus on the debug window and press Ctrl+C to copy all the debug messages to the clipboard. You can post the debug here. It maybe best to put them in a file and attach the file.

I want to check what the regex is for searching for an error to verify that no excludes are in there. That was the bug.
Title: Re: B4: Setting error markers and next error taking very long time
Post by: rowbearto on September 29, 2019, 08:06:36 PM
I don't see a debug window? How to bring that up?
Title: Re: B4: Setting error markers and next error taking very long time
Post by: rowbearto on September 29, 2019, 08:07:14 PM
You also want me to turn off the excludes?
Title: Re: B4: Setting error markers and next error taking very long time
Post by: Clark on September 29, 2019, 08:38:58 PM
I don't see a debug window? How to bring that up?

When you run the Build command for your project, you should see a debug window come up.

Keep the excludes on.
Title: Re: B4: Setting error markers and next error taking very long time
Post by: rowbearto on September 29, 2019, 09:39:37 PM
The debug window is not popping up. I left the excludes on. Maybe the "say" statement you added is not getting executed? I placed the error.e file into the macros directory while SE was not running, then started SE.
Title: Re: B4: Setting error markers and next error taking very long time
Post by: Clark on September 29, 2019, 10:04:00 PM
You have to manually load the macro Macro>Load Module...

This will compile the macro if it is out of date and load the resulting pcode. Make sure there is read/write permissions to the directory the macro resides in.
Title: Re: B4: Setting error markers and next error taking very long time
Post by: rowbearto on September 30, 2019, 12:01:07 AM
Macro>Load Module is what I was missing.

When I use the error.e file with just the fix, and without your debug print, now with the 2nd build after loading the python file, the error marker setting is very fast!

I didn't try your second error.e with the debug prints, as the first one seems to solve the problem, and I was missing the part about Macro>Load Module.

Thanks Clark!
Title: Re: B4: Setting error markers and next error taking very long time
Post by: Clark on September 30, 2019, 12:26:39 PM
Glad that works. We will add this as a hot fix.