Linux vsdiff not working

[StephenW]

SlickEdit 25.0.1.0 + hotfixes (Linux & Windows)

I have recently upgraded my main Ubuntu box to a brand new CPU and motherboard and as a consequence I am now using the Linux version of SlickEdit much more instead of working from my Windows box over a network connection.  So today I tried diffing two log files to see why my Python 3 version of the old Python 2 code was not working the same, and I discovered that on Linux vsdiff does not work - it does not see a lot of the differences between the files.  When I tried the Windows version on the same two files via SMB, vsdiff worked normally.  I got the same result (as expected) from doing the diff using the internal SlickEdit diff.

Has anyone else had a problem with vsdiff on Linux?

[Dan]

Is there any chance you can send me the files?

Are you running vsdiff with the filenames, or running it and then filling in the dialog?

What are your diff options to when you launch vsdiff?

Is Source Diff on?

When you run vs.exe, do you specify a configuration path?

[StephenW]

I have put the two log files on my web server:

http://www.jsw.gen.nz/vsdiff/bad.log
http://www.jsw.gen.nz/vsdiff/good.log

I am comparing them using "vsdiff bad.log good.log"  When I do that and click on the "Options" button, "Source Diff" is the only option with a tick and all the other tickable options are off.  "Intra-line diff length limit" is 4 KB, "Diff timeout" is 60 Seconds, "Date and Size Optimization" is set to "Always assume match if date and size match".  I am running vsdiff as root, without specifying a configuration path.  It looks like vsdiff is only seeing a difference between the files where good.log is longer than bad.log.  It is not seeing non-matching lines before then.  The first non matching line is actually line 1.  Turning off the "Source Diff" option and re-diffing the files makes no difference.

I am running this on an Ubuntu 20.04 system that has been upgraded probably since 14.04 originally.  Similarly, SlickEdit has been upgraded through a number of versions on the box since I originally bought my extra license to use it on Linux.

[StephenW]

I have just created a new VirtualBox Ubuntu 20.04 to test this problem on, so that I have a clean install.  I installed from a 20.04.1 DVD image and then did all the latest updates.  I installed SAMBA and set up file sharing with my main Ubuntu 20.04 box.  As root I installed SlickEdit from se_25000100_linux64.tar.gz.  I got the usual problem where the dialog to install the license did not work (it was a blank window), so I copied the license file manually.  Then I added the SlickEdit path to /etc/environment and rebooted.  From my default user I opened a terminal and ran "vs".  In the setup screens I only changed the emulation to Brief and told it to tag the available libraries.  I ran vs as my default user and installed the latest hotfixes.  Then I ran vsdiff on the log files via SMB to the main Ubuntu 20.04 box.  The result was the same - vsdiff only showed differences from where the good.log file was longer than bad.log.

[Clark]

I can easily reproduce this problem on Windows and Linux with a default config. I can reproduce the problem with 25.0.1 vsdiff or 25.0.1 vs. Interestingly, I can't reproduce the problem with 24.0.2 vsdiff.

I noticed these files have slightly inconsistent new line endings but if I replace all 13 characters with an 'x', I still see the same diff problems with 25.0.1.

[Dan]

@Stephen:  We will be working on finding the best solution for this.

For the moment, the best workaround is to set Go to Tools>Options>Files and set "Wrap line length" to 8000.  The default is 4000, the longest line in these files are ~6500 bytes.  So 8k will handle these files for sure.

You will need to restart the editor after you change that value.  I am also putting in a change so that this will automatically prompt you to restart.

[StephenW]

Great, that fixes it.  And it also explains why it worked on Windows - at some time in the past I have set my wrap line length there to a massive number.

[Clark]

Actually, there was a bug where Diff was setting the "Wrap line length" to a really large number. The GUI limits the "Wrap line length" to 20,000. It would fix this problem though.

We hope to enhance diff to handle longer lines in the future. Right now, the limit is 20,000.

[StephenW]

No, I spoke too soon.  With wrap line length set to 8000, the second difference line displays as "('len(blk[2]=4066',)" on the right hand side (good.log), but the left hand side (bad.log) has no matching lines shown for a great many lines after that.  But there are matching, partial matching and non-matching lines in bad.log after that, they just not displayed.  At the end of the files it starts working properly again (at "Matching MHEG channels to MythTV channels using map file"), the matching lines there are shown in both files.

I also had Ubuntu vs lock up ("not responding") when I was examining the diff display at the point where the bad.log lines disappear.  I let it wait for a bit, but it did not start responding again so I let the system kill it and restarted it.

With the "Wrap line length" on Windows, the GUI prompted me to change the massive number there to 20000 after I switched away in my KVM to my Ubuntu box and came back later.  It did not prompt me immediately when I clicked on that setting.  So I have now set it to 8000 there also.

[Clark]

As far as I can tell, with wrap length set to 8000, the diff output results are correct. They don't look as good as they could be but these files are extremely different. The file lengths are not even close (1400 lines vs 1339582 lines). If the files were closer with more matched lines, then the lines with partial matching would be detected.

Not sure why you got a hang. I have been creating tests that are modifications of these test files. One test I made the files were even more different and it took a really long time to produce the output (over 15 minutes but I wasn't timing them). Files that are extremely different like these can be really slow to diff.

[StephenW]

Take a look at line 86 in both files.  That is the last matching line that vsdiff shows at the top of the files.  But if you then look at the files individually rather than in vsdiff, line 87 also matches.  But vsdiff does not see that match, and in the bad.log display it is matched to line 1335613 of good.log but given a line number of -2629 instead of 87.  There are no colour marks in the column between the two file panes for the matches from there on to the bottom of the files.  The Next Diff button does find them despite that.

The negative line numbers shown at the bottom of bad.log are not a good sign - maybe that is what is causing some of the problems.

[Clark]

The negative line numbers are a bug. This will be fixed in 25.0.2. This wacky line number shouldn't effect anything else other than it's wrong and looks really bad.

Not sure why line 87 isn't picked up as a match "('len(blk[2]=4066',)". We will look into this. Either this is a bug or it is due to the algorithm bailing due to the massive number of mismatches. Under extreme conditions (very large files with many differences), the diff algorithm will bail which results in non-optimal diff output. This is done so the diff doesn't take as long. The only way we will know is to investigate this further.

[StephenW]

There is no rush for any fix for me as I also own a copy of bcompare (https://www.scootersoftware.com) that works on Linux and does diff the files correctly.  But it is much easier to do diffs using SlickEdit's builtin diff, so I prefer to use that normally.  I think I may write a macro to run bcompare from SlickEdit if I have to do it often.

[Clark]

The negative line numbers issue will be fixed for 25.0.2 (fix already checked in). We are looking into the matching to see what is going on there.

We have plans for improving the line length issue. I'll post here when when that happens.

[Dan]

Go to Macro>Set Macro Variable.

For the variable name, put "def_smart_diff_limit".  For the value, put 100,000 (no comma, I just wanted it to be clearly readable).  This should fix the issue with the one instance of "('len(blk[2]=4066',)" in file2 that has been matching at the bottom of the inserted section instead of the top. 

def_smart_diff_limit is a very, very old option, and we will either be setting a new limit, or possibly getting rid of it altogether in v26.0.0 (we'll have to do testing on files with a huge number of differences).

A few other lines before the inserted section show up as inserted too.  I'm not sure why that is exactly, but it is the same result as we got before the diff engine changed v23.0.0.  I'll be looking into this too.