SlickEdit skipping JARs in my classpath

[rygaard]

Hello,

It seems that SlickEdit is skipping some JAR entries in my class path when I invoke project-compile (via a keyboard shortcut).  I don't have much visibility into what is happening, but here are some experiments I ran with on particular JAR:

• Experiment 1
  • I added the JAR to my system CLASSPATH environment variable.
  • I restarted SlickEdit
  • I removed the JAR from Build->Java Options, in the Classpath tab.
  • I invoked project-compile (via a keyboard shortcut)
  • project-compile failed on references to classes in the JAR file
  • I double-checked my system CLASSPATH environment variable to ensure the new entry is correct
  • In the SlickEdit build window, I confirmed that the CLASSPATH environment variable includes the new entry
     
• Experiment 2
  • I removed the JAR from my system CLASSPATH environment variable.
  • I restarted SlickEdit
  • I added the jar to Build->Java Options, in the Classpath tab, using the Add Jar File... button.
  • I invoked project-compile (via a keyboard shortcut)
  • project-compile failed on references to classes in the JAR file
     
• Experiment 3
  • I added the JAR to my system CLASSPATH environment variable.
  • I restarted SlickEdit
  • I added the jar to Build->Java Options, in the Classpath tab, using the Add Jar File... button.
  • I invoked project-compile (via a keyboard shortcut)
  • project-compile failed on references to classes in the JAR file
  • I double-checked my system CLASSPATH environment variable to ensure the new entry is correct
  • In the SlickEdit build window, I confirmed that the CLASSPATH environment variable includes the new entry
     
• Experiment 4
  • I added the JAR to my system CLASSPATH environment variable.
  • Outside of SlickEdit, I compiled my Java source code, and let the compiler use the system CLASSPATH environment variable
  • The code compiled okay.
     
• Experiment 5
  • I added the JAR to my system CLASSPATH environment variable. as a wildcard
  • I restarted SlickEdit
  • I removed the JAR from Build->Java Options, in the Classpath tab.
  • I invoked project-compile (via a keyboard shortcut)
  • project-compile failed on references to classes in the JAR file
  • I double-checked my system CLASSPATH environment variable to ensure the new entry is correct
  • In the SlickEdit build window, I confirmed that the CLASSPATH environment variable includes the new entry
     
• Experiment 6
  • I removed the JAR from my system CLASSPATH environment variable.
  • I restarted SlickEdit
  • I added the jar to Build->Java Options, in the Classpath tab, using the Add Jar Wildcard... button.
  • I invoked project-compile (via a keyboard shortcut)
  • project-compile succeeded
     
• Experiment 7
  • I added the JAR to my system CLASSPATH environment variable. as a wildcard
  • I restarted SlickEdit
  • I added the jar to Build->Java Options, in the Classpath tab, using the Add Jar Wildcard... button.
  • I invoked project-compile (via a keyboard shortcut)
  • project-compile succeeded
     
• Experiment 8
  • I added the JAR to my system CLASSPATH environment variable, as a wildcard
  • Outside of SlickEdit, I compiled my Java source code, and let the compiler use the system CLASSPATH environment variable
  • The code compiled okay.
     

Some notes:

• Java Live Errors is not flagging the references to classes in the JAR.  This indicates that Java Live Errors is properly including my JAR entries.  It also indicates that this is not a problem with my class path.
• I haven't done extensive experimentation with other JARs, but it appears to be happening to all JARs in my class path.
• I am using a customized projconv.e, provided by @patrick.
• I'm using SlickEdit 26.
• I'm using Java 17.
• I haven't run these experiments on Linux or Macintosh.
• Experiments 6 and 7 showed me a work-around for this bug.

As you can see, I spent a LOT of time trying to figure out how to get around a SlickEdit bug.  I shouldn't have to do that.

I would prefer that you didn't touch my classpath.  However, if you do something with my classpath, you MUST let me know what you are doing.  When I run into problems, visibility into what is happening can help me understand what is going wrong.  Just as importantly, it will help me understand if the bug is mine or SlickEdit's.

[patrick]

For the system classpath environment variable: By default, we do not append the contents of the CLASSPATH variable to the classpath in the Java Options.   The question as to whether it's desirable or not is really up to details in the user's development environment and practices.  If you want to include the CLASSPATH environment variable, you can go to the Build -> Java Options -> Classpath tab, and click on the "Add Classpath" button.  That will add a "%(CLASSPATH)" entry, which is our syntax for expanding classpath variables. 

For live errors, it works off of the same class path as the build, and should not look at the CLASSPATH environment variable directly.   And that seems to be the case when I tested it this morning.

So my test setup is:
   1) project with enough jars to trigger the behavior where the classpath is passed in a jar file.
   2) %(CLASSPATH) added to the Java Options -> Classpath, to get the behavior you expected.
   3) A few jar files that are outside of the Java Options classpath, but specified in the CLASSPATH environment.

With that, and allowing for the CLASSPATH not be included by default, I've gone through your experiments, and I haven't had anything go wrong, except when I removed needed jars from CLASSPATH or the configuration.  (although I did shoot myself the foot a couple of times at the beginning and had to restart the tests, when I forgot that vsbuild invokes javac only when there source changes it knows about).

And in one respect, the last experiments you list as working don't make sense to me,  unless Add Jar Wildcard is adding more jars than expected (I have seen your post about that, but haven't dug into it yet).   The UI is different, but it adds them via the same mechanism as the other choices.

So we need debug output, and I'm still working on the code for that.  When that's done sometime later today, it will allow us to turn on debugging + saving that classpath jar via the Verbose checkbox that's in Project -> Project Properties -> Tools -> Build.   And from there, we can go with the simple configuration of what you wanted for 1, with some items in CLASSPATH, others in the Build configuration.   Avoiding "Add Jar Wildcard" until we can confirm it's not doing some extra unwanted inclusions.

[patrick]

Ok, I've finished adding and testing the debug output.   I've attached the two updated .e files, projconv.e (which includes the changes you had before), and compile.e.

Side note: when I was testing on windows, I noticed a windows specific problem which I fixed along with the debugging changes.  I don't think it caused your problems, because when it manifested, it would have dumped a lot of text and error messages into the Build window that don't come from the Java compiler. 

Loading:  like before, go to the SlickEdit command prompt and "load FILE.e" for the attached projconv.e and compile.e, in that order.

Enabling debugging: Open up your project, and go to Project -> Project Properties, click on the Tools tab, select the "Build" tool from the list, and then check the "Verbose output" checkbox.  Hit Ok to dismiss the properties dialog.

What enabling gets you: there's a lot of output for classpath debugging, so it gets routed to the Output tool window instead of the Build window.  The output window will show:
    1) The entire original classpath entries, before any entries are made relative to work with the jar file.
    2) The jar command that builds the classpath jar from the manifest
    3) The name of the copy of the jar that we save for checking later.  (by default TEMPDIR\PROJECTNAME.jar).
    4) The contents of the manifest.txt that we gave to jar.  Handy to have, as jar will reformat this when packing
        it up, so it's not necessarily the same as the MANIFEST.MF that you can extract from the jar.

That should be enough information to catch problems setting up the classpath at any stage.

So with that enabled, you can set up to do essentially your setup in Experiment 1, with some classes on classpath,  the %(CLASSPATH) entry in the Java Options classpath.  Change a source file before building, because we want to force vsbuild to execute javac so we can see the whole picture of what's being passed to javac as well.   And then we can try to see the differences between what you configured and what javac will see, and debug from there.

[rygaard]

I should have clarified:  In my project (and in all of my experiments), the first entry in the Build -> Java Options -> Classpath tab is %(CLASSPATH)

[rygaard]

TOO COOL!

Thanks for providing a hotfix that gives me visibility into SlickEdit's classpath.  It works as advertised.  As predicted, this has helped identify issues with the classpath handling.

One of the issues has broken the project-compile command again, but because of your hotfix, I know how to hack around it.

Another issue has resulted in output that exceeds the 17,000 byte limit for messages on this forum, so it's difficult to send you a description of what I found.  Also, the output has corporate sensitive data.  Is there a way I can send you an HTML document that shows the output and describes the problem, without posting it publicly?

[patrick]

I just sent you a PM with instructions on how to upload the files.  Let me know when you've got everything uploaded.

[rygaard]

I've uploaded the file.

[patrick]

I should hopefully have fixes for you sometime this morning.  I think your html makes it pretty clear where the problems are, thanks for providing that. 

[patrick]

I'll refer just to the section names from your posted data.

Wildcard handling
This is the big one, that plus the unwanted recursion is creating unwanted and invalid manifest entries.  I suspect this may be the first time we've run into a wildcard on the CLASSPATH environment variable, or we've just been lucky and it didn't obviously break the user's project.  Fixed by the attached projconv.e.  I didn't tally up the size of the valid entries, but it would not surprise me if it put you back under the limit for classpath size where it won't even use the jar manifest anymore.

Handling the ____\*.jar entry
This one is interesting.  The *.jar wildcard isn't expanded into individual jars because of the (unwanted) recursive handling, but because some clients of the functions that create this classpath want the expanded version with no wildcards.   It would be nice to leave that as a wildcard when we can, since it will noticeably shorten classpaths.  But it may be too big of a change for a hotfix.  Not sure yet, so not handled by the attached projconv.e file.

The JAR is not being saved
In this particular case the jar wasn't created because of the manifest exception.   The logging code blindly printed out the saved message, even though the jar wasn't there, fixed that.

The manifest file is not being saved
It wasn't saving the manifest file, but just copying it into the Output window.  That header line was supposed to come before the manifest file, but I incorrectly used an editing function whose behavior depends on the user editing configuration.  I fixed the logging, and changed it so it doesn't delete the manifest file in that case.  Included in the attached projconv.e

The problem with all wildcards being recursive is not fixed in the attached projconv.e.  Still working on that.   For that one, there are enough changes where I might choose just to generate the hotfix file for that (including all the other changes), to avoid a bunch of manual module loading.

[rygaard]

I think your html makes it pretty clear where the problems are, thanks for providing that.

I got a little carried away... 

[rygaard]

Cool!

I tried the new projectconv.e, and it is much closer!  The wildcard in my system CLASSPATH variable is including JARs in subdirectories, which does not follow the Java standard, quoted in the HTML document I sent you.  It should just include JARs in the indicated directory, and not in subdirectories.

Also, the classpath is an order of magnitude shorter!

[patrick]

I did mention the recursive wildcard searching isn't fixed yet.  I'm still working on that, should be done today.  I've had to change some signatures, so I've ended up touching more code than I would have guessed from the bug description.

[rygaard]

Yes, I should have let you know that there are no longer any roadblocks for me in SlickEdit regarding the classpath.  Including subdirectories is incorrect, but not a problem for me in my project.