SE 23.0.2 - Java import disorganized

[Marcel]

SE's automatic addition of Java import statements, as well as the "Organize Imports" function have some problems.

• The sorting within an import group should be alphabetic. In the attached project, the import for AtomicInteger is misplaced
• The automatic import really should resolve packages from the provided classpath. It should be able to find LogFactoryImpl
• Organize Imports discards import statements it cannot resolve. Again: LogFactoryImpl
• A wildcard import statement [org.apache.commons.logging.impl.*] generates an error message during Organize Imports

[Dennis]

I will look into this.

[Dennis]

This should be fixed in the next hot fix for 23.0.2.

The sort was case-sensitive.  I changed it to case-insensitive, but this could generate complaints, if so, I'll have to undo this change in a future hot fix.

The organize import / auto-import code is built on our context tagging.  If you want context tagging and symbol coloring to work, it is best to set up your project so that 3rdparty jar files are tagged (usually by creating a sub-project that contains the jar files).  We have had many requests to tag jar files in the class path, but this would be very tricky to implement, and java-specific, plus it could result in tagging a lot more than is necessary.  We're still considering this feature for a future version, but it isn't all that simple.

The error message you mention is correct.  It informs you that there are classes referenced in the source code that we have no idea where they came from.  I fixed the bug where it discarded import statements it could not resolve, but you should expect to get this error message still, even if you are not using a wildcard spec, it just will not discard unused imports in this case now.

[Marcel]

We have had many requests to tag jar files in the class path, but this would be very tricky to implement, and java-specific, plus it could result in tagging a lot more than is necessary.  We're still considering this feature for a future version, but it isn't all that simple.

My Java projects have rapidly gotten more complex. Most everything is now being built using Gradle. This means that external jars are referenced out of the Gradle cache, and/or subproject build artifacts. Gone are the days when I had a nice little set of jars that could be tagged separately. Now, their composition and contents can change from day to day.  I can still use SE for most of my bulk coding work, but needed to extract material out of Gradle artifacts to build temporary SE projects in order to edit, compile and do simple debug runs. When things get complicated, I call on big brother IntelliJ. It has excellent indexing, so finding references and adding imports, debugging into external libraries and running unit tests works without a hitch.  My bottom line is not to have bloated expectations of what SE can do, and use it for what it excels in, then switch toolbox if a bigger hammer is needed.

[jporkkahtc]

Yes, that exactly -- the java projects I work on tend to have *tons* of libraries from either the maven cache or gradle cache (and sometimes both).
Trying to manually setup Slick to index these is not easy at all - and then you have to manually update with new versions of the libraries. When I have tried this, slick seems to not have a problem with the overall size or number of libraries.

Also sometimes you have the normal library and another with the source -- slick should recognize this and ignore the binary and use the source library.

[rowbearto]

I'd like to add my voice to the chorus for the desire to have my jar files in the maven cache automatically get indexed without my needing to explicitly add them. If Intellij can do it then why not SE? This seems like a killer feature to make folks use Intellij instead of doing Java on SE.

[patrick]

For v24, the Gradle support will give the option to tag dependencies.  You can select which jars to tag, if you want to exclude some of the dependencies.  It tries to use source jars, but falls back on binary jars for dependencies we couldn't find the source for.  Real time errors for Java should work with it, though I haven't written the part that makes sure RTE gets the right classpath yet.  This support will probably be in the first beta for v24.

I'm not sure if I can do the same with Maven for v24.0.0 yet.  I think we're lacking some support code for Maven that I took for granted with Gradle, so I may have more work to do before I can even get on the dependencies.  I'll take a closer look when the Gradle support is finished.

[patrick]

Well, Maven makes extracting the dependency list and pulling sources for dependencies very easy.  So I'm going to say that at the very least it will be able to optionally add dependencies for tagging and RTE for java projects for v24. 

[rowbearto]

Thanks Patrick! This is wonderful! +1!

[Marcel]

This sounds promising.  Thanks Patrick.

If I may, I would like to pile on my personal wishlist:

• The external library indexing support is not going to be free, project initialization will take much more time, so making as much of is selectable would be wise.
• If you could mimic  Intellij a bit more: During a projects' Gradle refresh, it will not only prefetch the dependent jars, but also their sources (if available) into the Gradle cache (the cache has 2 jars). To look up definitions, it either dives into the source, or presents a (de-compiled?) version of the object code. Either way, you now have method signatures to work with.
• I have no need for a global reference index for the material in the Gradle cache.
• You could probably come up with an "index on demand" scheme, where the project refresh gets a coarse look at the available packages, and maybe does a deep dive only when needed.
• The index for the cached material is independent of the project, so that it can be used globally (pretty much like the compiler tag files).
• I have a lot of projects with subprojects.  I would prefer to have a single root set of *.v* files for it instead of having them spread throughout the subprojects.  This would probably also solve the scope problem you currently have.

Thanks

[patrick]

(edited for a typo)
Just a couple of quick comments on some of the items.  I haven't thought them through yet.

• The external library indexing support is not going to be free, project initialization will take much more time, so making as much of is selectable would be wise.

By project initialization, are you talking about when the project is first imported, or reconfigured through the "Gradle Options..." menu item? That can certainly take some more time when Gradle is cranking through the dependencies, or having to download binary and source jars.  You can select which dependencies get seen by the tagging, so that can reduce the time to pull source jars.  (and it remembers your selection so you don't have to re-check everything when you need to reconfigure the project to pick up updates in the dependencies).

It shouldn't effect startup times when you open the project outside of the extra time to load the tag file, which will be larger once you add dependencies.  I haven't noticed a difference with projects with >100 dependencies so far, but I'll circle back and see if I can measure a difference before the v24 beta.

• If you could mimic  Intellij a bit more: During a projects' Gradle refresh, it will not only prefetch the dependent jars, but also their sources (if available) into the Gradle cache (the cache has 2 jars). To look up definitions, it either dives into the source, or presents a (de-compiled?) version of the object code. Either way, you now have method signatures to work with.

v24 does download the source jars for tagging now when it can locate them, and will fall back on binary jars when we can't get the source jars.  Jars of class files are still tagged so we know about the types and methods defined in them, but if you push-tag a type or method defined in a class file, we still just pop up a message box saying you can't go to a tag defined in a class. We don't have a special view that just shows a skeleton of the class file, like you'd get from say javap. 
 

• I have no need for a global reference index for the material in the Gradle cache.

I'm unsure what you mean with this one.  One of the later items seems to want the index to be global.  Are you saying the dependency jars shouldn't be tagged with references? (ie, the tagging feature that adds entries to the index just for the purpose of making reference searches faster, at the cost of tag file size).   That makes sense to me.  If I'm doing a reference search in my project for 'log', I don't really care about how many times that's used in log4j.  I don't think we make that distinction now, but I'll take a look at it later, I might be wrong about that.

[Marcel]

Sorry about the confusion on that last one.  Yes, that is exactly what I meant. 

I don't think I am interested in having cross-references within the cache, so the tag file could maintain definitions only (with the goal of reducing resource consumption).  Of course, I am interested where in "my code" log4j is used, but not so much if one of the dependent Apache commons jars is using it.

As far a regenerating the Gradle config goes, that needs to be done whenever the dependency set changes. I think initially this can be done manually by rerunning the setup, or maybe a faster variant that just refreshes dependencies.

[patrick]

Support for tagging the source of dependencies for Gradle and Maven will be in the next v24 beta, beta 2.  (Limited support is also in beta 1, but there's a bug where the selected source dependencies could be lost.  Don't recommend it).

• Source dependencies can be set up in the "Build -> Gradle/Maven Options" page for existing projects.  For Gradle projects, the dependency setup is also part of the new project wizard.
• Source dependencies selected by the user are grouped underneath their own project in your workspace, with the tags going into a project-specific tag file. This project tag file is tagged without references by default.
• There is nothing special about the project created for dependencies.  If you have any other jar files or sources you want to add outside of the ones automatically added by the project code, you can add those files there too.
• Related: Added a per-project tagging option, "Tag files with project specific tag file, without references". This is at the bottom of the Project properties page, in the Files tab. We default to this for dependency projects. It can be used for any project.
• Dependency resolution can be costly for projects that have a large or deep dependency graph, so we don't update dependencies automatically, only when requested by the user via the Maven/Gradle options page.

Beta 2 will also fix a hang in importing some Gradle projects that turned up in beta 1.

[Marcel]

This thread should probably be moved .....
I just started exploring the new features you've added.

• Adding a source jar file to the External Tagging Dependencies and then generate a tag file worked fine.  I was able to find definitions.
• In the Gradle setup I always have to uncheck the Gradle wrapper check box. It doesn't seem to remember my preference
• The build option (Ctrl-M) doesn't work.  SE keeps feeding vsbuild this: \bin\gradle.bat Build.  Two things wrong with it:  The path should be %GRADLE_HOME%\bin\gradle.bat, and the task should be "build" (case) and shouldn't  be hard-coded but taken from the gradle setup
• I had a tussle with the Java compiler tags.  In an earlier experiment I was using Java12, then dropped back to Java8. For some reason it continued wanting to pull tags from the nonexistent Java 12 tag file. Finally went to "add a new compiler" and removed the Java 12.
• There are still some inconsistencies with the dependent jar handling in subprojects.  SE now finds 3rd-party jars (nice!), but can't find definitions for material in other subprojects, even though they are part of the overall project.  But overall, what we currently have is already a big improvement

[patrick]

You can go ahead and open another thread for this on the beta forum. Probably best not to move this one so the original opener of the topic doesn't keep getting notifications.  We can just put a link to this thread in the new one, so it won't be confusing looking back later.

Losing the check state for the gradle wrapper: is there a gradlew or gradlew.bat in the root of the project?  I think we might be seeing that file and assuming it should use the wrapper.  I'll look into that.

For the sub-projects you're not picking up source for, has SlickEdit created matching .vpj files for those sub-projects?  It may be there's a bug in how we detect and set up those projects, but it's not something I've run into with the test projects I've worked with.

That failure to build is odd.  There may be a failure to pick up from GROOVY_HOME in the setup, but it should prompt in the case that it thinks there is no groovy home is set.  There were some problems in the past with tasks having hard coded parts (or depending on gradle being on the path), but running Gradle Options on the project should have forced the tasks to be updated.  The task names are case insensitive.

I'll have to look into the compiler tagging.  Changing the default compiler in the Java Compiler properties page should change up the tag files used for lookups, so there may be something wrong there.

Probably the easiest thing to start with is to collect some files from you.  In a zip or tar, or whatever, could you pack up the following things:

• Your user.config.xml file that's in your configuration directory.
• Your workspace .vpw file
• The .vpj file for one of the projects that has that build problem.
• Go to "Build -> Gradle Options..." and step through the wizard pages again.  Once that's done, there will be some output I can use for debugging down in the Output tool window.  Right click in the Output window, and "Save As...", and save it to a file like output.log, and put that in the archive.
• And your settings.gradle from the root of your project.

We'll PM you with the instructions on how to upload the archive to support.  Create a new thread in the beta forum to let me know when it's uploaded, and we'll go from there.

Thanks for the detailed report.