Author Topic: v19.0.2.5 vsbuild does not compile all java classes  (Read 5173 times)

Marcel

  • Senior Community Member
  • Posts: 248
  • Hero Points: 26
v19.0.2.5 vsbuild does not compile all java classes
« on: May 09, 2015, 03:39:38 pm »
In a Java project with around 1300 source files I added a new unit test class. It showed up in the project tree alright, got saved when I hit build/rebuild, but it was never being compiled.  Inspecting the temp args file built by vsbuild showed that the list of source files was about 80 or so files short of what I expected it to be.

The next step was to remove source files from the project then retrying 'rebuild' until the class in question was being compiled (that was noticeable because it had a syntax error). After the project source file count was down to around 1210, vsbuild finally compiled my new class.  Adding one of the removed source files back in caused the file to be ignored again.

This does not look like a file count problem because the number of files in the temp argument list files varied depending on which source files were removed from the project.  Probably a buffer size limitation somewhere in or around vsbuild. The temp arg list files sizes are in the 120-124KB range.



Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 5735
  • Hero Points: 458
Re: v19.0.2.5 vsbuild does not compile all java classes
« Reply #1 on: May 09, 2015, 07:35:01 pm »
It does sound like some sort of limit. It's hard to say where the limit is. I don't think it's vsbuild but it could be either a windows limit or a cmd.exe limit. We've definitely had issues with cmd.exe (not sure if is used by vsbuild in this case).

Phil Barila

  • Senior Community Member
  • Posts: 742
  • Hero Points: 61
Re: v19.0.2.5 vsbuild does not compile all java classes
« Reply #2 on: May 10, 2015, 12:06:13 am »
Visual Studio (&MSBuild) avoids this by writing temporary command files containing the file list (and other stuff) and passing that into the compiler.

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 5735
  • Hero Points: 458
Re: v19.0.2.5 vsbuild does not compile all java classes
« Reply #3 on: May 10, 2015, 12:55:39 am »
We do that for C++. Not sure if vsbuild does that for Java. We will take a look.

Marcel

  • Senior Community Member
  • Posts: 248
  • Hero Points: 26
Re: v19.0.2.5 vsbuild does not compile all java classes
« Reply #4 on: May 10, 2015, 01:06:56 am »
vsbuild generates an arglist file when calling the java compiler. You can see that when you toggle the verbose flag in the build/rebuild tool menu.

I built a small repro test case but didn't have any success. vsbuild handled a project with 20K source files just fine.  The arglist file grew to 1.2MB.
I am going to rebuild the project that is giving grief from ground up. I had it for a few years, maybe there's something hosed in the vpj. I'll let you know how that goes.

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 5735
  • Hero Points: 458
Re: v19.0.2.5 vsbuild does not compile all java classes
« Reply #5 on: May 10, 2015, 04:54:04 am »
vsbuild has an undocumented "-k" (or "-keeptempfiles") option. I'm pretty sure this will allow you to check the contents generated in the temp list file after the compile finishes. Otherwise, it's always deleted.

Marcel

  • Senior Community Member
  • Posts: 248
  • Hero Points: 26
Re: v19.0.2.5 vsbuild does not compile all java classes
« Reply #6 on: May 10, 2015, 12:58:54 pm »
Recreating the project didn't help, it still doesn't compile all the files.

I made a batch file that just launches vsbuild with the "rebuild" command and the -k option as suggested. The ordering of the files listed in the argsfile looks like something a dependency tree dump would produce. If I had a hint which file should have been added to the list NEXT, I would be closer to find out where things break apart. I'll open a support case and see if I can get a vsbuild version with a bit more logging.

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 5735
  • Hero Points: 458
Re: v19.0.2.5 vsbuild does not compile all java classes
« Reply #7 on: May 10, 2015, 01:38:55 pm »
Given the size of your project, the log info would be extremely large. It would have to dump the dependencies for each file at least.

Does a rebuild compile all files?

Marcel

  • Senior Community Member
  • Posts: 248
  • Hero Points: 26
Re: v19.0.2.5 vsbuild does not compile all java classes
« Reply #8 on: May 10, 2015, 04:53:34 pm »
Case filed: CAS-67148-6934

@Clark:
Lots of output is OK, plenty of disk space around and I am quite handy with Perl  ;).

No, rebuild doesn't compile all files, and I took the usual precautions, clearing out the classes and deleting the vtg, starting with a clean -sc.

Some more tidbits that might help SE support building the debug vsbuild.
  • Contrary to what I said at the outset, I don't think vsbuild just stopped walking through the list of source files. Rather, it went all the way through, but just skipped a few while doing so. This is based on a comparison between the two arglist files (success, failure) which both end with the same file
  • The files that didn't get compiled are all unit tests, otherwise the problem would have surfaced a lot earlier. I run most unit test suites outside of SE.
  • Unit tests are kept in a separate physical directory than the classes they exercise; Same package, separate physical path
  • I have no idea how vsbuild does dependency checking, but UTs are roots and and don't have an explicit reference (import) to the class they test (same package path)

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 5735
  • Hero Points: 458
Re: v19.0.2.5 vsbuild does not compile all java classes
« Reply #9 on: May 10, 2015, 07:42:52 pm »
If you remove all .class files and build still doesn't create a .class file for every .java file in your project, that means that we don't need dependency information. When the .class file doesn't exist, that .java file must be compiled. All other checks are not needed.

Do you really have some .java files which never get an associated .class file? The dependencies are for fancy stuff (.class files kinda treated like header file dependencies). I thought something was going wrong with the fancy dependency stuff. This sounds much simpler.

Marcel

  • Senior Community Member
  • Posts: 248
  • Hero Points: 26
Re: v19.0.2.5 vsbuild does not compile all java classes
« Reply #10 on: May 10, 2015, 08:24:46 pm »
I agree with you. For the rebuild case, vsbuild has the seemingly simple task of collecting all the *.java file names in the project, minus exclusions, then feeding them into the compiler via the argsfile.
Alas, it doesn't do that.  The file names in the argsfile are all jumbled, not the kind of ordering you would expect from a  bunch of directory traversals, and the list is incomplete.

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 5735
  • Hero Points: 458
Re: v19.0.2.5 vsbuild does not compile all java classes
« Reply #11 on: May 10, 2015, 08:57:35 pm »
You added the files to the project statically (not a wildcard) right? That means each .java file is listed in the .vpj file (no *.java in the .vpj file).

Marcel

  • Senior Community Member
  • Posts: 248
  • Hero Points: 26
Re: v19.0.2.5 vsbuild does not compile all java classes
« Reply #12 on: May 10, 2015, 11:32:05 pm »
Nope - wildcards as far as the eye can see, for both the source files and the jars. 

This makes it possible to put the vpj/vpw under source control without having to mess with them every time files get added or removed. Of course, I have also learned to clean up vtg and classes directories when the project files inventory changes.

Clark

  • SlickEdit Team Member
  • Senior Community Member
  • *
  • Posts: 5735
  • Hero Points: 458
Re: v19.0.2.5 vsbuild does not compile all java classes
« Reply #13 on: May 11, 2015, 02:53:59 am »
In order to reduce the possible issues here. Try creating a project without wildcards and see if that builds. If it does build, that means vsbuild is failing to list all the wildcard files. If that is the problem, I would expect vsbuild to miss the same files all the time.

Marcel

  • Senior Community Member
  • Posts: 248
  • Hero Points: 26
Re: v19.0.2.5 vsbuild does not compile all java classes
« Reply #14 on: May 11, 2015, 03:49:09 am »
Recreated the project without wildcards.

The project contains 1307 java source files, the vsbuild arglist contains 1230 of them. The missing files are all from a directory tree containing unit tests, files are missing from various directories of that tree, i.e. they do not seem to be clustered further.