Compressing the Lucee 5 jar with pack200

#1

Here is a conversation I started with Micha on Slack that I wanted to be able to document and organize somewhere so I’m moving it to Discourse.

=============================================

On the topic of size again… I noticed that my CommandBox 4 builds are twice the size as CommandBox 3. (note the other day I was talking about the size of the Lucee server context after it starts up, here I’m just talking about the size of the box.exe binary itself). We use pack200 to highly compress all of the jar files in CommandBox and at first I thought I just forgot to run that process, but tested it today and realized that pack200 is running but it’s just not doing anything to the Lucee jar! It turns out pack200 will only compress the bytecode parts of the jars, but the way we’re building the Lucee 5 with all the extensions, bundles, and the lco file added into it, pack200 doesn’t touch any of those binaries. This is a big issue for us as it’s made the box.exe go from ~37 Megs to ~75 Megs since the way Lucee jar is built keeps pack200 from being able to work.
I need some way I can unpack the Lucee jar in our CommandBox build process so that all the jars can be compressed correctly via pack200 to get my smaller binary file sizes back. Can you help?

One of Micha’s replies was:

what you also can do is removing all jars from the lucee.jar and make sure they are added to the bundles folder when starting up.

My unanswered followup questions were as follows:

This is a good idea. What about the extensions? They are basically just archives in jar files. is there a way to compress those?
Also, what about the big lco file? Isn’t that basically just a jar?
Would it work to extract it, rename it to a jar, pack200 it, and then on first run, rename it back to an lco file? Does it have to be inside the lucee.jar or is there a way to have it already in the server context?

in another thread, Micha mentioned this:

as a side note, Lucee itself supports pack2000 internally for jars, so maybe in the future we can make a lucee pack2000 jar, that of course has bundled only pack2000 bundles. the bundles bundled in the lucee jar (at /bundles) can be pack200. extension need to be .pack.gz
this is an experimental feature i did never had the time to test

#2

So here is an overview of what’s bundled inside the Lucee 5 jar:

  1. Class files under railo, org, lucee, com, coldfusion folders
  2. core.lco file under the core folder, which is basically just a jar with another file extension. The core.lco file appears to only contain classes.
  3. Around 60 jar files (OSGI bundles) under the bundles folder containing class files
  4. 10-15 lex files (Lucee Extensions) in the extensions folder. lex files are basically zip archives, most of them containing a jars folder with one or more jar files containing class files.

So from my understanding of how Pack200 works, it only packs the class files contained directly in a jar and it ignores any other nested binary files. That means that #1 is the only thing that gets compressed by Pack200, which is only a very small portion of the over jar size.

So for #2, how can we compress it? Can I remove core.lex from the jar in my build, run it through Pack200 and put it back in? Can Lucee handle the core lco file being packed? Would it require a new file extension?

For #3, to confirm, it sounds like you’re saying I can manually take the jars out of the bundles folder in the jar and run them through pack200, leaving them with the file extension pack.gz. Can I put them back inside the Lucee jar like that and they will be used like normal and unpacked on the fly?

Now #4 is a bit more complicated because the jars are now 3 levels deep inside the Lucee jar. There are jar files inside of lex files inside of lucee.jar. Can the jar files in an extension be packed as well? Do they follow the same naming convention of .pack.gz and can I zip them all back up in the original lex file they came from and everything will still work?

#3

OK, I have reworked the CommandBox build process to disassemble every piece of the Lucee 5 jar, pack 200 all the jars at every level and put it all back together and then pack200 the entire jar. The great news is, the Lucee 5 har has gone from 70 Megs down to 37 Megs! The bad news, it doesn’t work :confused: Apparently Lucee is only halfway pack200 compatible. Here’s the rundown from the list above:

  1. Running pack200 on the main lucee jar works, but we already knew this since it was already happening.
  2. Running core.lco through pack200 reduces it in half (10 megs down to 5 megs) however this completely stops Lucee from starting. Lucee does not seem to look for a core.lco.pack.gz file. Can this be added as an enhancement?
  3. Packing all the OSGI bundles seems to work great. Lucee picks them up and uses them. This provides a modest 14 Meg loss in file size (56 Megs instead of 70 megs)(
  4. Breaking open the extensions, packing their jars if they exist, and rezipping does not cause errors, but I don’t think it works because I see a ton of network traffic and Lucee apparently re-downloads all the bundles for each extension plus I also see an error in the console regarding a bundle that can’t be downloaded. It seems that extensions don’t allow for pack.gz files either. Can this be added? Packing the extensions’ jars loses another 11 megs or so.
#4

On a side note, the Lucee Lite jar packs down to 16 megs with no extensions, which means if we can get core.lco packed, I think I could get Lucee down to an 11 Meg jar. This means a basic non-server version of CommandBox purely for distributable CLI stuffs could be as little as 15 Megs for the entire box.exe! Now THAT would put us in the same ballpark as a Go binary which is normally about 8-9 Megs.

#5

as you pointed out Lucee only cares about bundles bundled with the jar if they are pack200. Sure we can improve Lucee on this and make it also look for pack200, but if we do this, we also should compress all bundles with pack200 by default.

#6

Yes, I would like very much for the Lucee jar to make 100% use of Pack200. The CommandBox download sizes have suffered considerably due to this. Now that I’ve gone through all the work of converting the CommandBox build process to apply pack200 to everything I guess I don’t care one way or another if it’s done by default. (I have the parts of the build that compress the core.lco and extension jars commented out for now until Lucee can support it fully.

#7

Support core.lco to be compressed with Pack200
https://luceeserver.atlassian.net/browse/LDEV-1922

Allow jars in extensions to be compressed with Pack200
https://luceeserver.atlassian.net/browse/LDEV-1923