Lucee and its relationship to OSGi bundles

I’ve noticed that Lucee’s downloads page has a number of issues:

The first on is the rather hap-hazard versioning. Many extensions on the page have the latest release first, but postgresql has the latest release last.

The second is that the latest versions of the bundles are also not representative of what’s on Lucee’s Maven repo. For instance, EHCache on the Lucee downloads page shows 2.10.0.31 as the latest version while maven shows it at 2.10.4. Which is it?

The third issue is that some extensions aren’t even listed on the Lucee download page. For instance bouncycastle is available on maven.

The final issue I’m noticing is that the downloads are broken. It looks like Lucee merely proxies the download requests straight to maven’s repo via HTTP and not HTTPS. Maven has (as of this month) removed HTTP-only support.

I’d like to know: What is the purpose of Lucee hosting these files? Do they have differences from upstream releases? I’d also like to know why so many of them lag so far behind: Bouncycastle is years old and upstream has since issued security updates.

2 Likes

Friendly bump. The Lucee extensions downloads are still broken with regards to maven redirects.

@zingyb I would enter tickets in JIRA. The Lucee dev team doesn’t really use these forums. I’ll add I reported issues with the update servers download of OSGI bundles over a month ago (Jan 23rd) on an internal Lucee Slack team and I still have yet to even get it looked into. Other than putting in tickets and waiting, I’m afraid there’s no workflow in place to deal with these issues from what I can see, even though they affect production servers.

cc/ @IamSigmund

I think I can clarify some of this for you; at least I’ll give it a shot :wink:

The Lucee Downlaods page lists extensions (.lex files), along with the versions of those extensions. To use your example of ehcache;

What is listed here are the versions of the extension itself, and while the extensions might have similar looking version numbers to the dependencies that they use, they are not the same thing (nor can they even be, especially if an extension needs a bugfix but the underlying dependency versions don’t change). Perhaps that’s where some of the confusion stems from, it could be using it’s own version numbers like “1.31” instead of “2.10.0.31”.

The extension repo is found on GitHub where you can see the corresponding Releases;

(Note: It appears that the newest “2.10.0.31” doesn’t have a release on that GitHub page, I assume that a Git tag has been missed.)

The extension has a dependency on the “org.lucee.ehcache” bundle in Maven as you can see from the manifest Require-Bundle: org.lucee.ehcache;bundle-version=2.10.3 below;

So the JAR files in Maven are just the dependencies used by the extension and not the extension itself. Sometimes the version number might look very similar, sometimes they might not.

In the case of “bouncycastle”, there’s no Lucee Bouncy Castle extension but that bundle exists because the S3 extension uses it (and perhaps others did at some point in time);

As for Lucee extension downloads failing due to redirects, I don’t have any information on that but can you provide some concrete examples? If we have a set of steps to reproduce the problem then I’m sure someone can look at it – and that’s the perfect info to put into a ticket.

1 Like

Thanks for your response!

What is listed here are the versions of the extension itself, and while the extensions might have similar looking version numbers to the dependencies that they use, they are not the same thing (nor can they even be, especially if an extension needs a bugfix but the underlying dependency versions don’t change).

I’m confused. The PostgreSQL driver on the lucee page shows versions schemes that line up with upstream’s. Same with HttpComponents, ehcache, etc.

Typically, packaging software that requires downstream releases (e.g. Linux package managers) append extra information at the end, so it’s very reasonable to keep the versions in sync. For instance, my Bash version as packaged by my distro is 5.0.011-2, signifying that the packager pushed a change for the same version.

The bigger picture question is… What is the purpose of most of those .lex files in any case? With Lucee’s ability to run OSGi bundles the downloads page merely looks like an unloved re-hosting of various components! AFAICT there is no LTS version of Lucee; Only the latest version is supported. If that’s the case, why are .lex files still distributed when OSGi counterparts exist? My best guess is that it facilitates the one-click installs from the lucee admin but I argue that the ecosystem is made too complicated by forking these components (and in the case of e.g. the S3 extension’s dependence on an ancient bouncycastle, insecure!). The Lucee devs are great at making Lucee; they shouldn’t be in the business of packaging software.

As for Lucee extension downloads failing due to redirects, I don’t have any information on that but can you provide some concrete examples? If we have a set of steps to reproduce the problem then I’m sure someone can look at it – and that’s the perfect info to put into a ticket.

[~]$ curl "https://release.lucee.org/rest/update/provider/download/org.lucee.jets3t/0.9.4.0006L" -i
HTTP/1.1 302 Found
[...]
Location: http://repo1.maven.org/maven2/org/lucee/jets3t/0.9.4.0006L/jets3t-0.9.4.0006L.jar

[~]$ curl -I http://repo1.maven.org/maven2/org/lucee/jets3t/0.9.4.0006L/jets3t-0.9.4.0006L.jar
HTTP/1.1 501 HTTPS Required
[...]

While I can individually report bugs for broken downloads, I’m more interested in discussion of the extension ecosystem as a whole - Bug tracking systems are typically not an appropriate avenue for that. I’m not sure of a better place to start this discussion than Lucee’s own dev forum. :slight_smile:

I’m confused. The PostgreSQL driver on the lucee page shows versions schemes that line up with upstream’s.

Despite what Justin said, from my understanding, all Lucee extensions that are based heavily on a specific external Java library have a version that simply matches that of the bundled library. This is really just done for convenience since it would be difficult to remember which version of a JDBC driver bundled which actual version of the Java library.

Same with HttpComponents

There is no HTTPComponents extension so I’m not sure what you mean by that.

Typically, packaging software that requires downstream releases (e.g. Linux package managers) append extra information at the end, so it’s very reasonable to keep the versions in sync. For instance, my Bash version as packaged by my distro is 5.0.011-2 , signifying that the packager pushed a change for the same version.

You’re correct that is common, but I don’t think Lucee adds anything. Most upstream Java libs already have 4-party version numbers and Java libs don’t usually use a hyphen unless it’s a release identifier like -RC or -snapshot. So far as I know, there is no plan in place for how to deal with a second or third build to a Lucee extension that bundles the same upstream lib. This was all architected by Micha who doesn’t monitor this forum. I realize you don’t want to put in tickets, but you’re only going to get people guessing as to what Micha’s design was since no one else really knows or was consulted about it.

The bigger picture question is… What is the purpose of most of those .lex files in any case?

I believe you have a faulty understanding of what an extension is. This is like asking what the purpose of a McDonalds value meal is when you can just purchase a burger directly. An extension is just a package that can (or cannot) have a number of things

  • zero or more OSGI bundles
  • zero or more plugins to the Lucee admin
  • zero or more custom functions
  • zero or more custom tags
  • zero or more Cache drivers for the admin
  • zero or more JDBC drivers for the admin

So while a Lucee extension (which is bundled as a zip file bearing a .lex file extension) may have OSGI bundles inside of it, extensions serve a grander purpose than simply being a bundle.

With Lucee’s ability to run OSGi bundles the downloads page merely looks like an unloved re-hosting of various components!

How does the fact that Lucee uses OSGI bundles relate in any way to the download page for extensions? These are unrelated concepts. Also, what exactly is “unloved” about the download page?
Are you saying the contents look old? Or unsightly? I’m not following.

AFAICT there is no LTS version of Lucee; Only the latest version is supported.

Depends on how you look at it. The community is free to “support” Lucee for as long as they wish. It’s open source. LAS doesn’t support Lucee at all strictly speaking. You put in tickets and you just hope they get worked on. Companies such as Rasia provide paid support for pretty much any version of Lucee so long as you’re willing to pay for the fixes.

Now, as far as which versions get updates, Lucee does usually have more than one iron in the fire. For example, the 5.2 branch has gone dark, but the 5.3 branch just released 5.3.4.80, and there’s a release candidate out for 5.3.5 and also snapshot builds already present for 5.3.6. Case in point: 5.3.4.77 recently came out and some regressions were addressed in it which led to the 5.3.4.80 release even though 5.3.5. and 5.3.6 are already under development. So no, commits are not only added to the very latest version.

If that’s the case, why are .lex files still distributed when OSGi counterparts exist?

It’s not the case, but to answer your question-- for the same reason McDonalds still sells value meals given the fact they also sell fries and burgers. Extensions are a package of things which may or may not include any ODSI bundles.

My best guess is that it facilitates the one-click installs from the lucee admin

One click installs from the admin are great for extensions, but that’s not really why they exist. See above.

but I argue that the ecosystem is made too complicated

Software development is complicated. I don’t personally see any unnecessary complications here. Extensions tend to bundle self contained features. These features are often built upon a 3rd party jar. These jars are converted to OSGI bundles if necessary and packaged inside the extension they power.

by forking these components

Who’s forking anything? When Lucee bundles a 3rd party java lib, they don’t have a fork of it. They simply include the jars.

(and in the case of e.g. the S3 extension’s dependence on an ancient bouncycastle, insecure!).

Outdated dependencies are something we have to manage regardless. If you’re concerned about the version of bouncy being used in the S3 extension, please search te bug tracker and enter a new ticket if there isn’t one. If it’s incredibly important to you, hire Rasia and pay for it to get updated or submit a pull request for the update.

The Lucee devs are great at making Lucee; they shouldn’t be in the business of packaging software.

On what grounds? Lucee is inherently a modular architecture, made to have a light core and a collection of ancillary extensions that augment the functionality of the core. The very nature of this demands some sort of packaging and installation process.

HTTP/1.1 501 HTTPS Required

I actually thought I went through and removed all the HTTP URLs from our data provider the other day. Perhaps I missed one. The code is here, feel free to track it down and send a pull. You can spin up this site as a local REST API pretty easy with CommandBox, but you do have to create the REST mapping manually:

Note Lucee’s server doesn’t host these files, it just serves as a redirect to the actual Maven repo so Lucee can “auto-download” bundles if they are required but not installed. I’m not a huge fan of that functionality, but it is what it is I suppose.

While I can individually report bugs for broken downloads, I’m more interested in discussion of the extension ecosystem as a whole - Bug tracking systems are typically not an appropriate avenue for that. I’m not sure of a better place to start this discussion than Lucee’s own dev forum.

Right, the discussion part is fine. But anything that’s broken needs a ticket. As Micha says, if there’s no ticket, the bug doesn’t exist! Also, as soon as you get too far off in the weeds of “why was it designed this way” then you get to questions that really only Micha can answer. And the only place I’ve seem him speak “publicly” in years is in JIRA.

I just dug around in the provider source and found the last remaining reference to http://repo1.maven.org and changed it.

I also updated all the cached JSON indexes on the server. The URL above works. See if you have any other download issues.

1 Like

Most do match (as I acknowledged) but not all, specifically ehcache in the example above?

Lucee Extensions (.lex files) aren’t the same thing as OSGi bundles. Have a look at the lucee/extension-* repos on GitHub to see their source code and examples of the things they include and that might help explain why the Extensions on the download page are still important.

FYI there are some hardwired maven mappings in the update provider

Thanks for the response.

My attempt is to understand why old, unmaintained .jar files are being distributed underneath these .lex files, so I apologize to have conflated the existence of .lex files with these assets. I understand that there’s a desire to extend Lucee at the same time as providing external functionality. Bundling the outside functionality certainly does make it easier short-term.

From my vantage point, it’s as if a McDonald’s value meal comes not only with its own goodies but last week’s scraps from Burger King. What necessitates the redistribution of these .jars? If one uses a .lex file, expectations of certain libs can be fulfilled by the administrator.

Depends on how you look at it. The community is free to “support” Lucee for as long as they wish. It’s open source.

One wouldn’t want to run Ubuntu 19.04 - a recentl EOL version - based on the premise that “someone might come along and push a fix”. The word “supported” under a release number is important as it assures a project intends to support that release branch with certainty. I know that Ubuntu 19.04 is dead because there’s a clear maintenance schedule. In the absence of any such clarity, I can only assume that Lucee only supports the latest versions and none earlier. This is another discussion, admittedly, but since OSGi support is somewhat newer to Lucee I looked to supported Lucee versions for information.

Who’s forking anything? When Lucee bundles a 3rd party java lib, they don’t have a fork of it. They simply include the jars.

You are correct: “fork” is the wrong word here: “Bundle” would have been more appropriate.

On the grounds that they are unable to keep up with the updates of the bundled within years, even when CVEs are issued for the corresponding projects. If code is being bundled with the project then I would expect the development team to keep an eye on that code just like any other facet of the project.

The better solution is to not bundle all. If I want a jdbc driver for PostgreSQL, I should head over to Postgres’ website to get it.

I understand the desire to extend Lucee alongside these outside libraries, but I argue that bundling them is not correct.

Am I correct in interpreting this as dev.lucee.org only being a support portal and that discussion of Lucee and its architecture does not have any formal place other than tickets in Jira for one developer to view?

My attempt is to understand why old, unmaintained .jar files are being distributed underneath these .lex files

…they are unable to keep up with the updates of the bundled within years, even when CVEs are issued for the corresponding projects. If code is being bundled with the project then I would expect the development team to keep an eye on that code just like any other facet of the project.

This is an excellent question, and one I can only speculate on an answer. I think it boils down to a severe lack of development power available to LAS. Development is driven primarily from the dues of paying Lucee members (a great reason to be a LAS member) and historically there has been enough member dues to fund a single part time developer. That said, the amount of work LAS gets done is pretty fantastic and I know Micha works hard, so I’m not complaining there. But Lucee is a very large system with a lot of surface area, a massive backlog, and recently, a steady stream of regressions stealing the attention. Extensions that aren’t “broke” tend to get forgotten about because frankly there’s no one being paid to spend the time looking after them. My best suggestion in this area is to consider any or all of the following

  • Becoming a Lucee member or supporter
  • Working to help locate insecure libraries and ensure there are tickets filed for them
  • Rally noise in the community via the JIRA voting feature or here in Discourse for things that need addressed. @IamSigmund is good about keeping track of things like this.
  • Consider submitting pull requests yourself to help update old libraries

From my vantage point, it’s as if a McDonald’s value meal comes not only with its own goodies but last week’s scraps from Burger King. What necessitates the redistribution of these .jars

Hah, nice analogy :slight_smile: What necessitates these jars is the core functionality of CFML itself. Nearly every ancillary feature of CFML (Hibernate, mail, FTP, S3, HTTP, XML, EASPI, JDBC) are all based on external libraries. Sure, we could simply remove all of those, but you’d be left with any other bare-bones JVM lang like Kotlin and Lucee wouldn’t only be about 10% compatible with Adobe ColdFusion. The reliance on 3rd party libs are part and parcel with the CFML language itself. The work of keeping them all updated is an unfortunate byproduct.

The better solution is to not bundle all. If I want a jdbc driver for PostgreSQL, I should head over to Postgres’ website to get it. I understand the desire to extend Lucee alongside these outside libraries, but I argue that bundling them is not correct.

JDBC drivers are probably the only thing in all of Lucee that you could just drop in the 3rd party jar and use directly. The other 90% of Java libs have an integration built around them. For example, you could drop in a PDF library like iText but that won’t give you the CFDocument tag. You could drop in the Apache HTTP commons libraries, but that won’t give you the CFHTTP tag.

I think the honest truth here is if you want updates to libs promised with a pubic roadmap, and a paid support plan, Adobe ColdFusion is ready and waiting with open arms… that will reach around and swipe your wallet :slight_smile: If you want a free CFML implementation, you can use Lucee sever, but you’re getting on board a train that can only go as fast as there are people pouring money into it. I think a lot of people are willing to live with the fact that some libs are out of date so long as it works and they can get them updated eventually. It’s not instant, but it’s free!

Am I correct in interpreting this as dev.lucee.org only being a support portal and that discussion of Lucee and its architecture does not have any formal place other than tickets in Jira for one developer to view?

More/less. This is not an official stance, it’s more the current state of affairs. Ideally this would be the place to discuss such things, but in reality, you won’t get any contact with the core devs here 98% of the time. Don’t let that stop you from posting here, just know that you’re a lot more likely to get people’s opinions (like mine) from the community. And if you are wanting something specific changed (like a library update in an extension) then put in a ticket for sure or it will never happen.

1 Like

Thanks again for your cohesive replies.

First, I want to express that my intentions are absolutely not to disparage the hard work of those that have helped build Lucee. I’m active in my own communities and understand the feelings of being short-staffed! I am also not demanding work be done as a consumer that expects accommodation but rather a concerned outsider that views this as unhealthy to the Lucee project.

My intentions with this thread are to discuss/plan before going any further. Now that I have a little bit more insight into the ecosystem I feel more comfortable spending the time trying to get these objects updated.

I am also rallying my employer to become a LAS member. :slight_smile:

Sure, addition of e.g. tags are a good example of the use of the .lex files. I feel that having the .lex file for a specific tag should expect a library rather than include it, i.e. throw an error when said library is not installed. This shifts the burden of dependency management on to the administrator and removes it from Lucee development.

Absolutely. Now that I’m aware of how things work I’m certainly compelled to do so.

Thank you again for your clarifications.

2 Likes

Yikes, no.

What do you propose to fix the problem in the long-term?

I guess it depends what part of the problem you are most concerned about.

The Extensions themselves may expect a particular version of an external dependency so it makes sense to define those dependencies in the extension and have them bundled up at build time. I don’t think it’s feasible to start to externalise all those dependencies (i.e. to not bundle them) and have something else manage them, because the specific version requirements may still be necessary. Extensions are their own distributable unit and need to be treated like any software distribution.

If the problem is out of date dependencies (particularly with those that have CVEs against them) then that’s more of a process issue. We should have something to identify those issues so that tickets can be created and the dependencies updated, and we should have a more general “keep everything up to date” approach while being aware of backwards compatibility issues if major version bumps are happening. That might go some way to being a long term solution.

3 Likes

a lot of the OSGI bundles in Lucee are just custom OSGI wrappers because I think back when Lucee moved to OSGI, many libraries (.jar) didn’t include OSGI metadata

@zingyb if you want to help, you could always go thru and request upstream that these libraries add OSGI metadata, like I did here with zip4j

use OSGI version of zip4j for the compress extension
https://luceeserver.atlassian.net/browse/LDEV-2526

2 Likes