How to use an OSGi bundle which is not included by default

Hey all,

I am starting to play around with Lucee 5 and OSGi bundles. What would be the easiest solution to integrate a specific bundle?

I want to use the AWS SDK which is available here:
http://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-osgi

I tried regions = createObject( "java", "com.amazonaws.regions.Regions", "com.amazonaws.aws-java-sdk-osgi", "1.11.248" ); and regions = createObject( "java", "com.amazonaws.regions.Regions", "aws-java-sdk-osgi", "1.11.248" ); which both give me the same error saying that neither locally nor from the update provider http://release.lucee.org the bundle is available. Am I able to define an additional update provider so that Lucee directly downloads the bundles from maven? Or am I missing something?

Does Lucee only pick up bundles from local files? Or does it work with a remote maven repo as well out-of-the-box?

I found this thread where Michi posted that central maven is supposed to work if I understand it correctly:

Anywy so the main question is: How can I integrate an existing 3rd party OSGi bundle hosted on Maven Central without manually downloading it and putting it in a specific folder?

Any help would be appreciated.
Thanks
Jan

1 Like

Lucee Server
First of all a little basic on what Lucee does in case you are requesting a specific bundle, step by step.

  1. Lucee checks if that bundle exist locally, mean in the folder “/lucee-server/bundles” (loaded or not)
  2. it requests for that bundle the current defined update provider.

Update Provider
That’s it, BUT what is the update provider doing? He:

  1. checks if it has that bundle locally.
  2. it tries to load that bundle from maven central (more about that below), actually it redirects to maven (302)
  3. it log if it was not able to deliver that bundle (more about that below)

How does the Update Provider load bundles from Maven?
Problem here is that we have only the OSGi identification (Symbol name and version), but to get something from maven, we need a maven Identification (groupId, artifactId and version).
First we have a mapping collection (very new) in the update provider that maps osgi id to maven id, if that maven id is in that list it can redirect to maven.
If there is no mapping the update provider tries to convert the osgi id into a maven id by splitting the id

What is the log good for?
the update provider logs if it was not able to provide a certain lib. we go throw that log rom time to time and make the requested jars available by copy them locally or extending the mapping collection.

What can you do?
Make a pull request to extend the mapping collection with your mapping

i will check the log and extend the list, stay tuned

1 Like

Thanks for the reply @micstriit . I have a follow up question.

Lucee checks if that bundle exist locally, mean in the folder “/lucee-server/bundles” (loaded or not)

Is there a way around manually copying the bundle into that folder? I’m thinking of two specific possibilities.

  1. If the jar is in the dev’s source code repo, have something akin to this.javaSettings where we can simply point to it in our CFML code and Lucee will pick it up. This makes the CFML code more portable and not tied to additional manual steps of copying jars.
  2. Is there a way that createObject could directly specify the maven details so it could be auto-downloaded without needing to explicitly define it in the update provider.

#2 is of lesser importance to me since I think it’s probably best to have a local build that would grab jars off of Maven first and then just worry about getting Lucee to load them from disk. I assume devs could use Maven’s CLI or CommandBox’s jar endpoint today to get dependencies. I just shy away from processes that require manually copying stuff into the server installation. it doesn’t translate well to installations like Docker.

@Jan_Jannek it should work now

Remember that this feature is an addition to what we already had in 4.5. you can still use “javasettings” or define a path to the jar with createObject. Or to you talk about an addional way to get the data from remote somewhere.

actually i was thinking exactly the same… but now that is not possible yet.

Does that work with OSGI bundles? Meaning that if a jar in one of the paths that’s given to this.javaSettigns is an OSGI bundle, then Lucee will register the bundle? If so, that would be great. For some reason I just assumed OSGI bundles wouldn’t be picked up like that.

Thanks a lot Michi for adding this.
But would it also be possible to have that mapping defined dynamically anywhere in code?
Then it wouldn’t be required to have it hardcoded in lucee core.

Forget what I said, it’s the update provider and not lucee core, right? :wink:
So to correct my comment: Would it be possible to supply more info so that it will be able to deal with the mapping directly? That would be awesome…

@Jan_Jannek I believe that’s basically what I asked about above and Micha said he had thought about it but it wasn’t possible yet. Also, note that the “mapping” between bundle name and actual Maven artifact is not in the Lucee core, but in the code to the update provider web site that he linked to above.

@micstriit
Okay, tried it again and it failed - you probably see this in the update provider log as well, correct?

Unable to resolve com.amazonaws.aws-java-sdk-osgi [102](R 102.0): missing requirement [com.amazonaws.aws-java-sdk-osgi [102](R 102.0)] osgi.wiring.package; (&(osgi.wiring.package=com.fasterxml.jackson.annotation)(version>=2.6.0)(!(version>=3.0.0))) Unresolved requirements: [[com.amazonaws.aws-java-sdk-osgi [102](R 102.0)] osgi.wiring.package; (&(osgi.wiring.package=com.fasterxml.jackson.annotation)(version>=2.6.0)(!(version>=3.0.0)))]

I’m having this conversation in a couple places since Jan and I have been chatting in CFML Slack at the same time. I wanted to reproduce some sentiments on this thread as well that I had mentioned in slack.

FWIW, I think the auto-downloading stuff is kind of nifty, but I don’t think it’s really something Lucee should be doing (or at least, encouraging). Especially if it may work in some cases, but not others. I think that dependencies to an app (whether they are CFML libs or jars) should be documented in your app’s descriptor (package.json, box.json, pom.xml, whatever) and the artifacts produced as part of a build step. (npm install, box install, maven install, etc) I’m not sure if we should be pushing for magical downloads of Java libs in favor of using a package manager to produce them.

I think that is a more correct way for CF devs to deal with 3rd party jars with a package manager than to have auto-downloading magic, which isn’t something I’d want on a production server, especially a Docker container that would need to re-download on every deploy.

1 Like

actually i don’t know, i have to test …

i think having the possibility to define maven id with createObject is intressting, but of course a jar from maven not necessary is a OSGi bundle, but still intressting.

but it now fails because com.amazonaws.aws-java-sdk-osgi as dependecies. it misses “com.fasterxml.jackson.annotation”.

@micstriit, @bdw429s

I second Brad’s comment regarding package management. It seems strange that we would need to define those mappings in the update provider and that Lucee would pull in some external dependencies during runtime instead of during - let’s say - deploy time (which would be a way more controlled manner).

I have found that there are some bundle specifc cfadmin actions: getBundle, getBundles, readbundle, buildbundle. But there is no possibility to add a bundle, right?

I tried to use the Application.cfc setting

this.javasettings

to see if it would pick up the OSGi bundles but that did not work. I got the same error as before (now regarding jackson annotations).

So basically what we would need is a possibility to install an OSGi bundle (and the dependencies) in a controlled manner during deployment - and ideally without the need to touch the Application.cfc (in case it’s a module/plugin/etc.)

Could I wrap the AWS SDK OSGi bundle - including the dependencies - in a Lucee extension? Is it possible to programmatically install that extension then and would this approach make sense?
I saw in the Lucee documentation that this way I could start the included bundles.

To be clear, did you have the jackson annotations bundle in one of the folders pointed to by your this.javasettings? javasettings isn’t going to do anything to resolve dependencies, it’s just going to load whatever you’ve placed there (in theory). If you just put the initial OSGI bundle in your javasettings folder and you got the “jackson error” then wouldn’t that mean that it worked? i.e., this.javasettings was enough to register the bundle with Lucee/OSGI so that it could be loaded by createObject (and the dependency error would be completely expected at this point assuming you still haven’t actually included all the necessary dependencies.)

And finally-- I mentioned this on Slack the other day, but you weren’t testing the javasettings with the original bundle name that you started this thread out with were you? At this point, since MIcha has now added special sauce for that bundle in the update provider, you wouldn’t know (unless it’s in the logs perhaps) whether the initial bundle loaded due to your javasettings change or because Micha added it to the update provider. I mentioned in Slack that you need to find some brand new bundle that Micha hasn’t added any special sauce for and test it so you have a proper test in a vacuum.

So, to answer your question about the extension-- yes you can absolutely do that. It’s actually pretty easy. You’d just need to have the jars in the ext and create the proper manifest file and the bundles would just get loaded in when you installed the extension. So, here’s the caveats:

  1. At some point you’re still going to have to resolve the dependencies. There’s nothing special about extensions that go and track down dependencies for you. At this point, Maven’s CLI is the ideal tool for you to do a fresh install of the library that will pull all the deps you need.
  2. You’d still need to figure out what to do with the actual extension (lex) file. There’s no public location where one can publish a Lucee extension to. I’ve planned to make ForgeBox do this, but it’s another one of my projects that hasn’t happened yet. This means you’d need to actually host your own extension provider, or just stick the lex file somewhere and manually install it by copying the lex file into your deploy folder.
  3. You are correct that there is a way to tell Lucee what extensions to install when it starts, but that only works if the extensions are hosted on the official Lucee extension provider that comes pre-installed in Lucee. (Or a 3rd part provider you’ve added via the admin) So it’s still a bit of a catch 22 in regards to provisioning out a new server. Some day I need to add the ability to add Lucee extension providers to CFConfig. (Man, I’m really killing your workflow with all the projects I haven’t gotten done :slight_smile: )
  4. So at this point I’m not sure that the extension route does anything for you that you can’t just do with a maven install step and possibly javaSettings (if it does in fact work)

@bdw429s

Thanks for sharing your thoughts on this one.
Yes I tried it with the AWS SDK which Michi mapped on the update provider source code. But as it did not work before, I thought I first try it again with the javasettings. The update provider is still missing a mapping of faster jackson annotations right now.
And as the error did not change, even though I have that particular jar in question in my locally defined path (mentioned in javasettings) I was sure that there is no bundle support for the javasettings path.

But to be 100% certain I just used another Library which is not on the update provider:
http://mvnrepository.com/artifact/com.google.code.gson/gson/2.8.2

I first tried to directly use it and see what the update provider does:

gson = createObject( "java", "com.google.gson.Gson", "com.google.code.gson-gson", "2.8.2" );
writeDump(gson);

I got the same error as before saying:
The OSGi Bundle with name [com.google.code.gson-gson] in version [2.8.2] is not available locally or from the update provider.

Then I added a lib path which contains the jar (locally) to the Application.cfc:

component {

    this.name = "awstest-" & hash(getCurrentTemplatePath());
    this.javaSettings = {
          loadPaths = [ expandPath( "./osgi-lib" ) ]
        , reloadOnChange = true
    };
}

This resulted in the exact same error. Therefore I think that javasettings will not help.

And yes you are of course right - it does not really make sense to create a Lucee extension and copy it somewhere if I could also just copy all the OSGi bundle jars to Lucee’s /bundles directory. Because I think this is probably currently the only way it works (if we do not want to rely on what the update provider does).
I supposed it would work something like: resolve dependencies during build using mvn install. So the result is having all required bundle jars locally in a folder. Then on app load copy them over to lucee’s bundles directory. Can’t see another way currently.

@micstriit
What I do not get about the update provider code: Why are all the additional mappings for the dependencies needed? Those should be within the pom.xml within the first bundle jar (the AWS one). Why is it required to add those mappings as well? Lucee should be able to retrieve them from that pom. The maven artifacts are listed there. Or am I missing something here?

This resulted in the exact same error. Therefore I think that javasettings will not help

Ok, thanks for clarifying. So it sounds like javaSettings does not load in OSGI bundles that are found within those directories. Too bad. Can you put in a ticket for this? I Think it is really needed to allow people to use OSGI jars in the same way they use classic jars that doesn’t involve meddling with the server installation folders.

Lucee should be able to retrieve them from that pom.

So, two things. Firstly, it’s not a standard to require a pom.xml in every jar. They might exist, but it’s not quite like the box.json or package.json which is required to be inside the package or the package isn’t valid. Secondly, you’re getting back into the territory of Lucee rebuilding a package manager. While possible, I don’t think that’s the job of an app server. That’s the job of Maven (or my future CommandBox endpoint). The act of resolving dependency hierarchies and acquiring artifacts are always a build concern that happens outside of the app server. The app server just loads the libs and assumes you’ve already acquired everything. It’s my opinion that Lucee should focus on making it easy for CF devs to point to their jars/bundles and get them loaded up by Lucee and the actual package manager aspect should be handled externally. So to me, this isn’t really a question of why can’t Lucee do xyz, but should Lucee be doing this.

Totally agree with the whole last paragraph.
Created an issue regarding javasettings: https://luceeserver.atlassian.net/browse/LDEV-1624

1 Like