S3 extension stops working

dev
1

Hello! long time no see.
I am running Lucee 6.1.0.243 with the s3 extension installed (version 2.0.2.19-SNAPSHOT).
It works fine for a while, then all of the sudden it gives the following error. I restart Lucee and starts working again.

lucee.runtime.exp.NativeException: Unable to resolve org.lucee.s3.extension [84](R 84.0): missing requirement [org.lucee.s3.extension [84](R 84.0)] osgi.wiring.bundle; (&(osgi.wiring.bundle=org.lucee.aws-java-sdk-s3-all)(bundle-version>=1.12.758)) [caused by: Unable to resolve org.lucee.aws-java-sdk-s3-all [85](R 85.0): missing requirement [org.lucee.aws-java-sdk-s3-all [85](R 85.0)] osgi.wiring.package; (osgi.wiring.package=org.joda.time)] Unresolved requirements: [[org.lucee.s3.extension [84](R 84.0)] osgi.wiring.bundle; (&(osgi.wiring.bundle=org.lucee.aws-java-sdk-s3-all)(bundle-version>=1.12.758))] at org.apache.felix.framework.Felix.resolveBundleRevision(Felix.java:4398) at org.apache.felix.framework.Felix.startBundle(Felix.java:2308) at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:1006) at org.apache.felix.framework.BundleImpl.start(BundleImpl.java:992) at lucee.loader.osgi.BundleUtil.start(BundleUtil.java:112) at lucee.runtime.osgi.OSGiUtil._start(OSGiUtil.java:1475) at lucee.runtime.osgi.OSGiUtil._startIfNecessary(OSGiUtil.java:1419) at lucee.runtime.osgi.OSGiUtil.startIfNecessary(OSGiUtil.java:1411) at lucee.runtime.osgi.OSGiUtil._loadBundle(OSGiUtil.java:742) at lucee.runtime.osgi.OSGiUtil.loadBundle(OSGiUtil.java:680) at lucee.runtime.osgi.OSGiUtil$BundleDefinition.getBundle(OSGiUtil.java:2084) at lucee.commons.lang.ClassUtil.loadClassByBundle(ClassUtil.java:179) at lucee.commons.lang.ClassUtil.loadClassByBundle(ClassUtil.java:163) at lucee.transformer.library.ClassDefinitionImpl.getClazz(ClassDefinitionImpl.java:120) at lucee.transformer.library.ClassDefinitionImpl.getClazz(ClassDefinitionImpl.java:111) at lucee.commons.io.res.ResourcesImpl$ResourceProviderFactory.instance(ResourcesImpl.java:139) at lucee.commons.io.res.ResourcesImpl.getResource(ResourcesImpl.java:171) at lucee.runtime.config.ConfigImpl.getResource(ConfigImpl.java:2155) at lucee.runtime.config.ConfigWebImpl.getResource(ConfigWebImpl.java:1556) at lucee.commons.io.res.util.ResourceUtil.toResourceNotExisting(ResourceUtil.java:338) at lucee.runtime.functions.system.DirectoryExists.call(DirectoryExists.java:47) at lucee.runtime.functions.system.DirectoryExists.call(DirectoryExists.java:38)

As a side note, I was using a previous version of the extension, but it was randomly returning incorrect information such as empty folders that weren’t actually empty.

2

I know this post is a few months old now, but I am also seeing this exact same error. Lucee 6.1.1.118 and S3 Resource extension 2.0.2.21. Sometimes it works, other times it doesn’t. Just inspecting the Info > Bundle (Jar) list in the lucee admin gets it working again.

Would love to know if you ever got it working.

This is blocking us from being able to upgrade from Lucee 5 to 6. Something we’ve been trying to do for some time now.

3

6.1.1.118 is a bit old!

Is the problem still happening with the 6.2.1 RC?

Can you describe your stack? Is this a fresh install?

4

Our stack is: Ubuntu 22, Java 11, Tomcat 9, Nginx 1.26 and then Lucee 6.1.

We have tomcat deployed with a single application that runs under webapps/ROOT. The lucee jar sits in webapps/ROOT/WEB-INF/lib and then inside WEB-INF is lib, lucee, lucee-server folders.

We do have custom OSGi bundles and jars as well. The jars are in WEB-INF/lib, and the OSGi bundles are in WEB-INF/lucee-server/context/lib/

We want to upgrade to 6.2 eventually, but felt moving to 6.1 first would be less risky. However, I believe we were seeing similar issues with 6.2. We have not seen this issue with 5.4 - the rest of the stack is the same.

5

hmmm, what are the other osgi bundles & jars you are using?

I ran into this myself today, but I can’t figure out how to repo…

6

For jar libraries we have jsoup, pd4ml, and htmlparser.

For osgi - We have a few custom bundles. One is to access a government website (thank you soap) and the other is the more likely culprit. We have an osgi bundle with the AWS SDK v1.12.

It is not the whole SDK but it includes sts, ssm, kms, s3, among others. There are also dependencies bundled in there for aws things like commons-logging, jackson … and joda-time. Hmmm… Is it possible there is a clash there? I can’t figure out how to reproduce it. I thought the whole point of osgi magic was to sandbox the code and prevent clash problems.

7

hmm, yeah the other aws bundle does sound like possibly causing the conflict, especially given viewing the bundle in the admin solves the problem.

as you say, osgi should completely isolate these from each other, but sometimes there are problems!

that said I’m running a very fresh install with not much installed, there is how ever the redis bundle which also has an osgi aws bundle org.lucee.aws-java-sdk-secretsmanager-all

do you have a link to that AWS SDK v1.12 bundle?

8

@scottdoc , are you possibly receiving errors like this?

Cannot load class through its string name, because no definition for the class with the specified name [com.amazonaws.services.s3.AmazonS3ClientBuilder] could be found caused by (java.lang.ClassNotFoundException:com.amazonaws.services.s3.AmazonS3ClientBuilder not found by lucee.core [48];java.lang.ClassNotFoundException:com.amazonaws.services.s3.AmazonS3ClientBuilder;java.lang.ClassNotFoundException:com.amazonaws.services.s3.AmazonS3ClientBuilder;);cannot load class through its string name, because no definition for the class with the specified name [com.amazonaws.services.s3.AmazonS3ClientBuilder] could be found caused by (java.lang.ClassNotFoundException:com.amazonaws.services.s3.AmazonS3ClientBuilder not found by lucee.core [48];java.lang.ClassNotFoundException:com.amazonaws.services.s3.AmazonS3ClientBuilder;java.lang.ClassNotFoundException:com.amazonaws.services.s3.AmazonS3ClientBuilder;);cannot load class through its string name, because no definition for the class with the specified name [com.amazonaws.services.s3.AmazonS3ClientBuilder] could be found caused by (java.lang.ClassNotFoundException:com.amazonaws.services.s3.AmazonS3ClientBuilder not found by lucee.core [48];java.lang.ClassNotFoundException:com.amazonaws.services.s3.AmazonS3ClientBuilder;java.lang.ClassNotFoundException:com.amazonaws.services.s3.AmazonS3ClientBuilder;)

I’ve been having intermittent trouble with the S3 extensions, also. However, my issues do not go away when I visit the Info > Bundle (Jar) list. In that case, it says my S3 extension is just “loaded” – not active.

The extension fails randomly during server start. Sometimes it works, sometimes it doesn’t. If it doesn’t, I restart and it seems to work again.

9

I’m still running 6.1.0.243 and I’ve updated the extension to 2.0.2.21. It seems to be better lately. I haven’t been able to upgrade Lucee because it doesn’t start up after downloading the update.
For the record, I didn’t have any custom jars, just the plain install and I was still having the issue, so I doubt that is our problem.

10

In order to solve this extremely frustrating issue, what we need is a reliable way to reproduce the problem

Ideally as a github action, I put this together today for the Webservice extension, as it can’t be tested via our unit tests

11

Yes, reproducing it is very difficult.
Just for clarity, my setup is that we define credentials globally using

application.s3.accessKeyId = "xxxxx";
application.s3.awsSecretKey = "xxxxx";

Then The error happens on, for example, this: (the 3 slashes are not a typo):

fileExists("s3:///my-bucket/myfile.pdf");

@gooutside The error I get varies. I can get one and then refresh the page and get the other:

In the OSGi Bundle with the name [org.lucee.s3.extension] and the version [2.0.2.21] was no class with name [org.lucee.extension.resource.s3.S3ResourceProvider] found. org.lucee.extension.resource.s3.S3ResourceProvider

Or:

Unable to resolve org.lucee.s3.extension [79](R 79.0): missing requirement [org.lucee.s3.extension [79](R 79.0)] osgi.wiring.bundle; (&(osgi.wiring.bundle=org.lucee.aws-java-sdk-s3-all)(bundle-version>=1.12.758)) [caused by: Unable to resolve org.lucee.aws-java-sdk-s3-all [80](R 80.0): missing requirement [org.lucee.aws-java-sdk-s3-all [80](R 80.0)] osgi.wiring.package; (osgi.wiring.package=org.joda.time)] Unresolved requirements: [[org.lucee.s3.extension [79](R 79.0)] osgi.wiring.bundle; (&(osgi.wiring.bundle=org.lucee.aws-java-sdk-s3-all)(bundle-version>=1.12.758))]

@laura I am surprised to hear you don’t have any other libs installed. I thought for sure it was some sort of clash. Are you saying you don’t get the issue with 6.1.0.243 but you do with 6.1.1.118?

@Zackster Here is a link to the aws osgi bundle we use: https://fs.tst.axcelerate.com/platform/ax-aws-osgi-bundle-1.12.780.jar

12

My assumption is there’s other code running which somehow triggers this problem, just running that snippet alone isn’t going to trigger it right?

One thing you could try is enabling the Felix log, it’s pretty verbose, best not done in production, it will log the whole osgi processing out

Please try that with the 6.2.1 final RC

i.e. on Windows, I use the following env vars to dump all logs out to the console, which is good to watch, downside, the logs don’t get written to the filesystem for looking back

I really love this new feature in 6.2, makes debugging a lot easier

in the the lucee\tomcat\bin directory

set lucee_logging_force_level=trace
set lucee_logging_force_appender=console
set felix_log_level=debug
startup.bat

Adding a systemOutput(cgi.script_name); in application.cfc onRequestStart might help too

(works on Linux too, just use export and ./startup.sh)

there’s a shitload of noise in those logs, especially during startup, but it should help us with some good clues to track down the underlying issue

Also do you have any additional extensions installed? i.e Redis