Occasional java.lang.NoClassDefFoundError for pgsql on startup

lucee version: 5.2.1.9 (I had to roll back to this version due to [LDEV-1640] - Lucee and haven’t gotten around to upgrading since it’s been fixed)

Every once in a while my application fails to start spectacularly. The issue surfaces itself depending on load (i.e. a higher server load gives a higher chance of triggering). It’s likely a race condition but I’m not sure where to look (I’m new to CF).

These errors manifest and continue to spam logs until I kill and try again. Right now it’s a gamble on whether I get a “clean” startup.

"ERROR","Thread-62","11/17/2018","06:07:30","","Type: java.lang.NoClassDefFoundError
Time: 17-Nov-18 06:07 AM
Correlation ID:
Message: org/postgresql/util/ByteConverter
Detail:
Extended Info:
Error Code: 0

It’s always related to postgresql when it does appear.

My lucee config can be viewed here (new user, can’t upload files):

https://paste.debian.net/plainh/ce2c1ce6

Does any kind soul have an inkling as to why this is happening, or at least a good place for me to investigate?

Thank you!

what version of java are you running?

OpenJDK 8. This was experienced both with Debian and Alpine’s versions.

does the problem still occur with the latest 5.2.9.33-SNAPSHOT ?

I’ve been unable to replicate it so far on 5.2.9.33-SNAPSHOT but I can on 5.2.9.31.

This doesn’t mean that the issue isn’t there; sometimes it’ll hide itself for a bit before spitting fire out again. I’ll keep testing.

I’ve been testing some more with the SNAPSHOT and haven’t been able to get it. But then again, I’ve been unable to hit it on 5.2.9.31 today either. It really likes to hide sometimes.

Typically to test I’m running stress -j8 and startup up the application.

Is there any commit or bug report that leads one to believe that the latest -SNAPSHOT has a fix for this?

I think the issue is that the postgres driver is loading itself twice:

"INFO","Thread-13","12/15/2018","07:14:19","Instrumentation","not found [lucee.runtime.instrumentation.ExternalAgent] in ClassLoader [sun.misc.Launcher$AppClassLoader@18b4aac2]"
"INFO","Thread-13","12/15/2018","07:14:19","Instrumentation","not found [lucee.runtime.instrumentation.ExternalAgent] in ClassLoader [java.net.URLClassLoader@45fe3ee3]"
"INFO","Thread-13","12/15/2018","07:14:19","Instrumentation","not found [lucee.runtime.instrumentation.ExternalAgent] in ClassLoader [lucee.commons.io.res.util.ResourceClassLoader@8358a6d]"
"INFO","Thread-13","12/15/2018","07:14:19","Instrumentation","looking for tools.jar"
"INFO","Thread-13","12/15/2018","07:14:19","Instrumentation","couldn't find tools.jar at: /usr/lib/jvm/java-1.8-openjdk/jre/lib/tools.jar"
"INFO","Thread-13","12/15/2018","07:14:19","Instrumentation","for IBM SDK couldn't find /usr/lib/jvm/java-1.8-openjdk/lib/tools.jar"
"INFO","Thread-13","12/15/2018","07:14:19","Instrumentation","found /opt/lucee/server/lucee-server/context/tools-linux-7u25.jar"
"INFO","Thread-13","12/15/2018","07:14:19","Instrumentation","tools.jar used:/opt/lucee/server/lucee-server/context/tools-linux-7u25.jar"
"INFO","Thread-13","12/15/2018","07:14:19","Instrumentation","deploy /resource/bin/linux64/libattach.so to /opt/lucee/server/lucee-server/context/bin/libattach.so"
"INFO","Thread-13","12/15/2018","07:14:19","Instrumentation","add /opt/lucee/server/lucee-server/context/bin to library path"
"INFO","Thread-13","12/15/2018","07:14:19","Instrumentation","loaded VirtualMachine class:com.sun.tools.attach.VirtualMachine"
"INFO","Thread-13","12/15/2018","07:14:19","Instrumentation","create /opt/lucee/server/lucee-server/context/lucee-external-agent.jar"
"INFO","Thread-13","12/15/2018","07:14:19","Instrumentation","try to load agent (path:/opt/lucee/server/lucee-server/context/lucee-external-agent.jar)"
"INFO","Thread-13","12/15/2018","07:14:19","Instrumentation","pid:60"
start set instrumentation
sun.misc.Launcher$AppClassLoader
sun.misc.Launcher$AppClassLoader
sun.misc.Launcher$AppClassLoader
"INFO","Thread-13","12/15/2018","07:14:20","Instrumentation","found [lucee.runtime.instrumentation.ExternalAgent] in ClassLoader [sun.misc.Launcher$AppClassLoader@18b4aac2]"
"INFO","Thread-13","12/15/2018","07:14:20","Instrumentation","ExternalAgent does contain a Instrumentation instance"
"INFO","Thread-16","12/15/2018","07:14:21","schedule task:reloadConfig","first execution runs at {ts '2018-12-15 07:15:00'}"
"INFO","Thread-13","12/15/2018","07:14:22","OSGi","add bundle:/opt/lucee/server/lucee-server/bundles/org.postgresql.jdbc42-9.4.1212.jar"
"INFO","Thread-13","12/15/2018","07:14:22","OSGi","start bundle:org.postgresql.jdbc42:9.4.1212"
"INFO","Thread-15","12/15/2018","07:14:22","OSGi","start bundle:com.googlecode.libphonenumber:7.7.4"
"INFO","Thread-15","12/15/2018","07:14:23","OSGi","add bundle:/opt/lucee/server/lucee-server/bundles/s3.extension-0.9.4.113.jar"
"INFO","Thread-15","12/15/2018","07:14:23","OSGi","start bundle:s3.extension:0.9.4.113"
"INFO","Thread-15","12/15/2018","07:14:23","OSGi","add bundle:/opt/lucee/server/lucee-server/bundles/org.lucee.bouncycastle.bcprov-1.52.0.jar"
"INFO","Thread-15","12/15/2018","07:14:23","OSGi","start bundle:org.lucee.bouncycastle.bcprov:1.52.0"
"INFO","Thread-15","12/15/2018","07:14:23","OSGi","add bundle:/opt/lucee/server/lucee-server/bundles/org.lucee.jets3t-0.9.4.0003L.jar"
"INFO","Thread-15","12/15/2018","07:14:23","OSGi","start bundle:org.lucee.jets3t:0.9.4.0003L"
"INFO","Thread-15","12/15/2018","07:14:23","OSGi","start bundle:com.googlecode.libphonenumber:7.7.4"
"INFO","Thread-15","12/15/2018","07:14:23","OSGi","add bundle:/opt/lucee/server/lucee-server/bundles/org.postgresql.jdbc42-9.4.1212.jar"
"INFO","Thread-15","12/15/2018","07:14:23","OSGi","start bundle:org.postgresql.jdbc42:9.4.1212"
"INFO","Thread-15","12/15/2018","07:14:23","OSGi","start bundle:com.googlecode.libphonenumber:7.7.4"

Why would this be the case? It’s even loading it on a separate thread.

I would think that loading it twice would just cause it to use a different class. That’s a good question about why it’s loading it twice though… makes no sense. Is this a fresh install? What kind of install did you do? Pretty strange…