Lucee threads hanging (java.util.concurrent.locks.ReentrantLock$FairSync)

dev support
1

We have been seeing some issues lately with Lucee hanging. In our FusionReactor emails, we’re seeing threads are getting stuck waiting on locks to release.

Here’s a partial stack trace starting from the CFML that caused the issue.

jdk.internal.misc.Unsafe.park(Native Method)
- waiting on <0x1b9f3ae3> (a java.util.concurrent.locks.ReentrantLock$FairSync)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:917)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1240)
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:267)
org.apache.felix.framework.Felix.acquireBundleLock(Felix.java:5522)
org.apache.felix.framework.Felix.startBundle(Felix.java:2110)
org.apache.felix.framework.BundleImpl.start(BundleImpl.java:998)
org.apache.felix.framework.BundleImpl.start(BundleImpl.java:984)
lucee.runtime.osgi.OSGiUtil.extractAndLoadBundles(OSGiUtil.java:1979)
lucee.runtime.PageContextImpl.getResourceClassLoader(PageContextImpl.java:3502)
lucee.runtime.PageContextImpl.getClassLoader(PageContextImpl.java:3488)
lucee.runtime.functions.other.JavaProxy.loadClassByPath(JavaProxy.java:126)
lucee.runtime.functions.other.JavaProxy.loadClass(JavaProxy.java:65)
lucee.runtime.functions.other.JavaProxy.call(JavaProxy.java:60)
lucee.runtime.functions.other.CreateObject.doJava(CreateObject.java:140)
lucee.runtime.functions.other.CreateObject.call(CreateObject.java:62)
lucee.runtime.functions.other.CreateObject.call(CreateObject.java:49)
util.datetimeutil_cfc$cf.initComponent(/basecom/util/DateTimeUtil.cfc:118)

The line of the code in question causing issues is:

createObject("java", "java.util.TimeZone")

I think this might be getting caused by LDEV-4064, but I’m not positive.

Any thoughts?

OS: Linux (3.10.0-1160.80.1.el7.x86_64) 64bit
Java Version: 11.0.17 (Red Hat, Inc.) 64bit
Tomcat Version: Apache Tomcat/9.0.63
Lucee Version: 5.3.9.141

2

Did you try one of the versions of Lucee with the fix for that ticket?

5.3.10.26

5.3.9.148

1 Like
3

@bdw429s

I have not tried them yet, but that’s my plan once I can reliably re-create the issue.

However, while I was working on that I wanted to see if anyone else has seen the issue and if my suspicion might be correct in that is the problem.

4

I was able to put together a test case that is fixed by 5.3.10.97. It’s a little different than the production code, but it produces the same race condition of locks under load.

I plan on deploying 5.3.10.97, once we can get through testing of the application to make sure everything is stable.

I’m optimistic that the fix in [LDEV-4064] - Lucee has resolved the issue.

1 Like
5

thanks @dswitzer really appreciate the excellent work developing a test case and reproducing the problem