Are Lucee's custom class loaders single-threaded or multithreaded?

A question to Team Lucee: Are Lucee’s custom class loaders single-threaded or multithreaded?

I ask because our logs show many situations where a lock is associated with Lucee’s custom class loaders. Beyond a certain request load, the application’s performance falls sharply. Requests that should normally take several seconds begin to last 3 minutes.

The result is that our application, good for thousands of users per day, practically comes to a standstill. Related ticket.

Here is an excerpt from one of the typical stacktraces we’re getting from FusionReactor’s profiler:


dvnt.stdrdr.vo.basicuservo_cfc$cf.udfCall(/nl/dvnt/stdrdr/vo/BasicUserVO.cfc)
lucee.runtime.functions.decision.IsInstanceOf.call(IsInstanceOf.java)
lucee.runtime.reflection.Reflector.isInstaneOf(Reflector.java) (hide)
lucee.commons.lang.ClassUtil.loadClass(ClassUtil.java)
lucee.commons.lang.ClassUtil.loadClass(ClassUtil.java)
lucee.commons.lang.ClassUtil._loadClass(ClassUtil.java)
lucee.commons.lang.ClassUtil.__loadClass(ClassUtil.java)
lucee.commons.lang.ClassUtil$ClassLoaderBasedClassLoading.loadClass(ClassUtil.java)
java.lang.ClassLoader.loadClass(Unknown Source)
java.lang.ClassLoader.loadClass(Unknown Source)
Waiting on lock <0x54d4dccc> (a java.lang.Object)

The line of code at /nl/dvnt/stdrdr/vo/BasicUserVO.cfc is:

/*** /nl/dvnt/stdrdr/vo/BasicUserVO.cfc ***/
if ( isInstanceOf(arguments.data, "nl.dvnt.stdmtr.orm.UsersRO") ) {

See attached for more on the FusionReactor Profiler stacktrace.

The lock makes me wonder whether every custom class loader used by Lucee follows the “Recommendations for Multithreaded Custom Class Loaders”


FusionReactor_Profiler_stacktrace_longlasting_request.txt (16.7 KB)

The interesting thing about locks, is they don’t consume any CPU. One of your first posts regarding performance stated you were seeing CPU saturation, but on servers experiencing deadlocks or just over-aggressive locking, the CPU is often quite low even though pages can take a long time to finish. Just to clarify, are you seeing high CPU on the server at the same time that you’re seeing requests “wait on lock” or are these unrelated incidents?

Are Lucee’s custom class loaders single-threaded or multithreaded?

The good question here is whether a given class is written to correctly its internal integrity when used by more than one thread at a time. I can’t speak entirely for Micha, but I am quite sure he would say “Yes, of course Lucee’s class loaders are designed to be used by more than one thread at a time”.

The Oracle article is interesting, however

  • I see no evidence of deadlocks at this point (which isn’t the same as a lock)
  • The actual locks I’m seeing in the profile are coming from java.lang.ClassLoader (the core class loader class), not from any custom Lucee class loader, so I’m not even sure any of that article applies here!

The profiler will tell you what lines of code are running the most or the longest, but it won’t tell you what is locking your thread. For that, you need to capture full JVM thread dumps that catches in-the-act

  • the thread holding the lock
  • all the threads waiting to acquire the same lock
  • the actual lines of code this stuff is happening on (The profiler does not track at the line level for performance reasons)

Can you post a full JVM thread dump while this is going on. The question is what other thread was holding the lock and why. But without knowing the lines of code to go review the source, it’s all just guessing right now. Also, if you haven’t mentioned it elsewhere, can you let us know what exact version of Java you’re using? That can change what the line numbers in the stack mean.

Also, to add some more info-- I was looking at what it takes to register a class loader as parallel capable-- as the base class loader class actually behaves differently based on that.

It’s interesting to note that there are 12 classes in Lucee’s core that appear to extend a class loader class of some kind, but only 3 of them appear to register themselves as parallel (noted in bold)

  • Lucee/core/src/main/java/lucee/commons/io/res/util/RCL.java
  • Lucee/core/src/main/java/lucee/commons/io/res/util/ResourceClassLoader.java (parallel)
  • Lucee/core/src/main/java/lucee/commons/lang/ArchiveClassLoader.java
  • Lucee/core/src/main/java/lucee/commons/lang/ExtendableClassLoader.java (parallel)
  • Lucee/core/src/main/java/lucee/commons/lang/LogClassLoader.java
  • Lucee/core/src/main/java/lucee/commons/lang/MemoryClassLoader.java
  • Lucee/core/src/main/java/lucee/commons/lang/PClassLoader.java
  • Lucee/core/src/main/java/lucee/commons/lang/PCLBlock.java
  • Lucee/core/src/main/java/lucee/commons/lang/PhysicalClassLoader.java (parallel)
  • Lucee/core/src/main/java/lucee/commons/lang/compiler/DynamicClassLoader.java
  • Lucee/core/src/main/java/lucee/runtime/osgi/EnvClassLoader.java

Now, I have NO CLUE if that is causing any issues whatsoever as @micstriit would need to come along and comment on why that was the case. And I’ll point out again your stack shows the locking waiting happening inside of a core JDK class loader-- or at least it appears to. Reading stacks is tricky since if a custom class loader extends java.lang.ClassLoader and calls its super loadClass() method, the stack will show the load class method from the core class loader is running, but the actual class loader instance may still be a sub classed instance that has NOT registered itself as parallel capable. So without further debugging, there’s no easy way to tell if the

java.lang.ClassLoader.loadClass(Unknown Source)

call that was waiting on the lock was from an non-parallel-capable subclass or not.

It’s also worth noting that classes such as Lucee’s ExtendableClassLoader register themselves as parallel capable…

public abstract class ExtendableClassLoader extends ClassLoader {

	static {
		boolean res = registerAsParallelCapable();
	}
...

HOWEVER, this note in the Javadocs for ClassLoader gives me pause:

 * Note that the {@code ClassLoader} class is registered as parallel
 * capable by default. However, its subclasses still need to register themselves
 * if they are parallel capable.

This seems to imply that simply registering a base class is not enough, but every sub class must also register themselves explicitly as parallel capable. Again, we need @micstriit to chime in here and explain.

@Zackster Also just pointed out this ticket to me which indicates the addition of the parallel code was rather recent (last month). See my comment at the bottom of the ticket based on our discoveries here:
https://luceeserver.atlassian.net/browse/LDEV-4169

1 Like

Hi @bdw429s ,
Thank you for looking into this.

Sorry about my late reply. I have only been able to come up for air just now. It’s been hectic in our Lucee kitchen as we try to get a workable application going.

To answer your question on “wait on lock”, we do in fact see that on the same instance on occasions when the CPU is low.

I am fixing our thread-dump code and shall later share what the dumps say.

Thank you for your time and for the insights.