Recurring log error with possible thread issues

I have a strange recurring log entry that may or not be the cause of lucee running out of memory.

at java.base/java.lang.Thread.getStackTrace(Unknown Source)
at lucee.runtime.CFMLFactoryImpl.createRequestTimeoutException(CFMLFactoryImpl.java:671)
at lucee.runtime.util.PageContextUtil.checkRequestTimeout(PageContextUtil.java:213)
at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:924)
at lucee.runtime.PageContextImpl.doInclude(PageContextImpl.java:918)
at lucee.runtime.PageContextImpl.handlePageException(PageContextImpl.java:2037)
at lucee.runtime.util.PageContextUtil.getHandlePageException(PageContextUtil.java:227)
at lucee.runtime.exp.CatchBlockImpl.castToString(CatchBlockImpl.java:225)
at lucee.runtime.op.Caster.toString(Caster.java:2038)
at lucee.runtime.op.Caster.toString(Caster.java:2031)

This block will recur hundreds of times

at lucee.runtime.PageContextImpl.handlePageException(PageContextImpl.java:2061)
at lucee.runtime.util.PageContextUtil.getHandlePageException(PageContextUtil.java:227)
at lucee.runtime.exp.CatchBlockImpl.castToString(CatchBlockImpl.java:225)
at lucee.runtime.op.Caster.toString(Caster.java:2038)
at lucee.runtime.op.Caster.toString(Caster.java:2031)

Then an entry like this

lucee.runtime.exp.RequestTimeoutException: request /articles/details.cfm (C:\inetpub.…) has run into a timeout (timeout: 20 seconds) and has been stopped. The thread started 262050ms ago.

Then a repeat of that whole block, with the last line being some other cfm has timed out. The log file has grown over 12GB in a day. I’m not sure where to start…

Not sure if this is related, but this is near the top of the log after lucee reboot. I do not use the ROOT web app

16-Aug-2021 16:06:49.416 WARNING [Thread-1585] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Thread-10] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.11/java.lang.Thread.sleep(Native Method)
lucee.commons.io.SystemUtil.sleep(SystemUtil.java:682)
lucee.runtime.cache.ram.RamCache$Controler.run(RamCache.java:233)
16-Aug-2021 16:06:49.431 WARNING [Thread-1585] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Thread-255] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.11/java.lang.Thread.sleep(Native Method)
lucee.commons.io.SystemUtil.sleep(SystemUtil.java:682)
lucee.runtime.cache.ram.RamCache$Controler.run(RamCache.java:233)
16-Aug-2021 16:06:49.447 WARNING [Thread-1585] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Thread-19] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.11/java.lang.Thread.sleep(Native Method)
lucee.commons.io.SystemUtil.sleep(SystemUtil.java:682)
lucee.runtime.cache.ram.RamCache$Controler.run(RamCache.java:233)
16-Aug-2021 16:06:49.447 WARNING [Thread-1585] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Thread-1207] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.11/java.lang.Thread.sleep(Native Method)
lucee.commons.io.SystemUtil.sleep(SystemUtil.java:682)
lucee.runtime.cache.ram.RamCache$Controler.run(RamCache.java:233)
16-Aug-2021 16:06:54.615 WARNING [Thread-1585] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [FelixResolver-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.11/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.11/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base@11.0.11/java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.11/java.lang.Thread.run(Unknown Source)
16-Aug-2021 16:06:54.615 WARNING [Thread-1585] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [FelixResolver-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.11/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.11/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base@11.0.11/java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.11/java.lang.Thread.run(Unknown Source)
16-Aug-2021 16:06:54.615 WARNING [Thread-1585] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [FelixResolver-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.11/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.11/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base@11.0.11/java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.11/java.lang.Thread.run(Unknown Source)
16-Aug-2021 16:06:54.615 WARNING [Thread-1585] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [FelixResolver-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.11/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.11/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base@11.0.11/java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.11/java.lang.Thread.run(Unknown Source)
16-Aug-2021 16:06:54.615 WARNING [Thread-1585] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [FelixResolver-5] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.11/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.11/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base@11.0.11/java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.11/java.lang.Thread.run(Unknown Source)
16-Aug-2021 16:06:54.615 WARNING [Thread-1585] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [FelixResolver-6] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.11/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.11/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base@11.0.11/java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.11/java.lang.Thread.run(Unknown Source)
16-Aug-2021 16:06:54.615 WARNING [Thread-1585] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [FelixResolver-7] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.11/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.11/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base@11.0.11/java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.11/java.lang.Thread.run(Unknown Source)
16-Aug-2021 16:06:54.615 WARNING [Thread-1585] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [FelixResolver-8] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.11/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.11/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
java.base@11.0.11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
java.base@11.0.11/java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.base@11.0.11/java.lang.Thread.run(Unknown Source)
16-Aug-2021 16:06:54.615 INFO [Thread-1585] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler [“http-nio-8888”]
16-Aug-2021 16:06:54.615 INFO [Thread-1585] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler [“ajp-nio-127.0.0.1-8009”]
16-Aug-2021 16:06:54.630 INFO [Thread-1585] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler [“http-nio-8888”]
16-Aug-2021 16:06:54.630 INFO [Thread-1585] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler [“ajp-nio-127.0.0.1-8009”]
mpl.java:572)

OS : Windows Server 2012 R2
Java Version : 11.0.11 (AdoptOpenJDK) 64bit
Tomcat Version : Apache Tomcat/9.0.35
Lucee Version : Lucee 5.3.8.201

More endless recurring log entries found:

17-Aug-2021 07:09:03.705 WARNING [ajp-nio-127.0.0.1-8009-exec-23] javax.mail.Session.loadResource expected resource not found: /META-INF/javamail.default.address.map

I think you will find the problem much faster by a good old fashioned trace work.

Sounds like something somewhere is in a loop and/or service failure leading to a loop/retry scenario and if you start breaking down segments this activity WILL stop: leading you to the segment causing it.

In my entire coding history I have never had a “Memory Leak”. So my advice would be to trace from bottom up… rather than struggling through the logs.

I know this is not an answer. It’s only a philosophy… But it leads to me never having to look at logs… like ever.