Scheduled tasks dying and requiring server restart to start again (for a bit)


#1

I set up a task in Lucee Web Admin to run every 10 seconds (also tried 20 seconds). It runs fine for a while but then after 10-20 minutes, it quits running (all other scheduled tasks also stop running). I cannot manually “execute” any tasks listed in Scheduled Tasks via Admin, and only a restart of Lucee solves the problem (of course, the issue re-occurs after 10-20 minutes!). See error information below.

The task takes <2 seconds to complete, and just in case there was an issue of task taking longer than frequency interval, I set the request timeout for the task to less than 10 seconds. Unfortunately, that doesn’t help.

Any thoughts on what could be happening? Is running tasks with frequency of less than 30 seconds an issue with Lucee?

Thanks,
Cage

Lucee / OS / Java versions

Version	Lucee 5.2.2.4-SNAPSHOT
Version Name	Velvet
Release date	May 8, 2017
ColdFusion® compatibility version	2016.0.03.300357
Configuration File	/usr/local/apache-tomcat-8.0.28/webapps/lucee-5.1.0.34/WEB-INF/lucee-server/context/lucee-server.xml
OS	Mac OS X (10.12.4) 64bit
Remote IP	0:0:0:0:0:0:0:1
Host Name	localhost
Servlet Container	Apache Tomcat/8.0.28
Java	1.8.0_51 (Oracle Corporation) 64bit
Architecture	64bit

Error message in scheduler.log

"ERROR","Thread-1189","05/11/2017","18:13:00","","schedule task:broadcastMessages;Can't overwrite cause with java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.util.concurrent.locks.Condition]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.;java.lang.IllegalStateException: Can't overwrite cause with java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.util.concurrent.locks.Condition]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
        at java.lang.Throwable.initCause(Throwable.java:457)
        at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1344)
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1206)
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1167)
        at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1486)
        at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:79)
        at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1998)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:138)
        at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:306)
        at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)
        at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:192)
        at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:185)
        at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:107)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:276)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at lucee.commons.net.http.httpclient.HTTPEngine4Impl._invoke(HTTPEngine4Impl.java:288)
        at lucee.commons.net.http.httpclient.HTTPEngine4Impl.get(HTTPEngine4Impl.java:120)
        at lucee.commons.net.http.HTTPEngine.get(HTTPEngine.java:91)
        at lucee.runtime.schedule.ExecutionThread.execute(ExecutionThread.java:109)
        at lucee.runtime.schedule.ExecutionThread.run(ExecutionThread.java:59)
Caused by: java.lang.ClassNotFoundException
        at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1343)
        ... 25 more

#2

Hi. I also have a problem with scheduling: https://luceeserver.atlassian.net/browse/LDEV-1326
Maybe the problems are related to each other.
:sob:


#3

Exceptions look different but who knows. Does anyone out there have any thoughts on how to troubleshoot? :slight_smile: Thanks! Cage