My Application become unresponsive after a while

[Mustafa_Kara]

Hi,

My application working well on kubernetes but after a while 2-3 minute server become unresponsive.
I checked my application’s trace logs and found that there are many thread in waits status.
There is a similar ticket in the past and fixed in 5.4.0.77.

Resource leak in DatasourceConnectionPool 5.3.10

We recently updated some apps from 5.3.8.201 to 5.3.10.97. After updating, our apps no longer recover from interrupted database connection. A restart is required to get the app to reconnect to the (MSSQL) database. Stack traces show many threads waiting for on a lock in lucee.runtime.db.DatasourceConnectionPool.getDatasourceConnection(DatasourceConnectionPool.java:95) . Logs may also show [java.lang.ThreadDeath] but not always. Looking into the lucee db pool code, it seems when ge…

SEVERE [ajp-nio-8009-exec-678] ... ThreadDeath

This line is filling up my logs at about 100+MB: SEVERE [ajp-nio-8009-exec-678] org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun Error running socket processor java.lang.ThreadDeath at java.base/java.lang.Thread.stop(Thread.java:949) at lucee.commons.io.StopThread.run(SystemUtil.java:1497) What does it mean? Is generating an error.

I see below logs in containers logs

 19-Jul-2024 19:41:15.292 SEVERE [ajp-nio-127.0.0.1-8009-exec-7] org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun Error running socket processor
 	java.lang.ThreadDeath
 		at java.base/java.lang.Thread.stop(Thread.java:937)
 		at lucee.commons.io.SystemUtil._stop(SystemUtil.java:1343)
 		at lucee.commons.io.SystemUtil.stop(SystemUtil.java:1334)
 		at lucee.commons.io.StopThread.run(SystemUtil.java:1765)

Thanks for any feedback.

java.lang.Thread.run(Thread.java:829)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789)
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890)
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:433)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:51)
lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:97)
lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:1162)
lucee.runtime.engine.CFMLEngineImpl._service(CFMLEngineImpl.java:1216)
lucee.runtime.engine.Request.exe(Request.java:45)
lucee.runtime.PageContextImpl.executeCFML(PageContextImpl.java:2449)
lucee.runtime.PageContextImpl._execute(PageContextImpl.java:2478)
lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2493)
lucee.runtime.listener.MixedAppListener.onRequest(MixedAppListener.java:44)
lucee.runtime.listener.ModernAppListener._onRequest(ModernAppListener.java:218)
lucee.runtime.listener.ModernAppListener.call(ModernAppListener.java:444)
lucee.runtime.ComponentImpl.call(ComponentImpl.java:1933)
lucee.runtime.ComponentImpl._call(ComponentImpl.java:586)
lucee.runtime.ComponentImpl._call(ComponentImpl.java:698)
lucee.runtime.type.UDFImpl.call(UDFImpl.java:223)
lucee.runtime.type.UDFImpl._call(UDFImpl.java:350)
lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:112)
lib.fusebox5.application_cfc$cf.udfCall(/lib/fusebox5/Application.cfc)
lib.fusebox5.application_cfc$cf.udfCall1(/lib/fusebox5/Application.cfc:241)
lucee.runtime.PageContextImpl.doInclude(PageContextImpl.java:929)
lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:948)
lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1056)
parsed.aefisframework_welcome_cfm690$cf$g.call(/lib/parsed/aefisframework.welcome.cfm:14)
lucee.runtime.PageContextImpl.getFunctionWithNamedValues(PageContextImpl.java:1794)
lucee.runtime.util.VariableUtilImpl.callFunctionWithNamedValues(VariableUtilImpl.java:866)
lucee.runtime.ComponentImpl.callWithNamedValues(ComponentImpl.java:1952)
lucee.runtime.ComponentImpl._call(ComponentImpl.java:586)
lucee.runtime.ComponentImpl._call(ComponentImpl.java:699)
lucee.runtime.type.UDFImpl.callWithNamedValues(UDFImpl.java:213)
lucee.runtime.type.UDFImpl._call(UDFImpl.java:350)
lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:112)
controller.aefisframework_cfc$cf.udfCall(/controller/AefisFramework.cfc)
controller.aefisframework_cfc$cf.udfCall1(/controller/AefisFramework.cfc:70)
lucee.runtime.PageContextImpl.getFunction(PageContextImpl.java:1775)
lucee.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:787)
lucee.runtime.ComponentImpl.call(ComponentImpl.java:1933)
lucee.runtime.ComponentImpl._call(ComponentImpl.java:586)
lucee.runtime.ComponentImpl._call(ComponentImpl.java:698)
lucee.runtime.type.UDFImpl.call(UDFImpl.java:223)
lucee.runtime.type.UDFImpl._call(UDFImpl.java:350)
lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:112)
framework.application.applicationservice_cfc$cf.udfCall(/aefismodel/framework/application/ApplicationService.cfc)
framework.application.applicationservice_cfc$cf.udfCall7(/aefismodel/framework/application/ApplicationService.cfc:875)
lucee.runtime.PageContextImpl.getFunction(PageContextImpl.java:1775)
lucee.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:787)
lucee.runtime.ComponentImpl.call(ComponentImpl.java:1933)
lucee.runtime.ComponentImpl._call(ComponentImpl.java:586)
lucee.runtime.ComponentImpl._call(ComponentImpl.java:698)
lucee.runtime.type.UDFImpl.call(UDFImpl.java:223)
lucee.runtime.type.UDFImpl._call(UDFImpl.java:350)
lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:112)
framework.application.applicationservice_cfc$cf.udfCall(/aefismodel/framework/application/ApplicationService.cfc)
framework.application.applicationservice_cfc$cf.udfCall7(/aefismodel/framework/application/ApplicationService.cfc:894)
lucee.runtime.functions.orm.ORMExecuteQuery.call(ORMExecuteQuery.java:55)
lucee.runtime.functions.orm.ORMExecuteQuery._call(ORMExecuteQuery.java:73)
lucee.runtime.orm.ORMUtil.getSession(ORMUtil.java:54)
lucee.runtime.orm.ORMUtil.getSession(ORMUtil.java:58)
lucee.runtime.PageContextImpl.getORMSession(PageContextImpl.java:3497)
org.lucee.extension.orm.hibernate.HibernateORMEngine.createSession(HibernateORMEngine.java:78)
org.lucee.extension.orm.hibernate.HibernateORMSession.<init>(HibernateORMSession.java:127)
org.lucee.extension.orm.hibernate.HibernateORMSession.createSession(HibernateORMSession.java:188)
org.lucee.extension.orm.hibernate.HibernateORMSession$SessionAndConn.<init>(HibernateORMSession.java:68)
org.lucee.extension.orm.hibernate.HibernateORMSession$SessionAndConn.getSession(HibernateORMSession.java:72)
org.lucee.extension.orm.hibernate.HibernateORMSession$SessionAndConn.getConnection(HibernateORMSession.java:79)
org.lucee.extension.orm.hibernate.HibernateORMSession$SessionAndConn.connect(HibernateORMSession.java:93)
lucee.runtime.db.DatasourceManagerImpl.getConnection(DatasourceManagerImpl.java:82)
lucee.runtime.db.DatasourceConnectionPool.getDatasourceConnection(DatasourceConnectionPool.java:91)
lucee.commons.io.SystemUtil.wait(SystemUtil.java:730)
java.lang.Object.wait(Native Method)

OS: Kubernetes 1.26
Java Version: openjdk-11
Tomcat Version: tomcat9
Lucee Version: 5.4.0.80