I have built a new server for our production environment and everything runs fine with testing until it sees a load. Once it sees a load I see requests backing up in our SeeFusion monitor. Almost like it just hangs on some processes until it times out.
System Setup
Apache 2.4.6 (Proxy cf* request to tomcat/lucee)
Tomcat 8.5.33
Lucee 5.3.1.102
SQL Server 2012 Standard
I have tried using Microsofts database drive as well as jDTS driver and same experience. It appears to me to be an issue with tomcat but I am not having much luck tracking down the issue.
Any suggestions to get a deeper dive into why some processes appear to get into a hung state and die? Causing the system to backup in requests and ultimately stop responding?
On a side note: Currnet production runs fine with a similar setup but running lucee 4 and timcat 7.
Have 25GB set in java for lucee and server has 32GB allocated. Java never gets close to using all the available RAM before it starts having issues.
as for logs, catalina log just shows this. have not dug into as haven’t used log4j in the past.
log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAddCookies).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See Apache log4j 1.2 - Frequently Asked Technical Questions for more info.
I did just have these pop up in the catlina.log when trying to serve some traffic again tonight.
22-May-2019 22:35:02.422 SEVERE [ajp-nio-8009-exec-7] org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun
java.lang.ThreadDeath
at java.lang.Thread.stop(Thread.java:853)
at lucee.commons.io.StopThread.run(SystemUtil.java:1497)
22-May-2019 22:35:02.430 WARNING [ajp-nio-8009-exec-30] com.seefusion.Filter.doFilter Page request threw an unchecked error! This usually reflects a severe problem with the application.
java.lang.ThreadDeath
at java.lang.Thread.stop(Thread.java:853)
at lucee.commons.io.StopThread.run(SystemUtil.java:1497)
22-May-2019 22:35:02.431 SEVERE [ajp-nio-8009-exec-30] org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun
java.lang.ThreadDeath
at java.lang.Thread.stop(Thread.java:853)
at lucee.commons.io.StopThread.run(SystemUtil.java:1497)
22-May-2019 22:35:07.429 WARNING [ajp-nio-8009-exec-28] com.seefusion.Filter.doFilter Page request threw an unchecked error! This usually reflects a severe problem with the application.
java.lang.ThreadDeath
at java.lang.Thread.stop(Thread.java:853)
at lucee.commons.io.StopThread.run(SystemUtil.java:1497)
22-May-2019 22:35:07.429 SEVERE [ajp-nio-8009-exec-28] org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun
java.lang.ThreadDeath
at java.lang.Thread.stop(Thread.java:853)
at lucee.commons.io.StopThread.run(SystemUtil.java:1497)
In doing more digging tonight… Same issue on 5.3.1.95…
However, using this pre-release things ran allot better… any ideas when this might become an official release??? 5.3.2.74-RC
Actually, nevermind no pages load with RC, I just get a bunch of errors. Due to a hoth.cfc we use.
lucee.runtime.exp.NativeException: org.osgi.framework.BundleException:Duplicate import: com.sun.jdmk.comm
at org.apache.felix.framework.util.manifestparser.ManifestParser.normalizeImportClauses(ManifestParser.java:370)
at org.apache.felix.framework.util.manifestparser.ManifestParser.(ManifestParser.java:181)
at org.apache.felix.framework.BundleRevisionImpl.(BundleRevisionImpl.java:117)
at org.apache.felix.framework.BundleImpl.createRevision(BundleImpl.java:1282)
at org.apache.felix.framework.BundleImpl.(BundleImpl.java:113)
at org.apache.felix.framework.Felix.installBundle(Felix.java:3042)
at org.apache.felix.framework.BundleContextImpl.installBundle(BundleContextImpl.java:167)
at lucee.loader.osgi.BundleUtil.installBundle(BundleUtil.java:90)
at lucee.loader.osgi.BundleUtil.addBundle(BundleUtil.java:81)
at lucee.loader.osgi.BundleUtil.addBundle(BundleUtil.java:56)
at lucee.runtime.osgi.OSGiUtil.extractAndLoadBundles(OSGiUtil.java:1835)
at lucee.runtime.PageContextImpl.getRPCClassLoader(PageContextImpl.java:3388)
at lucee.runtime.PageContextImpl.getRPCClassLoader(PageContextImpl.java:3381)
at lucee.runtime.type.util.ComponentUtil.getComponentJavaAccess(ComponentUtil.java:116)
at lucee.runtime.ComponentImpl.getJavaAccessClass(ComponentImpl.java:2020)
at lucee.runtime.ComponentImpl.getMetaData(ComponentImpl.java:1531)
at lucee.runtime.ComponentImpl.getMetaData(ComponentImpl.java:1502)
at lucee.runtime.ComponentImpl.getMetaData(ComponentImpl.java:1463)
at lucee.runtime.functions.other.GetMetaData.getMetaData(GetMetaData.java:99)
at lucee.runtime.functions.other.GetMetaData.call(GetMetaData.java:69)
at lucee.runtime.functions.other.GetMetaData.call(GetMetaData.java:55)
at object.coreconfig_cfc$cf.udfCall(/hoth/object/CoreConfig.cfc:29)
at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:106)
at lucee.runtime.type.UDFImpl._call(UDFImpl.java:342)
at lucee.runtime.type.UDFImpl.call(UDFImpl.java:215)
at lucee.runtime.ComponentImpl._call(ComponentImpl.java:682)
at lucee.runtime.ComponentImpl._call(ComponentImpl.java:578)
at lucee.runtime.SuperComponent.call(SuperComponent.java:83)
at lucee.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:768)
at lucee.runtime.PageContextImpl.getFunction(PageContextImpl.java:1707)
at webpro.model.configs.hothconfig_cfc$cf.udfCall(/codebase/webpro/model/configs/HothConfig.cfc:76)
at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:106)
at lucee.runtime.type.UDFImpl._call(UDFImpl.java:342)
at lucee.runtime.type.UDFImpl.call(UDFImpl.java:215)
at lucee.runtime.ComponentImpl._call(ComponentImpl.java:682)
at lucee.runtime.ComponentImpl._call(ComponentImpl.java:570)
at lucee.runtime.ComponentImpl.call(ComponentImpl.java:1918)
at lucee.runtime.functions.other._CreateComponent.call(_CreateComponent.java:66)
at application_cfc$cf.udfCall1(/Application.cfc:284)
at application_cfc$cf.udfCall(/Application.cfc)
at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:106)
at lucee.runtime.type.UDFImpl._call(UDFImpl.java:342)
at lucee.runtime.type.UDFImpl.call(UDFImpl.java:215)
at lucee.runtime.ComponentImpl._call(ComponentImpl.java:682)
at lucee.runtime.ComponentImpl._call(ComponentImpl.java:570)
at lucee.runtime.ComponentImpl.call(ComponentImpl.java:1918)
at lucee.runtime.listener.ModernAppListener.call(ModernAppListener.java:436)
at lucee.runtime.listener.ModernAppListener.onError(ModernAppListener.java:424)
at lucee.runtime.listener.MixedAppListener.onError(MixedAppListener.java:123)
at lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2429)
at lucee.runtime.PageContextImpl._execute(PageContextImpl.java:2399)
at lucee.runtime.PageContextImpl.executeCFML(PageContextImpl.java:2374)
at lucee.runtime.engine.Request.exe(Request.java:43)
at lucee.runtime.engine.CFMLEngineImpl._service(CFMLEngineImpl.java:1037)
at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:983)
at lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:102)
at lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.seefusion.Filter.doFilter(Filter.java:109)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at mod_cfml.core.invoke(core.java:152)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at com.seefusion.SeeFusionValve.invoke(SeeFusionValve.java:52)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:479)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: org.osgi.framework.BundleException:Duplicate import: com.sun.jdmk.comm
… 85 more
Latest snapshot gives me same thing… guessing we have to update HOTH in the future.
Lucee 5.3.3.43-SNAPSHOT Error (java.io.IOException)
Message org.osgi.framework.BundleException:Duplicate import: com.sun.jdmk.comm
Stacktrace The Error Occurred in /src2/master/AppCode/thirdparty/Hoth-master/object/CoreConfig.cfc: line 29
27: public function init ()
28: { 29: local.md = getMetadata(this);
30: local.n = arrayLen(local.md.properties);
31: for (local.i=1; local.i <= local.n; local.i++) { called from /src2/master/AppCode/webpro/model/configs/HothConfig.cfc: line 76 called from /www2/weblink/Application.cfc: line 284
I did find a couple instances and am waiting for dev to get back to me. I only see this issue with the release candidate version. Not seeing these errors in the logs with 5.3.1.102, but maybe it is hidden and causing these same issues?