Tomcat 8/Lucee 5.3.1.102 Slowness Issues

I have built a new server for our production environment and everything runs fine with testing until it sees a load. Once it sees a load I see requests backing up in our SeeFusion monitor. Almost like it just hangs on some processes until it times out.

System Setup

  • Apache 2.4.6 (Proxy cf* request to tomcat/lucee)
  • Tomcat 8.5.33
  • Lucee 5.3.1.102
  • SQL Server 2012 Standard

I have tried using Microsofts database drive as well as jDTS driver and same experience. It appears to me to be an issue with tomcat but I am not having much luck tracking down the issue.

Any suggestions to get a deeper dive into why some processes appear to get into a hung state and die? Causing the system to backup in requests and ultimately stop responding?

On a side note: Currnet production runs fine with a similar setup but running lucee 4 and timcat 7.

CURRENT PROD

  • Apache 2.4.6 (Proxy cf* request to tomcat/lucee)
  • Tomcat 7.0.59
  • Lucee 4.5.5.006 final
  • SQL Server 2012 Standard

web.xml (167.6 KB)
ssl-weblink.conf (5.3 KB)
server.xml (8.4 KB)
httpd.conf (11.6 KB)

What does the memory usage look like?

Any exceptions in the logs?

Have 25GB set in java for lucee and server has 32GB allocated. Java never gets close to using all the available RAM before it starts having issues.

as for logs, catalina log just shows this. have not dug into as haven’t used log4j in the past.

log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAddCookies).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

I did just have these pop up in the catlina.log when trying to serve some traffic again tonight.

22-May-2019 22:35:02.422 SEVERE [ajp-nio-8009-exec-7] org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun
java.lang.ThreadDeath
at java.lang.Thread.stop(Thread.java:853)
at lucee.commons.io.StopThread.run(SystemUtil.java:1497)

22-May-2019 22:35:02.430 WARNING [ajp-nio-8009-exec-30] com.seefusion.Filter.doFilter Page request threw an unchecked error! This usually reflects a severe problem with the application.
java.lang.ThreadDeath
at java.lang.Thread.stop(Thread.java:853)
at lucee.commons.io.StopThread.run(SystemUtil.java:1497)

22-May-2019 22:35:02.431 SEVERE [ajp-nio-8009-exec-30] org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun
java.lang.ThreadDeath
at java.lang.Thread.stop(Thread.java:853)
at lucee.commons.io.StopThread.run(SystemUtil.java:1497)

22-May-2019 22:35:07.429 WARNING [ajp-nio-8009-exec-28] com.seefusion.Filter.doFilter Page request threw an unchecked error! This usually reflects a severe problem with the application.
java.lang.ThreadDeath
at java.lang.Thread.stop(Thread.java:853)
at lucee.commons.io.StopThread.run(SystemUtil.java:1497)

22-May-2019 22:35:07.429 SEVERE [ajp-nio-8009-exec-28] org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun
java.lang.ThreadDeath
at java.lang.Thread.stop(Thread.java:853)
at lucee.commons.io.StopThread.run(SystemUtil.java:1497)

In doing more digging tonight… Same issue on 5.3.1.95…

However, using this pre-release things ran allot better… any ideas when this might become an official release??? 5.3.2.74-RC

Actually, nevermind no pages load with RC, I just get a bunch of errors. Due to a hoth.cfc we use.

lucee.runtime.exp.NativeException: org.osgi.framework.BundleException:Duplicate import: com.sun.jdmk.comm
at org.apache.felix.framework.util.manifestparser.ManifestParser.normalizeImportClauses(ManifestParser.java:370)
at org.apache.felix.framework.util.manifestparser.ManifestParser.(ManifestParser.java:181)
at org.apache.felix.framework.BundleRevisionImpl.(BundleRevisionImpl.java:117)
at org.apache.felix.framework.BundleImpl.createRevision(BundleImpl.java:1282)
at org.apache.felix.framework.BundleImpl.(BundleImpl.java:113)
at org.apache.felix.framework.Felix.installBundle(Felix.java:3042)
at org.apache.felix.framework.BundleContextImpl.installBundle(BundleContextImpl.java:167)
at lucee.loader.osgi.BundleUtil.installBundle(BundleUtil.java:90)
at lucee.loader.osgi.BundleUtil.addBundle(BundleUtil.java:81)
at lucee.loader.osgi.BundleUtil.addBundle(BundleUtil.java:56)
at lucee.runtime.osgi.OSGiUtil.extractAndLoadBundles(OSGiUtil.java:1835)
at lucee.runtime.PageContextImpl.getRPCClassLoader(PageContextImpl.java:3388)
at lucee.runtime.PageContextImpl.getRPCClassLoader(PageContextImpl.java:3381)
at lucee.runtime.type.util.ComponentUtil.getComponentJavaAccess(ComponentUtil.java:116)
at lucee.runtime.ComponentImpl.getJavaAccessClass(ComponentImpl.java:2020)
at lucee.runtime.ComponentImpl.getMetaData(ComponentImpl.java:1531)
at lucee.runtime.ComponentImpl.getMetaData(ComponentImpl.java:1502)
at lucee.runtime.ComponentImpl.getMetaData(ComponentImpl.java:1463)
at lucee.runtime.functions.other.GetMetaData.getMetaData(GetMetaData.java:99)
at lucee.runtime.functions.other.GetMetaData.call(GetMetaData.java:69)
at lucee.runtime.functions.other.GetMetaData.call(GetMetaData.java:55)
at object.coreconfig_cfc$cf.udfCall(/hoth/object/CoreConfig.cfc:29)
at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:106)
at lucee.runtime.type.UDFImpl._call(UDFImpl.java:342)
at lucee.runtime.type.UDFImpl.call(UDFImpl.java:215)
at lucee.runtime.ComponentImpl._call(ComponentImpl.java:682)
at lucee.runtime.ComponentImpl._call(ComponentImpl.java:578)
at lucee.runtime.SuperComponent.call(SuperComponent.java:83)
at lucee.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:768)
at lucee.runtime.PageContextImpl.getFunction(PageContextImpl.java:1707)
at webpro.model.configs.hothconfig_cfc$cf.udfCall(/codebase/webpro/model/configs/HothConfig.cfc:76)
at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:106)
at lucee.runtime.type.UDFImpl._call(UDFImpl.java:342)
at lucee.runtime.type.UDFImpl.call(UDFImpl.java:215)
at lucee.runtime.ComponentImpl._call(ComponentImpl.java:682)
at lucee.runtime.ComponentImpl._call(ComponentImpl.java:570)
at lucee.runtime.ComponentImpl.call(ComponentImpl.java:1918)
at lucee.runtime.functions.other._CreateComponent.call(_CreateComponent.java:66)
at application_cfc$cf.udfCall1(/Application.cfc:284)
at application_cfc$cf.udfCall(/Application.cfc)
at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:106)
at lucee.runtime.type.UDFImpl._call(UDFImpl.java:342)
at lucee.runtime.type.UDFImpl.call(UDFImpl.java:215)
at lucee.runtime.ComponentImpl._call(ComponentImpl.java:682)
at lucee.runtime.ComponentImpl._call(ComponentImpl.java:570)
at lucee.runtime.ComponentImpl.call(ComponentImpl.java:1918)
at lucee.runtime.listener.ModernAppListener.call(ModernAppListener.java:436)
at lucee.runtime.listener.ModernAppListener.onError(ModernAppListener.java:424)
at lucee.runtime.listener.MixedAppListener.onError(MixedAppListener.java:123)
at lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2429)
at lucee.runtime.PageContextImpl._execute(PageContextImpl.java:2399)
at lucee.runtime.PageContextImpl.executeCFML(PageContextImpl.java:2374)
at lucee.runtime.engine.Request.exe(Request.java:43)
at lucee.runtime.engine.CFMLEngineImpl._service(CFMLEngineImpl.java:1037)
at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:983)
at lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:102)
at lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.seefusion.Filter.doFilter(Filter.java:109)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at mod_cfml.core.invoke(core.java:152)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at com.seefusion.SeeFusionValve.invoke(SeeFusionValve.java:52)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:479)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: org.osgi.framework.BundleException:Duplicate import: com.sun.jdmk.comm
… 85 more

Latest snapshot gives me same thing… guessing we have to update HOTH in the future.

Lucee 5.3.3.43-SNAPSHOT Error (java.io.IOException)
Message org.osgi.framework.BundleException:Duplicate import: com.sun.jdmk.comm
Stacktrace The Error Occurred in
/src2/master/AppCode/thirdparty/Hoth-master/object/CoreConfig.cfc: line 29

27: public function init ()
28: {
29: local.md = getMetadata(this);
30: local.n = arrayLen(local.md.properties);
31: for (local.i=1; local.i <= local.n; local.i++) {
called from /src2/master/AppCode/webpro/model/configs/HothConfig.cfc: line 76
called from /www2/weblink/Application.cfc: line 284

Same issue with 5.2.9.31

Have to download anything older. Suggestions on where to try next? Or anything I can provide log wise to determine issue.

NOTE: I need to get switched to these new lucee5 servers ASAP. Looking for any suggestions.

where does com.sun.jdmk.comm get referenced? I can’t see it in Hoth?

Looking to see what I can find this morning. Will reach out to developers.

are you loading a custom version of log4j or something else which loads log4j?

http://apache-felix.18485.x6.nabble.com/Duplicate-import-td4837280.html

I did find a couple instances and am waiting for dev to get back to me. I only see this issue with the release candidate version. Not seeing these errors in the logs with 5.3.1.102, but maybe it is hidden and causing these same issues?