Periodic issue leading to timeout errors


#1

We had an incident where our Lucee server started returning a bunch of timeout errors that are not related to any page request and for which we can’t find an explanation. There were 100 or so similar errors over 50 mins, then the rate reduced for a few hours before we restarted the server. We last saw this a month ago and when we didn’t restart the server, the issue progressed to IIS returning 403 responses (dynamic IP filtering restrictions that should not have been triggered) and then 503 responses. Restarting the server makes the issue disappear for a month or two. Stack trace is below… was hoping someone might make sense of this and have some suggestions for preventing a recurrence. We’re using Lucee 5.2.5 on Windows 2016.
Thanks… Simon

“ERROR”,“Thread-1610239”,“08/01/2018”,“00:51:53”,“controler”,"stop thread (11548) because run into a timeout (no path available).;java.lang.Throwable;java.lang.Throwable
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.awaitLatch(NioEndpoint.java:1114)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.awaitReadLatch(NioEndpoint.java:1116)
at org.apache.tomcat.util.net.NioBlockingSelector.read(NioBlockingSelector.java:184)
at org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:235)
at org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:216)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1241)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1221)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.read(NioEndpoint.java:1158)
at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:1309)
at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:625)
at org.apache.coyote.ajp.AjpProcessor.receive(AjpProcessor.java:587)
at org.apache.coyote.ajp.AjpProcessor.refillReadBuffer(AjpProcessor.java:695)
at org.apache.coyote.ajp.AjpProcessor$SocketInputBuffer.doRead(AjpProcessor.java:1433)
at org.apache.coyote.Request.doRead(Request.java:581)
at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:326)
at org.apache.catalina.connector.InputBuffer.checkByteBufferEof(InputBuffer.java:642)
at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:349)
at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:152)
at lucee.commons.io.IOUtil.copy(IOUtil.java:283)
at lucee.commons.io.IOUtil.copy(IOUtil.java:72)
at lucee.commons.io.IOUtil.toBytes(IOUtil.java:943)
at lucee.commons.io.IOUtil.toBytes(IOUtil.java:937)
at lucee.runtime.net.http.HTTPServletRequestWrap.getInputStream(HTTPServletRequestWrap.java:256)
at lucee.runtime.net.http.HTTPServletRequestWrap.getReader(HTTPServletRequestWrap.java:355)
at lucee.runtime.type.scope.FormImpl.initializeUrlEncodedOrTextPlain(FormImpl.java:251)
at lucee.runtime.type.scope.FormImpl.initialize(FormImpl.java:131)
at lucee.runtime.PageContextImpl.formScope(PageContextImpl.java:1175)
at lucee.runtime.type.scope.UndefinedImpl.reinitialize(UndefinedImpl.java:611)
at lucee.runtime.type.scope.UndefinedImpl.initialize(UndefinedImpl.java:589)
at lucee.runtime.PageContextImpl.initialize(PageContextImpl.java:522)
at lucee.runtime.CFMLFactoryImpl.getPageContextImpl(CFMLFactoryImpl.java:171)
at lucee.runtime.engine.CFMLEngineImpl._service(CFMLEngineImpl.java:1061)
at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:1039)
at lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:102)
at lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:486)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
"