Hi community,
For the last 2 weeks our production app has been getting random “org.apache.catalina.connector.ClientAbortException” errors.
Half the time the CFERROR is “Broken Pipe” and the other half the time it’s “Reset by peer”.
Hope one of the geniuses can help! @andreas, @Zackster et al!
We were able to recreate the issue by doing load tests using loader.io:—
Not all requests generated an error. 25 requests out of 250 requests led to server-side errors. Similarly, in load testing with Loadium.io, 23 requests generated errors out of 250 requests.
All templates contain a CFFLUSH tag, with a 0.5kb interval.
It seems that when requests are abruptly aborted and the request has already been flushed, a Lucee error is thrown. Why doesn’t it fail silently? Is this expected behaviour? Should we simply adapt our app’s error handling to silence this error?
With removal of this setting (no CFFLUSH), we were unable to recreate the issue.
The errors all look like this:—
Specimen:—
type : org.apache.catalina.connector.ClientAbortException
browser : Mozilla/5.0 (iPhone; CPU iPhone OS 15_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.4 Mobile/15E148 Safari/604.1
StackTrace : lucee.runtime.exp.NativeException: java.io.IOException: Connection reset by peer\n\tat org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:309)\n\tat org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:272)\n\tat org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118)\n\tat lucee.runtime.writer.CFMLWriterImpl._flush(CFMLWriterImpl.java:275)\n\tat lucee.runtime.writer.CFMLWriterImpl._check(CFMLWriterImpl.java:91)\n\tat lucee.runtime.writer.CFMLWriterImpl.print(CFMLWriterImpl.java:457)\n\tat lucee.runtime.writer.CFMLWriterWSPref.print(CFMLWriterWSPref.java:197)\n\tat lucee.runtime.writer.CFMLWriterWSPref.write(CFMLWriterWSPref.java:442)\n\tat lucee.runtime.writer.CFMLWriterWSPref.write(CFMLWriterWSPref.java:475)\n\tat lucee.runtime.PageContextImpl.write(PageContextImpl.java:756)\n\tat index_cfm$cf$3f.call(/index.cfm:32)\n\tat lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1034)\n\tat lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:926)\n\tat lucee.runtime.listener.ClassicAppListener._onRequest(ClassicAppListener.java:65)\n\tat lucee.runtime.listener.MixedAppListener.onRequest(MixedAppListener.java:45)\n\tat lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2460)\n\tat lucee.runtime.PageContextImpl._execute(PageContextImpl.java:2450)\n\tat lucee.runtime.PageContextImpl.executeCFML(PageContextImpl.java:2421)\n\tat lucee.runtime.engine.Request.exe(Request.java:45)\n\tat lucee.runtime.engine.CFMLEngineImpl._service(CFMLEngineImpl.java:1179)\n\tat lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:1125)\n\tat lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:97)\n\tat lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:51)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:741)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)\n\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)\n\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)\n\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)\n\tat org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)\n\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)\n\tat org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)\n\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)\n\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)\n\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590)\n\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\nCaused by: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Connection reset by peer\n\t... 45 more\nCaused by: java.io.IOException: Connection reset by peer\n\tat java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)\n\tat java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)\n\tat java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113)\n\tat java.base/sun.nio.ch.IOUtil.write(IOUtil.java:79)\n\tat java.base/sun.nio.ch.IOUtil.write(IOUtil.java:50)\n\tat java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:466)\n\tat org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:138)\n\tat org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)\n\tat org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:152)\n\tat org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1253)\n\tat org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:740)\n\tat org.apache.tomcat.util.net.SocketWrapperBase.flushBlocking(SocketWrapperBase.java:693)\n\tat org.apache.tomcat.util.net.SocketWrapperBase.flush(SocketWrapperBase.java:683)\n\tat org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.flush(Http11OutputBuffer.java:564)\n\tat org.apache.coyote.http11.filters.IdentityOutputFilter.flush(IdentityOutputFilter.java:117)\n\tat org.apache.coyote.http11.Http11OutputBuffer.flush(Http11OutputBuffer.java:217)\n\tat org.apache.coyote.http11.Http11Processor.flush(Http11Processor.java:1155)\n\tat org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:399)\n\tat org.apache.coyote.Response.action(Response.java:209)\n\tat org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:305)\n\t... 44 more\n
In every (Lucee) error generated, the Request Headers reports “connection: close” (rather than “connection: keep-alive”):—
OS: Ubuntu Linux 20.04.4.
Java Version: 11.0.7 (AdoptOpenJDK) 64bit.
Tomcat Version: Apache Tomcat/9.0.35.
Lucee Version: 5.3.9.141.
Webmin: 1.994.
Apache version 2.4.41.
RAM: 2.23 GiB used / 2.32 GiB cached / 7.66 GiB total.
In all cases, these are not long-running processes; they are normal-timed processes of 0.X-2 seconds. Our TomCat config is set to a 20000 second timeout, so I don’t believe this is a timeout issue, but rather a request abortion issue.
We’ve not received any user feedback about website inaccessibility. We don’t have a CloudFlare reverse proxy on this host.