‘Broken pipe’ errors with cfflush and incomplete HTTP requests

Hi community,

For the last 2 weeks our production app has been getting random “org.apache.catalina.connector.ClientAbortException” errors.

Half the time the CFERROR is “Broken Pipe” and the other half the time it’s “Reset by peer”.

Hope one of the geniuses can help! :eyes: @andreas, @Zackster et al!

We were able to recreate the issue by doing load tests using loader.io:—

Not all requests generated an error. 25 requests out of 250 requests led to server-side errors. Similarly, in load testing with Loadium.io, 23 requests generated errors out of 250 requests.

All templates contain a CFFLUSH tag, with a 0.5kb interval.

It seems that when requests are abruptly aborted and the request has already been flushed, a Lucee error is thrown. Why doesn’t it fail silently? Is this expected behaviour? Should we simply adapt our app’s error handling to silence this error?

With removal of this setting (no CFFLUSH), we were unable to recreate the issue.

Screenshot 2022-05-25 at 14.20.26

The errors all look like this:—

Specimen:—

type	:	org.apache.catalina.connector.ClientAbortException
browser	:	Mozilla/5.0 (iPhone; CPU iPhone OS 15_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.4 Mobile/15E148 Safari/604.1
StackTrace	:	lucee.runtime.exp.NativeException: java.io.IOException: Connection reset by peer\n\tat org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:309)\n\tat org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:272)\n\tat org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118)\n\tat lucee.runtime.writer.CFMLWriterImpl._flush(CFMLWriterImpl.java:275)\n\tat lucee.runtime.writer.CFMLWriterImpl._check(CFMLWriterImpl.java:91)\n\tat lucee.runtime.writer.CFMLWriterImpl.print(CFMLWriterImpl.java:457)\n\tat lucee.runtime.writer.CFMLWriterWSPref.print(CFMLWriterWSPref.java:197)\n\tat lucee.runtime.writer.CFMLWriterWSPref.write(CFMLWriterWSPref.java:442)\n\tat lucee.runtime.writer.CFMLWriterWSPref.write(CFMLWriterWSPref.java:475)\n\tat lucee.runtime.PageContextImpl.write(PageContextImpl.java:756)\n\tat index_cfm$cf$3f.call(/index.cfm:32)\n\tat lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1034)\n\tat lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:926)\n\tat lucee.runtime.listener.ClassicAppListener._onRequest(ClassicAppListener.java:65)\n\tat lucee.runtime.listener.MixedAppListener.onRequest(MixedAppListener.java:45)\n\tat lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2460)\n\tat lucee.runtime.PageContextImpl._execute(PageContextImpl.java:2450)\n\tat lucee.runtime.PageContextImpl.executeCFML(PageContextImpl.java:2421)\n\tat lucee.runtime.engine.Request.exe(Request.java:45)\n\tat lucee.runtime.engine.CFMLEngineImpl._service(CFMLEngineImpl.java:1179)\n\tat lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:1125)\n\tat lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:97)\n\tat lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:51)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:741)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)\n\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)\n\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)\n\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)\n\tat org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)\n\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)\n\tat org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)\n\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)\n\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)\n\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590)\n\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\nCaused by: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Connection reset by peer\n\t... 45 more\nCaused by: java.io.IOException: Connection reset by peer\n\tat java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)\n\tat java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)\n\tat java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113)\n\tat java.base/sun.nio.ch.IOUtil.write(IOUtil.java:79)\n\tat java.base/sun.nio.ch.IOUtil.write(IOUtil.java:50)\n\tat java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:466)\n\tat org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:138)\n\tat org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)\n\tat org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:152)\n\tat org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1253)\n\tat org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:740)\n\tat org.apache.tomcat.util.net.SocketWrapperBase.flushBlocking(SocketWrapperBase.java:693)\n\tat org.apache.tomcat.util.net.SocketWrapperBase.flush(SocketWrapperBase.java:683)\n\tat org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.flush(Http11OutputBuffer.java:564)\n\tat org.apache.coyote.http11.filters.IdentityOutputFilter.flush(IdentityOutputFilter.java:117)\n\tat org.apache.coyote.http11.Http11OutputBuffer.flush(Http11OutputBuffer.java:217)\n\tat org.apache.coyote.http11.Http11Processor.flush(Http11Processor.java:1155)\n\tat org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:399)\n\tat org.apache.coyote.Response.action(Response.java:209)\n\tat org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:305)\n\t... 44 more\n

In every (Lucee) error generated, the Request Headers reports “connection: close” (rather than “connection: keep-alive”):—
Screenshot 2022-05-25 at 13.46.18

OS: Ubuntu Linux 20.04.4.
Java Version: 11.0.7 (AdoptOpenJDK) 64bit.
Tomcat Version: Apache Tomcat/9.0.35.
Lucee Version: 5.3.9.141.
Webmin: 1.994.
Apache version 2.4.41.
RAM: 2.23 GiB used / 2.32 GiB cached / 7.66 GiB total.

In all cases, these are not long-running processes; they are normal-timed processes of 0.X-2 seconds. Our TomCat config is set to a 20000 second timeout, so I don’t believe this is a timeout issue, but rather a request abortion issue.

Screenshot 2022-05-25 at 13.33.57

We’ve not received any user feedback about website inaccessibility. We don’t have a CloudFlare reverse proxy on this host.

pretty sure that’s to be expected, once you have flushed once, if you try to flush again and the client is no longer listening, it will throw an error

1 Like

Thanks, @Zackster.

We’ll adapt our code.

We don’t want to stop using cfflush , as it really helps with TTFB for end users’ performance experience.

So we’ve used the snippet below to suppress errors of this kind. Posting for posterity for others.

<cfif (cferror.message contains "java.io.IOException: Connection reset by peer" OR cferror.message contains "java.io.IOException: Broken pipe") AND cferror.type is "org.apache.catalina.connector.ClientAbortException" AND cferror.StackTrace contains "org.apache.catalina.connector.OutputBuffer.flush">
1 Like