Resource leaks with s3 and cfimage

I’ve updated the extension to 2.0.3.0 and I’m now seeing a new warning in the logs:

[WARN] S3AbortableInputStream - Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use.

This is happening while using the following cfimage resize code. The destination path is an s3:// URL. The image does upload successfully, but the warning is still logged:

<cfimage action="resize"
       height=""
       width="#imagesize#"
       source="#uploadedImage#"
       destination="#MediaPath#/#dateAppend#-#newfilename#"
       overwrite="true" />

I’ve also noticed that the previous warning below caused issues for me in the past:

[WARN] HttpMethodReleaseInputStream - Attempting to release HttpMethod in finalize() as its response data stream has gone out of scope. This attempt will not always succeed and cannot be relied upon! Please ensure response data streams are always fully consumed or closed to avoid HTTP connection starvation.
[WARN] HttpMethodReleaseInputStream - Successfully released HttpMethod in finalize(). You were lucky this time… Please ensure response data streams are always fully consumed or closed.

In that case, it appeared to hang a thread, which eventually led to a timeout and an application restart. I’m not sure if this new warning could lead to similar behavior, but I’d prefer to avoid finding out the hard way :slight_smile:

Yes, this is causing the upload endpoints to hang resulting in Lucee needing restarted. Any clue what I need to do to prevent this?

We are experiencing the same issue. But it appears to me that it is not caused by any cffile/cfdirectory tags per se, but by the cfimage tag. What might be related is that we also see some warnings about not being able to read EXIF data of PNG files. Maybe after such a warning the cfimage code closes the stream before consuming it completely?

In my dev environment warnings disappear if I copy the image to the local temp folder first, then manipulate it with cfimage and then upload it again.

The Docker 6.2.4.24-nginx-tomcat11.0-jre21-temurin-noble image BTW.

1 Like

I love error messages and stacktraces, actionable info

https://luceeserver.atlassian.net/browse/LDEV-6080

1 Like

We have updated the upload process to use cffile for transferring files to the S3 bucket, rather than using cfimage for remote uploads. cfimage is now used only for local image manipulation. This change has eliminated the warning messages and improved the overall stability of the upload process. Thank you both for your input.

I have just published SNAPSHOTS with fixes for both the image and s3 extensions, which should solve these resource leaks

Details in the tickets

Image Ext

3.0.0.7-SNAPSHOT (v7)
2.0.0.32-SNAPSHOT (v6)

https://luceeserver.atlassian.net/browse/LDEV-6080

S3 Ext

2.0.3.1-SNAPSHOT

https://luceeserver.atlassian.net/browse/LDEV-6081

1 Like

I don’t have any stack trace from the AWS S3 SDK, just the error message that Mike posted. For the issue with EXIF data I have one:

Cannot invoke "org.apache.commons.imaging.formats.tiff.TiffImageMetadata.getItems()" because "_exif" is null
at org.lucee.extension.image.Metadata.addExifInfoToStruct(Metadata.java:89)
at org.lucee.extension.image.Image.info(Image.java:523)
at org.lucee.extension.image.Image.get(Image.java:2093)
at org.lucee.extension.image.StructSupport.get(StructSupport.java:244)
at lucee.runtime.util.VariableUtilImpl.get(VariableUtilImpl.java:280)
at lucee.runtime.PageContextImpl.us(PageContextImpl.java:1310)
at bestand_cfc$cf.udfCall7(/obj/bestand.cfc:933)
at bestand_cfc$cf.udfCall(/obj/bestand.cfc)
at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:112)
at lucee.runtime.type.UDFImpl._call(UDFImpl.java:357)
at lucee.runtime.type.UDFImpl.callWithNamedValues(UDFImpl.java:214)
at lucee.runtime.ComponentImpl._call(ComponentImpl.java:695)
at lucee.runtime.ComponentImpl._call(ComponentImpl.java:611)
at lucee.runtime.ComponentImpl.callWithNamedValues(ComponentImpl.java:2142)
at lucee.runtime.util.VariableUtilImpl.callFunctionWithNamedValues(VariableUtilImpl.java:858)
at lucee.runtime.PageContextImpl.getFunctionWithNamedValues(PageContextImpl.java:2105)
at bestand_cfc$cf.udfCall5(/obj/bestand.cfc:731)
at bestand_cfc$cf.udfCall(/obj/bestand.cfc)
at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:112)
at lucee.runtime.type.UDFImpl._call(UDFImpl.java:357)
at lucee.runtime.type.UDFImpl.callWithNamedValues(UDFImpl.java:214)
at lucee.runtime.ComponentImpl._call(ComponentImpl.java:695)
at lucee.runtime.ComponentImpl._call(ComponentImpl.java:611)
at lucee.runtime.ComponentImpl.callWithNamedValues(ComponentImpl.java:2142)
at lucee.runtime.util.VariableUtilImpl.callFunctionWithNamedValues(VariableUtilImpl.java:858)
at lucee.runtime.PageContextImpl.getFunctionWithNamedValues(PageContextImpl.java:2105)
at applicatie.model.app.app_upload_bestand_toevoegen_cfm$cf.call(/applicatie/model/app/app_upload_bestand_toevoegen.cfm:38)
at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1088)
at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1012)
at lucee.runtime.PageContextImpl.doInclude(PageContextImpl.java:993)
at applicatie.beheer.homepage.fbx_switch_cfm$cf.call(/applicatie/beheer/homepage/fbx_Switch.cfm:97)
at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1088)
at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1012)
at lucee.runtime.PageContextImpl.doInclude(PageContextImpl.java:993)
at applicatie.index_cfm$cf.call(/applicatie/index.cfm:166)
at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1088)
at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1012)
at lucee.runtime.listener.ModernAppListener._onRequest(ModernAppListener.java:215)
at lucee.runtime.listener.MixedAppListener.onRequest(MixedAppListener.java:43)
at lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2816)
at lucee.runtime.PageContextImpl._execute(PageContextImpl.java:2803)
at lucee.runtime.PageContextImpl.executeCFML(PageContextImpl.java:2774)
at lucee.runtime.engine.Request.exe(Request.java:45)
at lucee.runtime.engine.CFMLEngineImpl._service(CFMLEngineImpl.java:1113)
at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:1070)
at lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:97)
at lucee.loader.servlet.jakarta.CFMLServlet.service(CFMLServlet.java:52)
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:710)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:128)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:107)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:165)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:77)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:113)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:654)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:83)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:685)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:72)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:341)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:397)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:903)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1778)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:946)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:480)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:57)
at java.base/java.lang.Thread.run(Unknown Source)

fixed in the latest snapshots

https://luceeserver.atlassian.net/browse/LDEV-6082

2 Likes

@Mike_Hall can you confirm the fixes work?

Positive feedback so far, we are moving this towards staging.

1 Like

We have been running S3 Resource Extension 2.0.3.1-SNAPSHOT and Image extension 2.0.0.33-SNAPSHOTin production for a week and have not experienced any S3 connection pool issues or stacktraces about EXIF NPEs.

6.2.4.24-light-nginx-tomcat11.0-jre21-temurin-noble with a few million hits per day.