S3 issues when trying to create a thumbnail

nzflew · September 16, 2023, 9:27pm

I am having an issue with writing images to S3. I am finding that it works for a day or so then starts to have errors such as the one below. Restarting the lucee server sorts the issue out until it starts having the same issue again.

The S3 bucket has something like 500k+ files in this particular directory. It has been working fine up until a month or so ago.

Lucee Server: Lucee 5.4.3.2
OS: Ubuntu 22.04.3 (EC2 Instance)
S3 Resource Extension 0.9.4.156

lucee.runtime.exp.NativeException: destination folder [ directoryname ] doesn’t exist at org.lucee.extension.image.Image.writeOut(Image.java:1015) at org.lucee.extension.image.functions.ImageWrite.call(ImageWrite.java:62) at org.lucee.extension.image.functions.ImageWrite.invoke(ImageWrite.java:74) at lucee.runtime.functions.

Any help or suggestions would be appreciated.

Zackster · September 18, 2023, 10:09am

try the new v2 S3 extension? 2.0.1.10-SNAPSHOT

the 0.9.4 uses an old library and there’s been a lot of work on the newer v2 version

nzflew · September 24, 2023, 10:23pm

Thanks Zackster,

I have updated to the 2.0.1.10-SNAPSHOT, the S3 folder is reasonably large probably 400GB made up of 400k+ objects. The error is still occurring

The error seems to occur when I try to do several actions in a row on files, ie. upload 10 files and create thumbnails for each file, it will fail after about 10 files.

The error I am getting is “com.amazonaws.AbortedException”

lucee.runtime.exp.NativeException: com.amazonaws.AbortedException at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleInterruptedException(AmazonHttpClient.java:880) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:757) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5470) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5417) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5411) at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:1027) at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:1033) at org.lucee.extension.resource.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:1376) at org.lucee.extension.resource.s3.S3.list(S3.java:447) at org.lucee.extension.resource.s3.S3.get(S3.java:959) at org.lucee.extension.resource.s3.S3.exists(S3.java:768

Zackster · September 25, 2023, 8:07am

can you share the full stack trace?

could be a timeout, any idea about how long these processes are taking?

nzflew · September 25, 2023, 8:41pm

The server is just an EC2 instance so the connection to S3 should be good. I might try writing to a new folder within the bucket and see if that helps. Anywway find the stacktrace below, that might offer some more insight.

lucee.runtime.exp.NativeException: com.amazonaws.AbortedException at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleInterruptedException(AmazonHttpClient.java:880) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:757) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5470) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5417) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5411) at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:1027) at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:1033) at org.lucee.extension.resource.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:1376) at org.lucee.extension.resource.s3.S3.list(S3.java:447) at org.lucee.extension.resource.s3.S3.get(S3.java:959) at org.lucee.extension.resource.s3.S3.exists(S3.java:768) at org.lucee.extension.resource.s3.S3Resource.exists(S3Resource.java:325) at org.lucee.extension.resource.s3.S3Resource.getOutputStream(S3Resource.java:253) at lucee.commons.io.res.type.file.FileResource.copyTo(FileResource.java:116) at lucee.commons.io.IOUtil.copy(IOUtil.java:140) at lucee.runtime.tag.FileTag.actionCopy(FileTag.java:586) at lucee.runtime.tag.FileTag.doStartTag(FileTag.java:424) at cfc.module_general_cfc$cf$2.udfCall2(/cmsintranet/cfc/module_general.cfc:540) at cfc.module_general_cfc$cf$2.udfCall(/cmsintranet/cfc/module_general.cfc) at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:112) at lucee.runtime.type.UDFImpl._call(UDFImpl.java:350) at lucee.runtime.type.UDFImpl.callWithNamedValues(UDFImpl.java:213) at lucee.runtime.ComponentImpl._call(ComponentImpl.java:699) at lucee.runtime.ComponentImpl._call(ComponentImpl.java:586) at lucee.runtime.ComponentImpl.callWithNamedValues(ComponentImpl.java:1952) at lucee.runtime.util.VariableUtilImpl.callFunctionWithNamedValues(VariableUtilImpl.java:866) at lucee.runtime.PageContextImpl.getFunctionWithNamedValues(PageContextImpl.java:1794) at views.bs_general_cfm$cf.call(/cmsintranet/views/bs_general.cfm:44) at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1056) at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:948) at lucee.runtime.PageContextImpl.doInclude(PageContextImpl.java:940) at lucee.runtime.tag.CFTag.doInclude(CFTag.java:319) at lucee.runtime.tag.CFTag.cfmlStartTag(CFTag.java:245) at lucee.runtime.tag.CFTag.doStartTag(CFTag.java:179) at backend.api_cfm$cf.call(/backend/api.cfm:141) at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1056) at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:948) at lucee.runtime.PageContextImpl.doInclude(PageContextImpl.java:929) at application_cfc$cf.udfCall(/Application.cfc:212) at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:112) at lucee.runtime.type.UDFImpl._call(UDFImpl.java:350) at lucee.runtime.type.UDFImpl.call(UDFImpl.java:223) at lucee.runtime.ComponentImpl._call(ComponentImpl.java:698) at lucee.runtime.ComponentImpl._call(ComponentImpl.java:586) at lucee.runtime.ComponentImpl.call(ComponentImpl.java:1933) at lucee.runtime.listener.ModernAppListener.call(ModernAppListener.java:444) at lucee.runtime.listener.ModernAppListener._onRequest(ModernAppListener.java:218) at lucee.runtime.listener.MixedAppListener.onRequest(MixedAppListener.java:44) at lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2493) at lucee.runtime.PageContextImpl._execute(PageContextImpl.java:2478) at lucee.runtime.PageContextImpl.executeCFML(PageContextImpl.java:2449) at lucee.runtime.engine.Request.exe(Request.java:45) at lucee.runtime.engine.CFMLEngineImpl._service(CFMLEngineImpl.java:1216) at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:1162) at lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:97) at lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:51) at javax.servlet.http.HttpServlet.service(HttpServlet.java:623) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:926) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Unknown Source) Caused by: com.amazonaws.AbortedException: … 82 more Caused by: com.amazonaws.http.timers.client.SdkInterruptedException at com.amazonaws.http.AmazonHttpClient$RequestExecutor.checkInterrupted(AmazonHttpClient.java:935) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.checkInterrupted(AmazonHttpClient.java:921) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1115) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)

micstriit · September 26, 2023, 4:47pm

i think the action done when the exception is thrown is not the origin of the issue. this is caused by something else. looks to me like a timeout issue. what you see in the stacktrace that lucee list the buckets in order to check if a certain bucket exist, that is done that way because of access restrictions to find this out another way.
Workaround
You write a file to S3 with help of the tag , doing that way is a very generic approach and because of that comes with some overhead (lot of check like fileexists is done by the tag and maybe is not really necessary in your case). Is always better to use the “s3” function that give you a more direct access to S3, in that example it would be most likely one of the following

nzflew · October 8, 2023, 8:56am

Thanks…I have tried adjusting my code slightly to the following

If I want to try writing it using s3write I am using the following code, but struggling to get it to write to the correct location

this is dumping it in the root of the bucket, if I want to write it into a sub-directory how would I do that, I have tried a few different things but no success.

Gavin_Baumanis · October 31, 2023, 9:03pm

I realise you’re not using the AWS CLI…
But I do - and to get the destination to be a “folder” of the bucket - I include the folder in the objectName portion of the command.

So your code would be;
<cfset s3write(bucketName=***bucketname***,objectName="myFolder/mySubFolder/#variables.thumbfilename#",value=objThumbnail) />