Trouble writing image to AWS S3

Hi,

I’m having trouble writing an image into S3 where the object name includes
a “sub-folder”.

This works:

imgWritepath
= “s3://#myAccessKeyId#:#mySecretKey#@#myBucket#/image1.jpg”;
ImageWrite(myImg, imgWritePath, 1, true);

This does not:

imgWritepath
= “s3://#myAccessKeyId#:#mySecretKey#@#myBucket#/000/image1.jpg”;
ImageWrite(myImg, imgWritePath, 1, true);

The second one returns error:
“can’t write file [s3://[secretId]:[secretKey]@/[bucketName]/000/1062.jpg]
as a file,
missing parent directory [s3://[secretId]:[secretKey]@/[bucketName]/000]”

There *seems *to be a spurious “/” inserted after the @ symbol. I didn’t
put it there…

Mike

I’m pretty sure you need to explicitly create the subfolder if it does not yet exist.

imgWritepath = "s3://#myAccessKeyId#:#mySecretKey#@#myBucket#/000"; imgWritefile = "image1.jpg";

if (!directoryExists(imgWritepath))
directorycreate(imgWritepath)

ImageWrite(myImg, imgWritePath & “/” & imgWritefile, 1, true);
> On Nov 3, 2015, at 2:42 PM, Michael Wood <@Michael_Wood> wrote:

Hi,

I’m having trouble writing an image into S3 where the object name includes a “sub-folder”.

This works:

imgWritepath = “s3://#myAccessKeyId#:#mySecretKey#@#myBucket#/image1.jpg”;
ImageWrite(myImg, imgWritePath, 1, true);

This does not:

imgWritepath = “s3://#myAccessKeyId#:#mySecretKey#@#myBucket#/000/image1.jpg”;
ImageWrite(myImg, imgWritePath, 1, true);

The second one returns error:
“can’t write file [s3://[secretId]:[secretKey]@/[bucketName]/000/1062.jpg] as a file,
missing parent directory [s3://[secretId]:[secretKey]@/[bucketName]/000]”

There seems to be a spurious “/” inserted after the @ symbol. I didn’t put it there…

Mike


Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html

You received this message because you are subscribed to the Google Groups “Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/ab473c81-3dac-443c-b420-ec7cb59c568d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

There aren’t actually sub-folders in S3; only objects. But I did try
pre-creating the that folder by manually creating another object with that
folder name. It didn’t fix the issue.On Tuesday, November 3, 2015 at 11:47:03 AM UTC-8, Sean Daniels wrote:

I’m pretty sure you need to explicitly create the subfolder if it does not
yet exist.

imgWritepath = "s3://#myAccessKeyId#:#mySecretKey#@#myBucket#/000"; imgWritefile = "image1.jpg";

if (!directoryExists(imgWritepath))
directorycreate(imgWritepath)

ImageWrite(myImg, imgWritePath & “/” & imgWritefile, 1, true);

On Nov 3, 2015, at 2:42 PM, Michael Wood <mw...@stpo.com <javascript:>> wrote:

Hi,

I’m having trouble writing an image into S3 where the object name
includes a “sub-folder”.

This works:

imgWritepath =
“s3://#myAccessKeyId#:#mySecretKey#@#myBucket#/image1.jpg”;
ImageWrite(myImg, imgWritePath, 1, true);

This does not:

imgWritepath =
“s3://#myAccessKeyId#:#mySecretKey#@#myBucket#/000/image1.jpg”;
ImageWrite(myImg, imgWritePath, 1, true);

The second one returns error:
“can’t write file
[s3://[secretId]:[secretKey]@/[bucketName]/000/1062.jpg] as a file,
missing parent directory
[s3://[secretId]:[secretKey]@/[bucketName]/000]”

There seems to be a spurious “/” inserted after the @ symbol. I didn’t
put it there…

Mike


Love Lucee? Become a supporter and be part of the Lucee project today! -
http://lucee.org/supporters/become-a-supporter.html

You received this message because you are subscribed to the Google
Groups “Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to lucee+un...@googlegroups.com <javascript:>.
To post to this group, send email to lu...@googlegroups.com
<javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/ab473c81-3dac-443c-b420-ec7cb59c568d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Hi Andrew,

Sorry about the repeated posts and deletes - I’m trying to figure out how
to use this interface.

Thank you for your suggestion and help. Your suggestion REALLY helped but
it took me the intervening time to figure out how to
get/install/configure/implement it.

So the Lucee-AWS documentation was of limited use but then I’m not an
expert on image processing or handling. To potentially save others my pain
here’s some instructions and sample code:

  1. Get the Lucee-AWS extension installed on your box however you can.
    There’s limited help at https://github.com/mso-net/lucee-aws. I installed
    and used the recommended CommandBox tool but figuring out how to install
    CommandBox on my remote Linux server was a bit if a puzzle to me in itself.

  2. Once Lucee-AWS is installed then you can put it to use - And it’s pretty
    simple to use once installed BUT only if you know that you need to use the
    binary read and base64 conversions.

  3. You add this to your Application.cfc file:
    this.javaSettings = { loadPaths: [ ‘/aws/aws-java-sdk/’]};

  4. Here’s sample code how to USE the extension in a CFM template.

// Setup to use the method calls s3 = new aws.s3( account = '#awsAccessKeyId#', secret = '#awsSecretKey#', bucket = 'myBucket' );

// To read an image from S3
myObj = s3.getObject( ‘myImage.jpg’ );
myImg = imageReadBase64(myObj.content);
// You can now treat myImg as a ColdFusion image

// To write an image to S3 first read the image file as binary and then
convert it to base64
myImg = toBase64(fileReadBinary(‘mytest.jpg’));
// This is how you send the file to S3 - Note that you have to insert
the “data:image/jpg;base64,” in front of the image base64 string.
// Change the file extension as needed but I’m pretty sure it has to
match the extension of the file you read in, perhaps not
s3.putObject( ‘myFolder/myImage.jpg’ , ‘data:image/jpg;base64,#myImg#’ );
// Delete an image object from S3
s3.deleteObject( ‘myFolder/myImage.jpg’ );

Mike>

Pretty sure like with most things there are advantages and disadvantages to
all ways of doing it.

If however you do go down the route of uploading directly to S3 I would
warn you that doing image manipulation on an image stored on S3 already via
Lucee is SLOW!!! If you are going to upload directly to S3 I would suggest
the Lambda route would be far quicker.

Kind regards,

Andrew
about.me http://about.me/andrew_dixon - mso http://www.mso.net - Lucee
Association Member http://lucee.orgOn 4 November 2015 at 20:44, Michael Wood <@Michael_Wood> wrote:

Guys, is there a disadvantage to doing this the way I’m doing it; that
is uploading to an EC2 instance, processing the images, and then moving
them into S3?

Or saying it the other way around, are there potential compelling
advantages to uploading straight to S3 and then post-processing the images?

Our pre-processing in EC2 includes image/file type validation, security
checks, image conversion to jpg (as a standard), image re-sizing, creation
of a thumbnail, AND we do a web service call to get an id to name the file
which also determines a “folder name” to use in the object name for S3. It
seems like we will have a lot more control this way. And amazingly all of
this happens pretty quickly. Of course I haven’t had an opportunity to do
load testing yet. Uploading directly to S3 might be more robust.

The other advantage to pre-processing is that the end user immediately
SEES the uploaded/processed image and thumbnail (OR the failure to
upload/process and can try again).

Mike Wood

On Wednesday, November 4, 2015 at 10:55:49 AM UTC-8, Pete Freitag wrote:

As Michael Van Leest suggested you could have another bucket that
receives the raw uploads, and then process those using a bucket event
notification:
https://docs.aws.amazon.com/AmazonS3/latest/UG/SettingBucketNotifications.html
which would be a callback that AWS makes to your server after the upload is
complete.

A lambda function would probably be the ideal solution for this, but if
you want to process it on your server you can do a bucket event instead.

Pete Freitag


Love Lucee? Become a supporter and be part of the Lucee project today! -
http://lucee.org/supporters/become-a-supporter.html


You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/9e021b2f-edf8-4dc8-a469-89e0999a5350%40googlegroups.com
https://groups.google.com/d/msgid/lucee/9e021b2f-edf8-4dc8-a469-89e0999a5350%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

As Michael Van Leest suggested you could have another bucket that receives
the raw uploads, and then process those using a bucket event notification:
https://docs.aws.amazon.com/AmazonS3/latest/UG/SettingBucketNotifications.html
which would be a callback that AWS makes to your server after the upload is
complete.

A lambda function would probably be the ideal solution for this, but if you
want to process it on your server you can do a bucket event instead.–
Pete Freitag
https://foundeo.com/ http://foundeo.com/ - ColdFusion Consulting &
Products
http://hackmycf.com - CFML Server Security Scanner

On Wed, Nov 4, 2015 at 12:48 PM, Michael Wood <@Michael_Wood> wrote:

Hi Pete!

Oh I wish I could but we do processing on the images (re-size, create
thumbnails, etc) in EC2 before storing them in S3. I can’t think of another
way to pre-process the images before storing them in S3, can you?

This is something new we’re trying. We have the upload pages working in an
iFrame so that the upload and image processing is offloaded entirely to the
client and to EC2/S3. They will communicate directly freeing this load from
our main server. Running Lucee on Amazon Linux in EC2 and using S3 for
storage made this idea possible/practical/affordable…

I *wish *we could do this in Lambda but I’m pretty sure the programming
required is too far out of my skill set.

Mike

On Wednesday, November 4, 2015 at 6:56:44 AM UTC-8, Pete Freitag wrote:

You can also upload the file directly to S3 from the client’s browser,
here’s how you can do it in CFML:
https://www.petefreitag.com/item/833.cfm this approach is nice because
it does not utilize your server resources for the upload, but may be a bit
more work to get going.


Pete Freitag
https://foundeo.com/ http://foundeo.com/ - ColdFusion Consulting &
Products
http://hackmycf.com - CFML Server Security Scanner


Love Lucee? Become a supporter and be part of the Lucee project today! -
http://lucee.org/supporters/become-a-supporter.html

You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/0fd09b98-36e4-4bdb-a51e-125495f8a636%40googlegroups.com
https://groups.google.com/d/msgid/lucee/0fd09b98-36e4-4bdb-a51e-125495f8a636%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

Guys, is there a disadvantage to doing this the way I’m doing it; that is
uploading to an EC2 instance, processing the images, and then moving them
into S3?

Or saying it the other way around, are there potential compelling
advantages to uploading straight to S3 and then post-processing the images?

Our pre-processing in EC2 includes image/file type validation, security
checks, image conversion to jpg (as a standard), image re-sizing, creation
of a thumbnail, AND we do a web service call to get an id to name the file
which also determines a “folder name” to use in the object name for S3. It
seems like we will have a lot more control this way. And amazingly all of
this happens pretty quickly. Of course I haven’t had an opportunity to do
load testing yet. Uploading directly to S3 might be more robust.

The other advantage to pre-processing is that the end user immediately SEES
the uploaded/processed image and thumbnail (OR the failure to
upload/process and can try again).

Mike WoodOn Wednesday, November 4, 2015 at 10:55:49 AM UTC-8, Pete Freitag wrote:

As Michael Van Leest suggested you could have another bucket that receives
the raw uploads, and then process those using a bucket event notification:
https://docs.aws.amazon.com/AmazonS3/latest/UG/SettingBucketNotifications.html
which would be a callback that AWS makes to your server after the upload is
complete.

A lambda function would probably be the ideal solution for this, but if
you want to process it on your server you can do a bucket event instead.

Pete Freitag

HI Michael,

You could use the AWS Lambda service to process the images directly on AWS
with the Lambda function triggered by the “store” event in S3, it is one of
the examples in the AWS Lambda docs:

http://docs.aws.amazon.com/lambda/latest/dg/java-wt-s3-log-event-data.html

Kind regards,

Andrew
about.me http://about.me/andrew_dixon - mso http://www.mso.net - Lucee
Association Member http://lucee.orgOn 4 November 2015 at 17:48, Michael Wood <@Michael_Wood> wrote:

Hi Pete!

Oh I wish I could but we do processing on the images (re-size, create
thumbnails, etc) in EC2 before storing them in S3. I can’t think of another
way to pre-process the images before storing them in S3, can you?

This is something new we’re trying. We have the upload pages working in an
iFrame so that the upload and image processing is offloaded entirely to the
client and to EC2/S3. They will communicate directly freeing this load from
our main server. Running Lucee on Amazon Linux in EC2 and using S3 for
storage made this idea possible/practical/affordable…

I *wish *we could do this in Lambda but I’m pretty sure the programming
required is too far out of my skill set.

Mike

On Wednesday, November 4, 2015 at 6:56:44 AM UTC-8, Pete Freitag wrote:

You can also upload the file directly to S3 from the client’s browser,
here’s how you can do it in CFML:
https://www.petefreitag.com/item/833.cfm this approach is nice because
it does not utilize your server resources for the upload, but may be a bit
more work to get going.


Pete Freitag
https://foundeo.com/ http://foundeo.com/ - ColdFusion Consulting &
Products
http://hackmycf.com - CFML Server Security Scanner


Love Lucee? Become a supporter and be part of the Lucee project today! -
http://lucee.org/supporters/become-a-supporter.html

You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/0fd09b98-36e4-4bdb-a51e-125495f8a636%40googlegroups.com
https://groups.google.com/d/msgid/lucee/0fd09b98-36e4-4bdb-a51e-125495f8a636%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

You can also upload the file directly to S3 from the client’s browser,
here’s how you can do it in CFML: https://www.petefreitag.com/item/833.cfm
this approach is nice because it does not utilize your server resources
for the upload, but may be a bit more work to get going.–
Pete Freitag
https://foundeo.com/ http://foundeo.com/ - ColdFusion Consulting &
Products
http://hackmycf.com - CFML Server Security Scanner

On Tue, Nov 3, 2015 at 3:00 PM, Andrew Dixon <@Andrew_Dixon> wrote:

Hi Michael,

S3 is problematic using the built in functions, check out this project:

https://github.com/mso-net/lucee-aws

Kind regards,

Andrew
about.me http://about.me/andrew_dixon - mso http://www.mso.net - Lucee
Association Member http://lucee.org

On 3 November 2015 at 19:52, Michael Wood <@Michael_Wood> wrote:

There aren’t actually sub-folders in S3; only objects. But I did try
pre-creating the that folder by manually creating another object with that
folder name. It didn’t fix the issue.

On Tuesday, November 3, 2015 at 11:47:03 AM UTC-8, Sean Daniels wrote:

I’m pretty sure you need to explicitly create the subfolder if it does
not yet exist.

imgWritepath = "s3://#myAccessKeyId#:#mySecretKey#@#myBucket#/000"; imgWritefile = "image1.jpg";

if (!directoryExists(imgWritepath))
directorycreate(imgWritepath)

ImageWrite(myImg, imgWritePath & “/” & imgWritefile, 1, true);

On Nov 3, 2015, at 2:42 PM, Michael Wood mw...@stpo.com wrote:

Hi,

I’m having trouble writing an image into S3 where the object name
includes a “sub-folder”.

This works:

imgWritepath =
“s3://#myAccessKeyId#:#mySecretKey#@#myBucket#/image1.jpg”;
ImageWrite(myImg, imgWritePath, 1, true);

This does not:

imgWritepath =
“s3://#myAccessKeyId#:#mySecretKey#@#myBucket#/000/image1.jpg”;
ImageWrite(myImg, imgWritePath, 1, true);

The second one returns error:
“can’t write file
[s3://[secretId]:[secretKey]@/[bucketName]/000/1062.jpg] as a file,
missing parent directory
[s3://[secretId]:[secretKey]@/[bucketName]/000]”

There seems to be a spurious “/” inserted after the @ symbol. I didn’t
put it there…

Mike


Love Lucee? Become a supporter and be part of the Lucee project today!


You received this message because you are subscribed to the Google
Groups “Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/ab473c81-3dac-443c-b420-ec7cb59c568d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


Love Lucee? Become a supporter and be part of the Lucee project today! -
http://lucee.org/supporters/become-a-supporter.html


You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/800d48fb-3e80-4a7f-82b9-494d92ed1f98%40googlegroups.com
https://groups.google.com/d/msgid/lucee/800d48fb-3e80-4a7f-82b9-494d92ed1f98%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.


Love Lucee? Become a supporter and be part of the Lucee project today! -
http://lucee.org/supporters/become-a-supporter.html

You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/CAG1WijXC_WPuxg6TU3j9kNRgHu3eT0gonFOGDb4DbA3TdK5yQw%40mail.gmail.com
https://groups.google.com/d/msgid/lucee/CAG1WijXC_WPuxg6TU3j9kNRgHu3eT0gonFOGDb4DbA3TdK5yQw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

Hi Pete!

Oh I wish I could but we do processing on the images (re-size, create
thumbnails, etc) in EC2 before storing them in S3. I can’t think of another
way to pre-process the images before storing them in S3, can you?

This is something new we’re trying. We have the upload pages working in an
iFrame so that the upload and image processing is offloaded entirely to the
client and to EC2/S3. They will communicate directly freeing this load from
our main server. Running Lucee on Amazon Linux in EC2 and using S3 for
storage made this idea possible/practical/affordable…

I *wish *we could do this in Lambda but I’m pretty sure the programming
required is too far out of my skill set.

MikeOn Wednesday, November 4, 2015 at 6:56:44 AM UTC-8, Pete Freitag wrote:

You can also upload the file directly to S3 from the client’s browser,
here’s how you can do it in CFML: https://www.petefreitag.com/item/833.cfm
this approach is nice because it does not utilize your server resources
for the upload, but may be a bit more work to get going.


Pete Freitag
https://foundeo.com/ http://foundeo.com/ - ColdFusion Consulting &
Products
http://hackmycf.com - CFML Server Security Scanner

@Michael Wood: Is a intermediate upload bucket an option? After the upload
is done, you can get the image from S3, process the image and place it in
the final bucket?2015-11-04 18:48 GMT+01:00 Michael Wood <@Michael_Wood>:

Hi Pete!

Oh I wish I could but we do processing on the images (re-size, create
thumbnails, etc) in EC2 before storing them in S3. I can’t think of another
way to pre-process the images before storing them in S3, can you?

This is something new we’re trying. We have the upload pages working in an
iFrame so that the upload and image processing is offloaded entirely to the
client and to EC2/S3. They will communicate directly freeing this load from
our main server. Running Lucee on Amazon Linux in EC2 and using S3 for
storage made this idea possible/practical/affordable…

I *wish *we could do this in Lambda but I’m pretty sure the programming
required is too far out of my skill set.

Mike

On Wednesday, November 4, 2015 at 6:56:44 AM UTC-8, Pete Freitag wrote:

You can also upload the file directly to S3 from the client’s browser,
here’s how you can do it in CFML:
https://www.petefreitag.com/item/833.cfm this approach is nice because
it does not utilize your server resources for the upload, but may be a bit
more work to get going.


Pete Freitag
https://foundeo.com/ http://foundeo.com/ - ColdFusion Consulting &
Products
http://hackmycf.com - CFML Server Security Scanner


Love Lucee? Become a supporter and be part of the Lucee project today! -
http://lucee.org/supporters/become-a-supporter.html

You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/0fd09b98-36e4-4bdb-a51e-125495f8a636%40googlegroups.com
https://groups.google.com/d/msgid/lucee/0fd09b98-36e4-4bdb-a51e-125495f8a636%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.


Michael van Leest