The second one returns error:
“can’t write file [s3://[secretId]:[secretKey]@/[bucketName]/000/1062.jpg]
as a file,
missing parent directory [s3://[secretId]:[secretKey]@/[bucketName]/000]”
There *seems *to be a spurious “/” inserted after the @ symbol. I didn’t
put it there…
The second one returns error:
“can’t write file [s3://[secretId]:[secretKey]@/[bucketName]/000/1062.jpg] as a file,
missing parent directory [s3://[secretId]:[secretKey]@/[bucketName]/000]”
There seems to be a spurious “/” inserted after the @ symbol. I didn’t put it there…
There aren’t actually sub-folders in S3; only objects. But I did try
pre-creating the that folder by manually creating another object with that
folder name. It didn’t fix the issue.On Tuesday, November 3, 2015 at 11:47:03 AM UTC-8, Sean Daniels wrote:
I’m pretty sure you need to explicitly create the subfolder if it does not
yet exist.
The second one returns error:
“can’t write file
[s3://[secretId]:[secretKey]@/[bucketName]/000/1062.jpg] as a file,
missing parent directory
[s3://[secretId]:[secretKey]@/[bucketName]/000]”
There seems to be a spurious “/” inserted after the @ symbol. I didn’t
put it there…
Sorry about the repeated posts and deletes - I’m trying to figure out how
to use this interface.
Thank you for your suggestion and help. Your suggestion REALLY helped but
it took me the intervening time to figure out how to
get/install/configure/implement it.
So the Lucee-AWS documentation was of limited use but then I’m not an
expert on image processing or handling. To potentially save others my pain
here’s some instructions and sample code:
Once Lucee-AWS is installed then you can put it to use - And it’s pretty
simple to use once installed BUT only if you know that you need to use the
binary read and base64 conversions.
You add this to your Application.cfc file:
this.javaSettings = { loadPaths: [ ‘/aws/aws-java-sdk/’]};
Here’s sample code how to USE the extension in a CFM template.
// Setup to use the method calls
s3 = new aws.s3(
account = '#awsAccessKeyId#',
secret = '#awsSecretKey#',
bucket = 'myBucket'
);
// To read an image from S3
myObj = s3.getObject( ‘myImage.jpg’ );
myImg = imageReadBase64(myObj.content);
// You can now treat myImg as a ColdFusion image
// To write an image to S3 first read the image file as binary and then
convert it to base64
myImg = toBase64(fileReadBinary(‘mytest.jpg’));
// This is how you send the file to S3 - Note that you have to insert
the “data:image/jpg;base64,” in front of the image base64 string.
// Change the file extension as needed but I’m pretty sure it has to
match the extension of the file you read in, perhaps not
s3.putObject( ‘myFolder/myImage.jpg’ , ‘data:image/jpg;base64,#myImg#’ );
// Delete an image object from S3
s3.deleteObject( ‘myFolder/myImage.jpg’ );
Pretty sure like with most things there are advantages and disadvantages to
all ways of doing it.
If however you do go down the route of uploading directly to S3 I would
warn you that doing image manipulation on an image stored on S3 already via
Lucee is SLOW!!! If you are going to upload directly to S3 I would suggest
the Lambda route would be far quicker.
Guys, is there a disadvantage to doing this the way I’m doing it; that
is uploading to an EC2 instance, processing the images, and then moving
them into S3?
Or saying it the other way around, are there potential compelling
advantages to uploading straight to S3 and then post-processing the images?
Our pre-processing in EC2 includes image/file type validation, security
checks, image conversion to jpg (as a standard), image re-sizing, creation
of a thumbnail, AND we do a web service call to get an id to name the file
which also determines a “folder name” to use in the object name for S3. It
seems like we will have a lot more control this way. And amazingly all of
this happens pretty quickly. Of course I haven’t had an opportunity to do
load testing yet. Uploading directly to S3 might be more robust.
The other advantage to pre-processing is that the end user immediately
SEES the uploaded/processed image and thumbnail (OR the failure to
upload/process and can try again).
Mike Wood
On Wednesday, November 4, 2015 at 10:55:49 AM UTC-8, Pete Freitag wrote:
As Michael Van Leest suggested you could have another bucket that receives
the raw uploads, and then process those using a bucket event notification:
which would be a callback that AWS makes to your server after the upload is
complete.
A lambda function would probably be the ideal solution for this, but if you
want to process it on your server you can do a bucket event instead.–
Pete Freitag https://foundeo.com/http://foundeo.com/ - ColdFusion Consulting &
Products http://hackmycf.com - CFML Server Security Scanner
On Wed, Nov 4, 2015 at 12:48 PM, Michael Wood <@Michael_Wood> wrote:
Hi Pete!
Oh I wish I could but we do processing on the images (re-size, create
thumbnails, etc) in EC2 before storing them in S3. I can’t think of another
way to pre-process the images before storing them in S3, can you?
This is something new we’re trying. We have the upload pages working in an
iFrame so that the upload and image processing is offloaded entirely to the
client and to EC2/S3. They will communicate directly freeing this load from
our main server. Running Lucee on Amazon Linux in EC2 and using S3 for
storage made this idea possible/practical/affordable…
I *wish *we could do this in Lambda but I’m pretty sure the programming
required is too far out of my skill set.
Mike
On Wednesday, November 4, 2015 at 6:56:44 AM UTC-8, Pete Freitag wrote:
You can also upload the file directly to S3 from the client’s browser,
here’s how you can do it in CFML: Upload Files Directly to Amazon S3 using ColdFusion this approach is nice because
it does not utilize your server resources for the upload, but may be a bit
more work to get going.
Guys, is there a disadvantage to doing this the way I’m doing it; that is
uploading to an EC2 instance, processing the images, and then moving them
into S3?
Or saying it the other way around, are there potential compelling
advantages to uploading straight to S3 and then post-processing the images?
Our pre-processing in EC2 includes image/file type validation, security
checks, image conversion to jpg (as a standard), image re-sizing, creation
of a thumbnail, AND we do a web service call to get an id to name the file
which also determines a “folder name” to use in the object name for S3. It
seems like we will have a lot more control this way. And amazingly all of
this happens pretty quickly. Of course I haven’t had an opportunity to do
load testing yet. Uploading directly to S3 might be more robust.
The other advantage to pre-processing is that the end user immediately SEES
the uploaded/processed image and thumbnail (OR the failure to
upload/process and can try again).
Mike WoodOn Wednesday, November 4, 2015 at 10:55:49 AM UTC-8, Pete Freitag wrote:
You could use the AWS Lambda service to process the images directly on AWS
with the Lambda function triggered by the “store” event in S3, it is one of
the examples in the AWS Lambda docs:
Oh I wish I could but we do processing on the images (re-size, create
thumbnails, etc) in EC2 before storing them in S3. I can’t think of another
way to pre-process the images before storing them in S3, can you?
This is something new we’re trying. We have the upload pages working in an
iFrame so that the upload and image processing is offloaded entirely to the
client and to EC2/S3. They will communicate directly freeing this load from
our main server. Running Lucee on Amazon Linux in EC2 and using S3 for
storage made this idea possible/practical/affordable…
I *wish *we could do this in Lambda but I’m pretty sure the programming
required is too far out of my skill set.
Mike
On Wednesday, November 4, 2015 at 6:56:44 AM UTC-8, Pete Freitag wrote:
You can also upload the file directly to S3 from the client’s browser,
here’s how you can do it in CFML: Upload Files Directly to Amazon S3 using ColdFusion this approach is nice because
it does not utilize your server resources for the upload, but may be a bit
more work to get going.
On 3 November 2015 at 19:52, Michael Wood <@Michael_Wood> wrote:
There aren’t actually sub-folders in S3; only objects. But I did try
pre-creating the that folder by manually creating another object with that
folder name. It didn’t fix the issue.
On Tuesday, November 3, 2015 at 11:47:03 AM UTC-8, Sean Daniels wrote:
I’m pretty sure you need to explicitly create the subfolder if it does
not yet exist.
The second one returns error:
“can’t write file
[s3://[secretId]:[secretKey]@/[bucketName]/000/1062.jpg] as a file,
missing parent directory
[s3://[secretId]:[secretKey]@/[bucketName]/000]”
There seems to be a spurious “/” inserted after the @ symbol. I didn’t
put it there…
Mike
–
Love Lucee? Become a supporter and be part of the Lucee project today!
Oh I wish I could but we do processing on the images (re-size, create
thumbnails, etc) in EC2 before storing them in S3. I can’t think of another
way to pre-process the images before storing them in S3, can you?
This is something new we’re trying. We have the upload pages working in an
iFrame so that the upload and image processing is offloaded entirely to the
client and to EC2/S3. They will communicate directly freeing this load from
our main server. Running Lucee on Amazon Linux in EC2 and using S3 for
storage made this idea possible/practical/affordable…
I *wish *we could do this in Lambda but I’m pretty sure the programming
required is too far out of my skill set.
MikeOn Wednesday, November 4, 2015 at 6:56:44 AM UTC-8, Pete Freitag wrote:
You can also upload the file directly to S3 from the client’s browser,
here’s how you can do it in CFML: Upload Files Directly to Amazon S3 using ColdFusion
this approach is nice because it does not utilize your server resources
for the upload, but may be a bit more work to get going.
@Michael Wood: Is a intermediate upload bucket an option? After the upload
is done, you can get the image from S3, process the image and place it in
the final bucket?2015-11-04 18:48 GMT+01:00 Michael Wood <@Michael_Wood>:
Hi Pete!
Oh I wish I could but we do processing on the images (re-size, create
thumbnails, etc) in EC2 before storing them in S3. I can’t think of another
way to pre-process the images before storing them in S3, can you?
This is something new we’re trying. We have the upload pages working in an
iFrame so that the upload and image processing is offloaded entirely to the
client and to EC2/S3. They will communicate directly freeing this load from
our main server. Running Lucee on Amazon Linux in EC2 and using S3 for
storage made this idea possible/practical/affordable…
I *wish *we could do this in Lambda but I’m pretty sure the programming
required is too far out of my skill set.
Mike
On Wednesday, November 4, 2015 at 6:56:44 AM UTC-8, Pete Freitag wrote:
You can also upload the file directly to S3 from the client’s browser,
here’s how you can do it in CFML: Upload Files Directly to Amazon S3 using ColdFusion this approach is nice because
it does not utilize your server resources for the upload, but may be a bit
more work to get going.