Hi,
We have been using Lucee’s built in support for S3 for some time now, which works very well. One issue we’ve recently come up against is funny characters in a file name. Namely, the “@” sign.
Previously, we configured our application like this:
application.s3 = {};
application.s3.accessKeyId = "xxx";
application.s3.awsSecretKey = "yyy";
application.basePath = "S3:///my-bucket";
However, if we then use:
FileExists(application.basePath & "/some@file_with_dodgy_characters%!_123.txt");
We get the error: The AWS Access Key Id you provided does not exist in our records.
The error does NOT happen if we set application.basePath to be “S3://xxx:yyy@my-bucket”. That works fine and is what I’ve changed it to.
The reason I am posting is that I was wondering if anyone else has come across this. Is there a work-around or is this an issue that maybe needs to be addressed? I liked not having to put the keys in the path, using the application.s3 struct. It kept them a little more secure. If there was some code that dumped the full path on an error message for some reason, such as "file S3://xx:yyy@my-bucket/whatever does not exist) it would expose our s3 access keys.
Update: I guess the other thing I’d like to know, are there other characters that will cause this issue. I have done some testing, and as far as I can tell @ is the only one that causes this problem.
Having filenames with @ symbols in them will (and does) cause issues over HTTP in general (even if say, you were authenticating with an FTP site over HTTP to fetch a file with an @ in it, you’d also get an error).
ACF has the same condition, but it is not something I would consider a bug, per se. I’m frankly surprised it works when you change the basePath to use the id:key@bucket format with Lucee - I suspect you might run into the issue again with other file/directory functions even with that format when dealing with files with @ symbols in them, given their usually https calls to the S3 API under the hood, but I could be wrong.
The issue here is one of standards. Essentially, this is Basic Auth. user:pass@somedomain has been around since the beginning of the internet. This is generally parsed by separating the last found @ symbol, then using the left hand side of the : as the username and the right hand side as the password.
So, for instance: ftp://user:pass@somedomain.com/this@file.zip would take user as the username, but then pass@somedomain.com/this as the password, and file.zip as the file.
The reason the last found @ symbol is chosen as the delimiter is to allow for an @ symbol in the password itself.
The workaround is pretty simple… just remove the @ from filenames for files you’ll be storing or accessing over HTTP (incl. S3).
HTH
– Denny
1 Like
Hi Denny,
Yes, my guess is that Lucee is checking for an @ symbol, and then assuming the credentials are being passed in the S3 path rather than using the stored credentials. You are right, avoiding the @ is the only answer at the moment. Our application offers a direct upload, so I have to get around it server-side by first doing a check on the filename. And in my testing @ seems to be the only standard character that causes trouble. Even colon : is allowed somehow.
I would say that perhaps better would be for Lucee to check for the 3 slash pattern s3:///mybucket/path when deciding whether to use the stored credentials, since I’m pretty sure this is what the docs say it does.