S3 access keys in the path

We have been using Lucee’s built in support for S3 for some time now, which works very well. One issue we’ve recently come up against is funny characters in a file name. Namely, the “@” sign.

Previously, we configured our application like this:

application.s3 = {};
application.s3.accessKeyId = "xxx";
application.s3.awsSecretKey = "yyy";
application.basePath = "S3:///my-bucket";

However, if we then use:

FileExists(application.basePath & "/some@file_with_dodgy_characters%!_123.txt");

We get the error: The AWS Access Key Id you provided does not exist in our records.

The error does NOT happen if we set application.basePath to be “S3://xxx:yyy@my-bucket”. That works fine and is what I’ve changed it to.

The reason I am posting is that I was wondering if anyone else has come across this. Is there a work-around or is this an issue that maybe needs to be addressed? I liked not having to put the keys in the path, using the application.s3 struct. It kept them a little more secure. If there was some code that dumped the full path on an error message for some reason, such as "file S3://xx:yyy@my-bucket/whatever does not exist) it would expose our s3 access keys.

Update: I guess the other thing I’d like to know, are there other characters that will cause this issue. I have done some testing, and as far as I can tell @ is the only one that causes this problem.

Having filenames with @ symbols in them will (and does) cause issues over HTTP in general (even if say, you were authenticating with an FTP site over HTTP to fetch a file with an @ in it, you’d also get an error).

ACF has the same condition, but it is not something I would consider a bug, per se. I’m frankly surprised it works when you change the basePath to use the id:key@bucket format with Lucee - I suspect you might run into the issue again with other file/directory functions even with that format when dealing with files with @ symbols in them, given their usually https calls to the S3 API under the hood, but I could be wrong.

The issue here is one of standards. Essentially, this is Basic Auth. user:pass@somedomain has been around since the beginning of the internet. This is generally parsed by separating the last found @ symbol, then using the left hand side of the : as the username and the right hand side as the password.

So, for instance: ftp://user:pass@somedomain.com/this@file.zip would take user as the username, but then pass@somedomain.com/this as the password, and file.zip as the file.

The reason the last found @ symbol is chosen as the delimiter is to allow for an @ symbol in the password itself.

The workaround is pretty simple… just remove the @ from filenames for files you’ll be storing or accessing over HTTP (incl. S3).


– Denny

1 Like

Hi Denny,

Yes, my guess is that Lucee is checking for an @ symbol, and then assuming the credentials are being passed in the S3 path rather than using the stored credentials. You are right, avoiding the @ is the only answer at the moment. Our application offers a direct upload, so I have to get around it server-side by first doing a check on the filename. And in my testing @ seems to be the only standard character that causes trouble. Even colon : is allowed somehow.

I would say that perhaps better would be for Lucee to check for the 3 slash pattern s3:///mybucket/path when deciding whether to use the stored credentials, since I’m pretty sure this is what the docs say it does.