S3 Directory Listing Gives "... exists, but isn't a directory" Error in 4.5.1.000

dev google-group-archive
1

This *seems *like a Lucee bug, but I wanted to run it by the list first.

I’m having trouble listing certain directories using S3 URIs. I haven’t
figured out a pattern, and I can’t find anything on the S3 side that looks
like a problem. In fact, using the same credentials, I have no problem
accessing anything, whatsoever, from another client (s3cmd).

This seems to happen with S3 mappings configured in the admin, as well as
with S3 URIs used directly.

Here is a test: Lucee 4.5.1.000 Problem with S3 Directory Listing: "exists, but isn't a directory" · GitHub

It has:

  • List bucket (successful)
  • List /reports directory (successful)
  • Read /a/foo.txt (successful)
  • List /a directory (failure)

I’ve seen other S3-related posts lately, and I realize that some have
abandoned Lucee S3 functionality due to reliability problems, but it would
be nice to simply have robust native support, so it’s something worth
pursuing.

Thoughts?

Thanks,
Jamie

2

El Bumpito.On Friday, May 15, 2015 at 12:09:32 PM UTC-4, Jamie Jackson wrote:

This *seems *like a Lucee bug, but I wanted to run it by the list first.

I’m having trouble listing certain directories using S3 URIs. I haven’t
figured out a pattern, and I can’t find anything on the S3 side that looks
like a problem. In fact, using the same credentials, I have no problem
accessing anything, whatsoever, from another client (s3cmd).

This seems to happen with S3 mappings configured in the admin, as well as
with S3 URIs used directly.

Here is a test: Lucee 4.5.1.000 Problem with S3 Directory Listing: "exists, but isn't a directory" · GitHub

It has:

  • List bucket (successful)
  • List /reports directory (successful)
  • Read /a/foo.txt (successful)
  • List /a directory (failure)

I’ve seen other S3-related posts lately, and I realize that some have
abandoned Lucee S3 functionality due to reliability problems, but it would
be nice to simply have robust native support, so it’s something worth
pursuing.

Thoughts?

Thanks,
Jamie

3

Hmm. There is something different about working vs. not working
directories, as viewed from the S3 console:

tchiverton@ev34:~$ aws s3 ls s3://xxx-files/extravisionbeta2/
PRE _archivedMessageContent/
PRE docs/
PRE images/
tchiverton@ev34:~$ aws s3 ls s3://xxx-files/extravision/
PRE _archivedMessageContent/
PRE _editor/
PRE docs/
PRE images/
PRE static/
2015-05-15 11:10:59 1

I’m not sure what that 1 (byte ?) file is. But we didn’t mean to put it
there… maybe this is the symptom of the Lucee bug ?

Jamie, could you check what you get ?

TomOn Friday, May 15, 2015 at 5:09:32 PM UTC+1, Jamie Jackson wrote:

This *seems *like a Lucee bug, but I wanted to run it by the list first.

I’m having trouble listing certain directories using S3 URIs. I haven’t
figured out a pattern, and I can’t find anything on the S3 side that looks
like a problem. In fact, using the same credentials, I have no problem
accessing anything, whatsoever, from another client (s3cmd).

This seems to happen with S3 mappings configured in the admin, as well as
with S3 URIs used directly.

Here is a test: Lucee 4.5.1.000 Problem with S3 Directory Listing: "exists, but isn't a directory" · GitHub

It has:

  • List bucket (successful)
  • List /reports directory (successful)
  • Read /a/foo.txt (successful)
  • List /a directory (failure)

I’ve seen other S3-related posts lately, and I realize that some have
abandoned Lucee S3 functionality due to reliability problems, but it would
be nice to simply have robust native support, so it’s something worth
pursuing.

Thoughts?

Thanks,
Jamie

4

I saw this too recently, using S3 URLs directly. I assumed it was something
crazy in Amazon. Removing and recreating the top-level folder in the bucket
(aws s3 sync is your friend) fixed it.
Curiously, other folders in the same bucket were fine.

TomOn Tuesday, May 19, 2015 at 2:39:30 PM UTC+1, Jamie Jackson wrote:

El Bumpito.

On Friday, May 15, 2015 at 12:09:32 PM UTC-4, Jamie Jackson wrote:

This *seems *like a Lucee bug, but I wanted to run it by the list first.

I’m having trouble listing certain directories using S3 URIs. I haven’t
figured out a pattern, and I can’t find anything on the S3 side that looks
like a problem. In fact, using the same credentials, I have no problem
accessing anything, whatsoever, from another client (s3cmd).

This seems to happen with S3 mappings configured in the admin, as well as
with S3 URIs used directly.

Here is a test: Lucee 4.5.1.000 Problem with S3 Directory Listing: "exists, but isn't a directory" · GitHub

It has:

  • List bucket (successful)
  • List /reports directory (successful)
  • Read /a/foo.txt (successful)
  • List /a directory (failure)

I’ve seen other S3-related posts lately, and I realize that some have
abandoned Lucee S3 functionality due to reliability problems, but it would
be nice to simply have robust native support, so it’s something worth
pursuing.

Thoughts?

Thanks,
Jamie

5

I am defeintly able to restore my ability to list effected top level
buckets by running the following Python code.
It basically finds a one byte object, and removes it, from the indicated
folder:

s3_conn = boto.connect_s3(aws_access_key, aws_secret_key)
bucket = s3_conn.get_bucket(bucket_name)
for key in bucket.list(‘extravision/’):
if key.size == 1:
print(key.key+’ '+str(key.size))
print key.get_metadata(‘Content-Type’)
key.delete()

Tom

6

Hey Tom,

Nice job finding that! Yes, that was what was causing my problem, too.

After a little investigation, it looks like that 0-byte object is a
“directory” object. One way it comes into existence is if you use the AWS
S3 web interface to first create the directory (as you might do before
uploading files into the directory).

If you create an object directly by doing something like:

aws s3 cp test.txt s3://
my.bucket.com/directory-that-doesnt-exist-yet/test.txt

… then one of those “directory” objects is not generated.

Aside from your solution, an easy way to delete a single one of those
0-byte directory objects is simply to delete that “directory” object (in
this case, “foo”):

aws s3 rm s3://my.bucket.com/foo/

… and the “contained” files remain intact.

With all that said, other clients don’t seem to have much trouble with
“directories” containing these “directory” objects, so Lucee should be able
to handle the situation as well, so I’ll file a ticket. (Though now that I
know what we’re dealing with, I finally understand why *s3cmd *throws a
warning about an empty object during some operations.)

Thanks,
JamieOn Wed, May 20, 2015 at 5:41 AM, Tom Chiverton <@Tom_Chiverton> wrote:

I am defeintly able to restore my ability to list effected top level
buckets by running the following Python code.
It basically finds a one byte object, and removes it, from the indicated
folder:

s3_conn = boto.connect_s3(aws_access_key, aws_secret_key)
bucket = s3_conn.get_bucket(bucket_name)
for key in bucket.list(‘extravision/’):
if key.size == 1:
print(key.key+’ '+str(key.size))
print key.get_metadata(‘Content-Type’)
key.delete()

Tom

–
You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/90bd85a0-9bf9-4f22-8a99-4f30c536c00f%40googlegroups.com
https://groups.google.com/d/msgid/lucee/90bd85a0-9bf9-4f22-8a99-4f30c536c00f%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

7

I meant 1-byte files, not 0-byte files, BTW.On Wed, May 20, 2015 at 10:06 AM, Jamie Jackson <@Jamie_Jackson> wrote:

Hey Tom,

Nice job finding that! Yes, that was what was causing my problem, too.

After a little investigation, it looks like that 0-byte object is a
“directory” object. One way it comes into existence is if you use the AWS
S3 web interface to first create the directory (as you might do before
uploading files into the directory).

If you create an object directly by doing something like:

aws s3 cp test.txt s3://
my.bucket.com/directory-that-doesnt-exist-yet/test.txt

… then one of those “directory” objects is not generated.

Aside from your solution, an easy way to delete a single one of those
0-byte directory objects is simply to delete that “directory” object (in
this case, “foo”):

aws s3 rm s3://my.bucket.com/foo/

… and the “contained” files remain intact.

With all that said, other clients don’t seem to have much trouble with
“directories” containing these “directory” objects, so Lucee should be able
to handle the situation as well, so I’ll file a ticket. (Though now that I
know what we’re dealing with, I finally understand why *s3cmd *throws a
warning about an empty object during some operations.)

Thanks,
Jamie

On Wed, May 20, 2015 at 5:41 AM, Tom Chiverton <@Tom_Chiverton> wrote:

I am defeintly able to restore my ability to list effected top level
buckets by running the following Python code.
It basically finds a one byte object, and removes it, from the indicated
folder:

s3_conn = boto.connect_s3(aws_access_key, aws_secret_key)
bucket = s3_conn.get_bucket(bucket_name)
for key in bucket.list(‘extravision/’):
if key.size == 1:
print(key.key+’ '+str(key.size))
print key.get_metadata(‘Content-Type’)
key.delete()

Tom

–
You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/90bd85a0-9bf9-4f22-8a99-4f30c536c00f%40googlegroups.com
https://groups.google.com/d/msgid/lucee/90bd85a0-9bf9-4f22-8a99-4f30c536c00f%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

8

This happened again today, with a different top level folder.

I restarted Lucee and the error does not clear.

Our code looks like:

if( ! directoryExists(base) ){
writeLog(‘client dir missing for #base#’,‘debug’);
directoryCreate(base);
writelog('FileBrowser created client dir for
'&clientFolder);
}

and the directoryCreate() throws ‘it exist a file with same name’.

For now we’re going to try catch’ing and swallowing this error…

TomOn Wednesday, May 20, 2015 at 8:56:16 AM UTC+1, Tom Chiverton wrote:

I saw this too recently, using S3 URLs directly. I assumed it was
something crazy in Amazon. Removing and recreating the top-level folder in
the bucket (aws s3 sync is your friend) fixed it.
Curiously, other folders in the same bucket were fine.

Tom

On Tuesday, May 19, 2015 at 2:39:30 PM UTC+1, Jamie Jackson wrote:

El Bumpito.

On Friday, May 15, 2015 at 12:09:32 PM UTC-4, Jamie Jackson wrote:

This *seems *like a Lucee bug, but I wanted to run it by the list first.

I’m having trouble listing certain directories using S3 URIs. I haven’t
figured out a pattern, and I can’t find anything on the S3 side that looks
like a problem. In fact, using the same credentials, I have no problem
accessing anything, whatsoever, from another client (s3cmd).

This seems to happen with S3 mappings configured in the admin, as well
as with S3 URIs used directly.

Here is a test:
Lucee 4.5.1.000 Problem with S3 Directory Listing: "exists, but isn't a directory" · GitHub

It has:

  • List bucket (successful)
  • List /reports directory (successful)
  • Read /a/foo.txt (successful)
  • List /a directory (failure)

I’ve seen other S3-related posts lately, and I realize that some have
abandoned Lucee S3 functionality due to reliability problems, but it would
be nice to simply have robust native support, so it’s something worth
pursuing.

Thoughts?

Thanks,
Jamie

9

Here’s the Lucee ticket: [LDEV-359] - Lucee Wed, May 20, 2015 at 10:06 AM, Jamie Jackson <@Jamie_Jackson> wrote:

Hey Tom,

Nice job finding that! Yes, that was what was causing my problem, too.

After a little investigation, it looks like that 0-byte object is a
“directory” object. One way it comes into existence is if you use the AWS
S3 web interface to first create the directory (as you might do before
uploading files into the directory).

If you create an object directly by doing something like:

aws s3 cp test.txt s3://
my.bucket.com/directory-that-doesnt-exist-yet/test.txt

… then one of those “directory” objects is not generated.

Aside from your solution, an easy way to delete a single one of those
0-byte directory objects is simply to delete that “directory” object (in
this case, “foo”):

aws s3 rm s3://my.bucket.com/foo/

… and the “contained” files remain intact.

With all that said, other clients don’t seem to have much trouble with
“directories” containing these “directory” objects, so Lucee should be able
to handle the situation as well, so I’ll file a ticket. (Though now that I
know what we’re dealing with, I finally understand why *s3cmd *throws a
warning about an empty object during some operations.)

Thanks,
Jamie

On Wed, May 20, 2015 at 5:41 AM, Tom Chiverton <@Tom_Chiverton> wrote:

I am defeintly able to restore my ability to list effected top level
buckets by running the following Python code.
It basically finds a one byte object, and removes it, from the indicated
folder:

s3_conn = boto.connect_s3(aws_access_key, aws_secret_key)
bucket = s3_conn.get_bucket(bucket_name)
for key in bucket.list(‘extravision/’):
if key.size == 1:
print(key.key+’ '+str(key.size))
print key.get_metadata(‘Content-Type’)
key.delete()

Tom

–
You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/90bd85a0-9bf9-4f22-8a99-4f30c536c00f%40googlegroups.com
https://groups.google.com/d/msgid/lucee/90bd85a0-9bf9-4f22-8a99-4f30c536c00f%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

10

If we swallow the createDirectory error, subsequent directoryList() fail
with ‘isn’t a directory’.

I think directoryExists() is broken ? And maybe then directoryCreate() does
something funny with the S3 bucket, which leaves it permanently broken ?On Wednesday, May 20, 2015 at 9:52:08 AM UTC+1, Tom Chiverton wrote:

This happened again today, with a different top level folder.

I restarted Lucee and the error does not clear.

Our code looks like:

if( ! directoryExists(base) ){
writeLog(‘client dir missing for #base#’,‘debug’);
directoryCreate(base);
writelog('FileBrowser created client dir for
'&clientFolder);
}

and the directoryCreate() throws ‘it exist a file with same name’.

For now we’re going to try catch’ing and swallowing this error…

Tom

On Wednesday, May 20, 2015 at 8:56:16 AM UTC+1, Tom Chiverton wrote:

I saw this too recently, using S3 URLs directly. I assumed it was
something crazy in Amazon. Removing and recreating the top-level folder in
the bucket (aws s3 sync is your friend) fixed it.
Curiously, other folders in the same bucket were fine.

Tom

On Tuesday, May 19, 2015 at 2:39:30 PM UTC+1, Jamie Jackson wrote:

El Bumpito.

On Friday, May 15, 2015 at 12:09:32 PM UTC-4, Jamie Jackson wrote:

This *seems *like a Lucee bug, but I wanted to run it by the list
first.

I’m having trouble listing certain directories using S3 URIs. I haven’t
figured out a pattern, and I can’t find anything on the S3 side that looks
like a problem. In fact, using the same credentials, I have no problem
accessing anything, whatsoever, from another client (s3cmd).

This seems to happen with S3 mappings configured in the admin, as well
as with S3 URIs used directly.

Here is a test:
Lucee 4.5.1.000 Problem with S3 Directory Listing: "exists, but isn't a directory" · GitHub

It has:

  • List bucket (successful)
  • List /reports directory (successful)
  • Read /a/foo.txt (successful)
  • List /a directory (failure)

I’ve seen other S3-related posts lately, and I realize that some have
abandoned Lucee S3 functionality due to reliability problems, but it would
be nice to simply have robust native support, so it’s something worth
pursuing.

Thoughts?

Thanks,
Jamie