File Upload Questions (Related to Larger Uploads Crashing Lucee)

dev google-group-archive
1

Hi Folks,

I’m having some problems where larger files are crashing my Lucee
(4.5.1.023) server.

First question:

If I give fileUpload(All) a file in the 0-50MB range, it flushes all at
once to the temp directory (full-sized file appears); however, if I give it
a file greater than about 55MB, it flushes to the temp directory
incrementally (file grows until it’s full-sized).

Can anyone explain this threshold/behavior?

I looked at the guts of fileUploadAll()
https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/runtime/tag/FileTag.java?at=master&fileviewer=file-view-default#FileTag.java-813,
but I didn’t spot anything.

Thanks,
Jamie

2

After doing some step-debugging in the Lucee core, I see that (on Windows,
at least), there is no difference in behavior between Lucee (via,
FileOutputStream)
https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/commons/io/IOUtil.java?at=master&fileviewer=file-view-default#IOUtil.java-278 writing
a large vs. small file. Both seem to write incrementally on each iteration
of out.write().

So, I’m not sure why I saw the behavior I’d previously seen/described (on
Linux), but I wasn’t as scientific with the previous experiment (since I
can’t easily step debug on the server), and have to watch files/directories
change full-speed.

Back to the drawing board, I guess.

In the meantime, please let me know if you’ve had any experience with
large(r) files crashing your Railo/Lucee server.

Thanks,
JamieOn Wednesday, September 23, 2015 at 10:20:02 AM UTC-4, Jamie Jackson wrote:

Hi Folks,

I’m having some problems where larger files are crashing my Lucee
(4.5.1.023) server.

First question:

If I give fileUpload(All) a file in the 0-50MB range, it flushes all at
once to the temp directory (full-sized file appears); however, if I give it
a file greater than about 55MB, it flushes to the temp directory
incrementally (file grows until it’s full-sized).

Can anyone explain this threshold/behavior?

I looked at the guts of fileUploadAll()
https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/runtime/tag/FileTag.java?at=master&fileviewer=file-view-default#FileTag.java-813,
but I didn’t spot anything.

Thanks,
Jamie

3

I have had serious issues with large file uploads crashing servers in the past, so much so that in a new project I wrote an AJAX-based file uploader that splits uploads into 1MB increments, uploads them separately and then re-assembles the files on the other end. If you are interested in my solution, I am going to put it into an open source project soon. I need to add broken upload resume and some other features before I open source it.

There is at least one other solution like mine, PLUpload, though I think you need to buy a commercial license for production. Ben Nadel has written about it in the past.

Robert> On Sep 24, 2015, at 10:57 AM, Jamie Jackson <@Jamie_Jackson> wrote:

After doing some step-debugging in the Lucee core, I see that (on Windows, at least), there is no difference in behavior between Lucee (via, FileOutputStream) https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/commons/io/IOUtil.java?at=master&fileviewer=file-view-default#IOUtil.java-278 writing a large vs. small file. Both seem to write incrementally on each iteration of out.write().

So, I’m not sure why I saw the behavior I’d previously seen/described (on Linux), but I wasn’t as scientific with the previous experiment (since I can’t easily step debug on the server), and have to watch files/directories change full-speed.

Back to the drawing board, I guess.

In the meantime, please let me know if you’ve had any experience with large(r) files crashing your Railo/Lucee server.

Thanks,
Jamie

On Wednesday, September 23, 2015 at 10:20:02 AM UTC-4, Jamie Jackson wrote:
Hi Folks,

I’m having some problems where larger files are crashing my Lucee (4.5.1.023) server.

First question:

If I give fileUpload(All) a file in the 0-50MB range, it flushes all at once to the temp directory (full-sized file appears); however, if I give it a file greater than about 55MB, it flushes to the temp directory incrementally (file grows until it’s full-sized).

Can anyone explain this threshold/behavior?

I looked at the guts of fileUploadAll() https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/runtime/tag/FileTag.java?at=master&fileviewer=file-view-default#FileTag.java-813, but I didn’t spot anything.

Thanks,
Jamie

–
See Lucee at CFCamp Oct 22 & 23 2015 @ Munich Airport, Germany - Get your ticket NOW - http://www.cfcamp.org/ http://www.cfcamp.org/

You received this message because you are subscribed to the Google Groups “Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+unsubscribe@googlegroups.com mailto:lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com mailto:lucee@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/3f2cae40-d105-4a68-a2c3-9bfb00a0e340%40googlegroups.com https://groups.google.com/d/msgid/lucee/3f2cae40-d105-4a68-a2c3-9bfb00a0e340%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

4

Can you raise a bug for this on JIRA please:

https://luceeserver.atlassian.net/secure/Dashboard.jspa

Thanks.

Kind regards,

Andrew
about.me http://about.me/andrew_dixon - mso http://www.mso.net - Lucee
Association Member http://lucee.orgOn 24 September 2015 at 18:57, Jamie Jackson <@Jamie_Jackson> wrote:

After doing some step-debugging in the Lucee core, I see that (on Windows,
at least), there is no difference in behavior between Lucee (via,
FileOutputStream)
https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/commons/io/IOUtil.java?at=master&fileviewer=file-view-default#IOUtil.java-278 writing
a large vs. small file. Both seem to write incrementally on each iteration
of out.write().

So, I’m not sure why I saw the behavior I’d previously seen/described (on
Linux), but I wasn’t as scientific with the previous experiment (since I
can’t easily step debug on the server), and have to watch files/directories
change full-speed.

Back to the drawing board, I guess.

In the meantime, please let me know if you’ve had any experience with
large(r) files crashing your Railo/Lucee server.

Thanks,
Jamie

On Wednesday, September 23, 2015 at 10:20:02 AM UTC-4, Jamie Jackson wrote:

Hi Folks,

I’m having some problems where larger files are crashing my Lucee
(4.5.1.023) server.

First question:

If I give fileUpload(All) a file in the 0-50MB range, it flushes all at
once to the temp directory (full-sized file appears); however, if I give it
a file greater than about 55MB, it flushes to the temp directory
incrementally (file grows until it’s full-sized).

Can anyone explain this threshold/behavior?

I looked at the guts of fileUploadAll()
https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/runtime/tag/FileTag.java?at=master&fileviewer=file-view-default#FileTag.java-813,
but I didn’t spot anything.

Thanks,
Jamie

–
See Lucee at CFCamp Oct 22 & 23 2015 @ Munich Airport, Germany - Get your
ticket NOW - http://www.cfcamp.org/

You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/3f2cae40-d105-4a68-a2c3-9bfb00a0e340%40googlegroups.com
https://groups.google.com/d/msgid/lucee/3f2cae40-d105-4a68-a2c3-9bfb00a0e340%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

5

I have also noticed this in the passed (on railo), firstly you need to max
your jvm memory I have mine set at 2GB but still never managed files larger
than ~50mb, it causes outofmemoryexeption’s so it seems the limitation is
between java and lucee since it tries to loas the entire file in memory.

This should be done using a filestream instead but I have never dug into it.

Regards

GXOn Wednesday, September 23, 2015 at 5:20:02 PM UTC+3, Jamie Jackson wrote:

Hi Folks,

I’m having some problems where larger files are crashing my Lucee
(4.5.1.023) server.

First question:

If I give fileUpload(All) a file in the 0-50MB range, it flushes all at
once to the temp directory (full-sized file appears); however, if I give it
a file greater than about 55MB, it flushes to the temp directory
incrementally (file grows until it’s full-sized).

Can anyone explain this threshold/behavior?

I looked at the guts of fileUploadAll()
https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/runtime/tag/FileTag.java?at=master&fileviewer=file-view-default#FileTag.java-813,
but I didn’t spot anything.

Thanks,
Jamie

6

Here’s the ticket: [LDEV-548] - Lucee Saturday, September 26, 2015 at 7:41:00 AM UTC-4, Andrew Dixon wrote:

Can you raise a bug for this on JIRA please:

https://luceeserver.atlassian.net/secure/Dashboard.jspa

Thanks.

Kind regards,

Andrew
about.me http://about.me/andrew_dixon - mso http://www.mso.net - Lucee
Association Member http://lucee.org

On 24 September 2015 at 18:57, Jamie Jackson <jamie...@gmail.com <javascript:>> wrote:

After doing some step-debugging in the Lucee core, I see that (on
Windows, at least), there is no difference in behavior between Lucee
(via, FileOutputStream)
https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/commons/io/IOUtil.java?at=master&fileviewer=file-view-default#IOUtil.java-278 writing
a large vs. small file. Both seem to write incrementally on each iteration
of out.write().

So, I’m not sure why I saw the behavior I’d previously seen/described (on
Linux), but I wasn’t as scientific with the previous experiment (since I
can’t easily step debug on the server), and have to watch files/directories
change full-speed.

Back to the drawing board, I guess.

In the meantime, please let me know if you’ve had any experience with
large(r) files crashing your Railo/Lucee server.

Thanks,
Jamie

On Wednesday, September 23, 2015 at 10:20:02 AM UTC-4, Jamie Jackson wrote:

Hi Folks,

I’m having some problems where larger files are crashing my Lucee
(4.5.1.023) server.

First question:

If I give fileUpload(All) a file in the 0-50MB range, it flushes all at
once to the temp directory (full-sized file appears); however, if I give it
a file greater than about 55MB, it flushes to the temp directory
incrementally (file grows until it’s full-sized).

Can anyone explain this threshold/behavior?

I looked at the guts of fileUploadAll()
https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/runtime/tag/FileTag.java?at=master&fileviewer=file-view-default#FileTag.java-813,
but I didn’t spot anything.

Thanks,
Jamie

–
See Lucee at CFCamp Oct 22 & 23 2015 @ Munich Airport, Germany - Get your
ticket NOW - http://www.cfcamp.org/

You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+un...@googlegroups.com <javascript:>.
To post to this group, send email to lu...@googlegroups.com <javascript:>
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/3f2cae40-d105-4a68-a2c3-9bfb00a0e340%40googlegroups.com
https://groups.google.com/d/msgid/lucee/3f2cae40-d105-4a68-a2c3-9bfb00a0e340%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

7

Thanks, Robert. Were you ever able to track down what the ultimate problem
was? I started a ticket, but it isn’t a very clean bug
report: [LDEV-548] - Lucee

I’ll keep an eye on your uploader if nothing comes of the ticket.On Friday, September 25, 2015 at 7:09:30 PM UTC-4, Robert Munn wrote:

I have had serious issues with large file uploads crashing servers in the
past, so much so that in a new project I wrote an AJAX-based file uploader
that splits uploads into 1MB increments, uploads them separately and then
re-assembles the files on the other end. If you are interested in my
solution, I am going to put it into an open source project soon. I need to
add broken upload resume and some other features before I open source it.

There is at least one other solution like mine, PLUpload, though I think
you need to buy a commercial license for production. Ben Nadel has written
about it in the past.

Robert

On Sep 24, 2015, at 10:57 AM, Jamie Jackson <jamie...@gmail.com <javascript:>> wrote:

After doing some step-debugging in the Lucee core, I see that (on Windows,
at least), there is no difference in behavior between Lucee (via,
FileOutputStream)
https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/commons/io/IOUtil.java?at=master&fileviewer=file-view-default#IOUtil.java-278 writing
a large vs. small file. Both seem to write incrementally on each iteration
of out.write().

So, I’m not sure why I saw the behavior I’d previously seen/described (on
Linux), but I wasn’t as scientific with the previous experiment (since I
can’t easily step debug on the server), and have to watch files/directories
change full-speed.

Back to the drawing board, I guess.

In the meantime, please let me know if you’ve had any experience with
large(r) files crashing your Railo/Lucee server.

Thanks,
Jamie

On Wednesday, September 23, 2015 at 10:20:02 AM UTC-4, Jamie Jackson wrote:

Hi Folks,

I’m having some problems where larger files are crashing my Lucee
(4.5.1.023) server.

First question:

If I give fileUpload(All) a file in the 0-50MB range, it flushes all at
once to the temp directory (full-sized file appears); however, if I give it
a file greater than about 55MB, it flushes to the temp directory
incrementally (file grows until it’s full-sized).

Can anyone explain this threshold/behavior?

I looked at the guts of fileUploadAll()
https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/runtime/tag/FileTag.java?at=master&fileviewer=file-view-default#FileTag.java-813,
but I didn’t spot anything.

Thanks,
Jamie

–
See Lucee at CFCamp Oct 22 & 23 2015 @ Munich Airport, Germany - Get your
ticket NOW - http://www.cfcamp.org/

You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+un...@googlegroups.com <javascript:>.
To post to this group, send email to lu...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/3f2cae40-d105-4a68-a2c3-9bfb00a0e340%40googlegroups.com
https://groups.google.com/d/msgid/lucee/3f2cae40-d105-4a68-a2c3-9bfb00a0e340%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

8

Hi Guys,

in the mean while check this out

I will see if we can use this as the basis of the fix

Thanks

AlexOn 29 September 2015 at 14:14, Jamie Jackson <@Jamie_Jackson> wrote:

I’m back to looking at this flushing behavior on Linux. I seem to have
been right, on Linux, files smaller than the magic threshold (~55MB) write
to the temp directory all at once, while larger files flush incrementally,
from the very beginning.

So, to sum up my questions. (I’ve got a new one, too.)

Flushing

  1. Why/how do files get written differently to the temp directory,
    depending on their size, on Linux JVMs? (Incrementally for files greater
    than ~55MB, and all at once for smaller files.)

Transfer Rate

  1. Why is a cURL upload 2.5x faster than a real form upload from a
    browser? Throughput Difference in Form File Upload vs. cURL Upload · GitHub (Using
    Chrome, FWIW.)

Thanks,

Jamie

On Thursday, September 24, 2015 at 1:57:35 PM UTC-4, Jamie Jackson wrote:

After doing some step-debugging in the Lucee core, I see that (on
Windows, at least), there is no difference in behavior between Lucee
(via, FileOutputStream)
https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/commons/io/IOUtil.java?at=master&fileviewer=file-view-default#IOUtil.java-278 writing
a large vs. small file. Both seem to write incrementally on each iteration
of out.write().

So, I’m not sure why I saw the behavior I’d previously seen/described (on
Linux), but I wasn’t as scientific with the previous experiment (since I
can’t easily step debug on the server), and have to watch files/directories
change full-speed.

Back to the drawing board, I guess.

In the meantime, please let me know if you’ve had any experience with
large(r) files crashing your Railo/Lucee server.

Thanks,
Jamie

On Wednesday, September 23, 2015 at 10:20:02 AM UTC-4, Jamie Jackson wrote:

Hi Folks,

I’m having some problems where larger files are crashing my Lucee
(4.5.1.023) server.

First question:

If I give fileUpload(All) a file in the 0-50MB range, it flushes all at
once to the temp directory (full-sized file appears); however, if I give it
a file greater than about 55MB, it flushes to the temp directory
incrementally (file grows until it’s full-sized).

Can anyone explain this threshold/behavior?

I looked at the guts of fileUploadAll()
https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/runtime/tag/FileTag.java?at=master&fileviewer=file-view-default#FileTag.java-813,
but I didn’t spot anything.

Thanks,
Jamie

–
See Lucee at CFCamp Oct 22 & 23 2015 @ Munich Airport, Germany - Get your
ticket NOW - http://www.cfcamp.org/

You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/ffa8600b-a023-4236-abb6-1f84b9e309e0%40googlegroups.com
https://groups.google.com/d/msgid/lucee/ffa8600b-a023-4236-abb6-1f84b9e309e0%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

–
Alex Skinner
Managing Director

Pixl8 Interactive, 3 Tun Yard, Peardon Street, London
SW8 3HT, United Kingdom

T: +44 [0] 845 260 0726• W: www.pixl8.co.uk• E: info@pixl8.co.uk

Follow us on: Facebook http://www.facebook.com/pixl8 Twitter
http://www.twitter.com/pixl8 LinkedIn http://www.linkedin.com/pixl8

CONFIDENTIAL AND PRIVILEGED - This e-mail and any attachment is intended
solely for the addressee, is strictly confidential and may also be subject
to legal, professional or other privilege or may be protected by work
product immunity or other legal rules. If you are not the addressee please
do not read, print, re-transmit, store or act in reliance on it or any
attachments. Instead, please email it back to the sender and then
immediately permanently delete it. Pixl8 Interactive Ltd Registered in
England. Registered number: 04336501. Registered office: 8 Spur Road,
Cosham, Portsmouth, Hampshire, PO6 3EB

9

What is the Web server your running…

I would bet its not a CF issueOn Wednesday, September 23, 2015 at 10:20:02 AM UTC-4, Jamie Jackson wrote:

Hi Folks,

I’m having some problems where larger files are crashing my Lucee
(4.5.1.023) server.

First question:

If I give fileUpload(All) a file in the 0-50MB range, it flushes all at
once to the temp directory (full-sized file appears); however, if I give it
a file greater than about 55MB, it flushes to the temp directory
incrementally (file grows until it’s full-sized).

Can anyone explain this threshold/behavior?

I looked at the guts of fileUploadAll()
https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/runtime/tag/FileTag.java?at=master&fileviewer=file-view-default#FileTag.java-813,
but I didn’t spot anything.

Thanks,
Jamie

10

I’m running Apache HTTPD on the (Linux) servers.

When I was step-debugging locally, I was on Windows, using the debugging
server which runs on Jetty.

Why, what are you thinking?On Tue, Sep 29, 2015 at 11:40 AM, Terry Whitney <@Terry_Whitney> wrote:

What is the Web server your running…

I would bet its not a CF issue

On Wednesday, September 23, 2015 at 10:20:02 AM UTC-4, Jamie Jackson wrote:

Hi Folks,

I’m having some problems where larger files are crashing my Lucee
(4.5.1.023) server.

First question:

If I give fileUpload(All) a file in the 0-50MB range, it flushes all at
once to the temp directory (full-sized file appears); however, if I give it
a file greater than about 55MB, it flushes to the temp directory
incrementally (file grows until it’s full-sized).

Can anyone explain this threshold/behavior?

I looked at the guts of fileUploadAll()
https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/runtime/tag/FileTag.java?at=master&fileviewer=file-view-default#FileTag.java-813,
but I didn’t spot anything.

Thanks,
Jamie

–
See Lucee at CFCamp Oct 22 & 23 2015 @ Munich Airport, Germany - Get your
ticket NOW - http://www.cfcamp.org/

You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/779f3186-2c76-4c03-abe4-b6e6db38b760%40googlegroups.com
https://groups.google.com/d/msgid/lucee/779f3186-2c76-4c03-abe4-b6e6db38b760%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

11

I’m back to looking at this flushing behavior on Linux. I seem to have
been right, on Linux, files smaller than the magic threshold (~55MB) write
to the temp directory all at once, while larger files flush incrementally,
from the very beginning.

So, to sum up my questions. (I’ve got a new one, too.)

Flushing

  1. Why/how do files get written differently to the temp directory,
    depending on their size, on Linux JVMs? (Incrementally for files greater
    than ~55MB, and all at once for smaller files.)

It sounds to me like you might be hitting something in the servlet
container that might be causing this, perhaps the in
web.xml is set, or needs to be set:

MultipartConfig (Servlet 3.0 API Documentation - Apache Tomcat 7.0.109) Tue, Sep 29, 2015 at 9:14 AM, Jamie Jackson <@Jamie_Jackson> wrote:

–
Pete Freitag
https://foundeo.com/ http://foundeo.com/ - ColdFusion Consulting &
Products
http://hackmycf.com - CFML Server Security Scanner

12

I’m back to looking at this flushing behavior on Linux. I seem to have been
right, on Linux, files smaller than the magic threshold (~55MB) write to
the temp directory all at once, while larger files flush incrementally,
from the very beginning.

So, to sum up my questions. (I’ve got a new one, too.)

Flushing

  1. Why/how do files get written differently to the temp directory,
    depending on their size, on Linux JVMs? (Incrementally for files greater
    than ~55MB, and all at once for smaller files.)

Transfer Rate

  1. Why is a cURL upload 2.5x faster than a real form upload from a browser?
    Throughput Difference in Form File Upload vs. cURL Upload · GitHub (Using Chrome,
    FWIW.)

Thanks,

JamieOn Thursday, September 24, 2015 at 1:57:35 PM UTC-4, Jamie Jackson wrote:

After doing some step-debugging in the Lucee core, I see that (on Windows,
at least), there is no difference in behavior between Lucee (via,
FileOutputStream)
https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/commons/io/IOUtil.java?at=master&fileviewer=file-view-default#IOUtil.java-278 writing
a large vs. small file. Both seem to write incrementally on each iteration
of out.write().

So, I’m not sure why I saw the behavior I’d previously seen/described (on
Linux), but I wasn’t as scientific with the previous experiment (since I
can’t easily step debug on the server), and have to watch files/directories
change full-speed.

Back to the drawing board, I guess.

In the meantime, please let me know if you’ve had any experience with
large(r) files crashing your Railo/Lucee server.

Thanks,
Jamie

On Wednesday, September 23, 2015 at 10:20:02 AM UTC-4, Jamie Jackson wrote:

Hi Folks,

I’m having some problems where larger files are crashing my Lucee
(4.5.1.023) server.

First question:

If I give fileUpload(All) a file in the 0-50MB range, it flushes all at
once to the temp directory (full-sized file appears); however, if I give it
a file greater than about 55MB, it flushes to the temp directory
incrementally (file grows until it’s full-sized).

Can anyone explain this threshold/behavior?

I looked at the guts of fileUploadAll()
https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/runtime/tag/FileTag.java?at=master&fileviewer=file-view-default#FileTag.java-813,
but I didn’t spot anything.

Thanks,
Jamie

13

Tomcat has a max upload set of 50 meg by default.On Wednesday, September 23, 2015 at 10:20:02 AM UTC-4, Jamie Jackson wrote:

Hi Folks,

I’m having some problems where larger files are crashing my Lucee
(4.5.1.023) server.

First question:

If I give fileUpload(All) a file in the 0-50MB range, it flushes all at
once to the temp directory (full-sized file appears); however, if I give it
a file greater than about 55MB, it flushes to the temp directory
incrementally (file grows until it’s full-sized).

Can anyone explain this threshold/behavior?

I looked at the guts of fileUploadAll()
https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/runtime/tag/FileTag.java?at=master&fileviewer=file-view-default#FileTag.java-813,
but I didn’t spot anything.

Thanks,
Jamie

14

Hi Pete, according to (my interpretation of) the docs you sent, the default
is unlimited (-1), and since I don’t see any overrides in the stock Lucee
web.xml, I don’t think this is in play.On Tuesday, September 29, 2015 at 1:15:17 PM UTC-4, Pete Freitag wrote:

On Tue, Sep 29, 2015 at 9:14 AM, Jamie Jackson <jamie...@gmail.com <javascript:>> wrote:

I’m back to looking at this flushing behavior on Linux. I seem to have
been right, on Linux, files smaller than the magic threshold (~55MB) write
to the temp directory all at once, while larger files flush incrementally,
from the very beginning.

So, to sum up my questions. (I’ve got a new one, too.)

Flushing

  1. Why/how do files get written differently to the temp directory,
    depending on their size, on Linux JVMs? (Incrementally for files greater
    than ~55MB, and all at once for smaller files.)

It sounds to me like you might be hitting something in the servlet
container that might be causing this, perhaps the in
web.xml is set, or needs to be set:

MultipartConfig (Servlet 3.0 API Documentation - Apache Tomcat 7.0.109)

–
Pete Freitag
https://foundeo.com/ http://foundeo.com/ - ColdFusion Consulting &
Products
http://hackmycf.com - CFML Server Security Scanner

15

That is unlikely to be the correct one: it will control the manager, but
not Lucee. Depending on how you installed Tomcat the correct web.xml is
either in //conf/, /etc/tomcat?/ or
//webapps/ROOT/.

JochemOn Sep 30, 2015 22:05, “Terry Whitney” wrote:

webapps/manager/WEB-INF/web.xml

16

Hi Terry. I don’t see anything specified in web.xml (is that where I should
be looking), and from what I can tell, the default is “-1L”, which is
crappily documented
https://tomcat.apache.org/tomcat-7.0-doc/servletapi/javax/servlet/annotation/MultipartConfig.html#maxFileSize(),
but I assume translates to “unlimited.”

Can you tell me how you know about this default?On Tuesday, September 29, 2015 at 1:43:03 PM UTC-4, Terry Whitney wrote:

Tomcat has a max upload set of 50 meg by default.

On Wednesday, September 23, 2015 at 10:20:02 AM UTC-4, Jamie Jackson wrote:

Hi Folks,

I’m having some problems where larger files are crashing my Lucee
(4.5.1.023) server.

First question:

If I give fileUpload(All) a file in the 0-50MB range, it flushes all at
once to the temp directory (full-sized file appears); however, if I give it
a file greater than about 55MB, it flushes to the temp directory
incrementally (file grows until it’s full-sized).

Can anyone explain this threshold/behavior?

I looked at the guts of fileUploadAll()
https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/runtime/tag/FileTag.java?at=master&fileviewer=file-view-default#FileTag.java-813,
but I didn’t spot anything.

Thanks,
Jamie

17

I have not seen where -1 ever is equal to unlimited in tomcat, but Meh

Make sure you restart the service after you make the change

/etc/init.d/tomcat(versionNumber) restart or on RHel 7 / CentOs 7

systemctl restart tomcat

webapps/manager/WEB-INF/web.xml

I would edit the the web.xml file to look like this below and then restart
the java server (tomcat)

  <max-file-size>252428800</max-file-size>
  <max-request-size>252428800</max-request-size>
  <file-size-threshold>0<</file-size-threshold>
18

Just a random thought,

Are you using the express version with jetty?On Wednesday, September 23, 2015 at 10:20:02 AM UTC-4, Jamie Jackson wrote:

Hi Folks,

I’m having some problems where larger files are crashing my Lucee
(4.5.1.023) server.

First question:

If I give fileUpload(All) a file in the 0-50MB range, it flushes all at
once to the temp directory (full-sized file appears); however, if I give it
a file greater than about 55MB, it flushes to the temp directory
incrementally (file grows until it’s full-sized).

Can anyone explain this threshold/behavior?

I looked at the guts of fileUploadAll()
https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/runtime/tag/FileTag.java?at=master&fileviewer=file-view-default#FileTag.java-813,
but I didn’t spot anything.

Thanks,
Jamie

19

Based on the rest of the discussion, it looks like maybe the Tomcat config issue will fix it, but I never got that far because i decided that i would rather chunk large files into parts. A benefit of this strategy is broken file upload resume so if, for instance, your 700MB ISO upload fails midway through, you can resume without re-uploading the entire file.> On Sep 28, 2015, at 12:40 PM, Jamie Jackson <@Jamie_Jackson> wrote:

Thanks, Robert. Were you ever able to track down what the ultimate problem was? I started a ticket, but it isn’t a very clean bug report: [LDEV-548] - Lucee

I’ll keep an eye on your uploader if nothing comes of the ticket.

On Friday, September 25, 2015 at 7:09:30 PM UTC-4, Robert Munn wrote:
I have had serious issues with large file uploads crashing servers in the past, so much so that in a new project I wrote an AJAX-based file uploader that splits uploads into 1MB increments, uploads them separately and then re-assembles the files on the other end. If you are interested in my solution, I am going to put it into an open source project soon. I need to add broken upload resume and some other features before I open source it.

There is at least one other solution like mine, PLUpload, though I think you need to buy a commercial license for production. Ben Nadel has written about it in the past.

Robert

On Sep 24, 2015, at 10:57 AM, Jamie Jackson <jamie...@gmail.com <javascript:>> wrote:

After doing some step-debugging in the Lucee core, I see that (on Windows, at least), there is no difference in behavior between Lucee (via, FileOutputStream) https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/commons/io/IOUtil.java?at=master&fileviewer=file-view-default#IOUtil.java-278 writing a large vs. small file. Both seem to write incrementally on each iteration of out.write().

So, I’m not sure why I saw the behavior I’d previously seen/described (on Linux), but I wasn’t as scientific with the previous experiment (since I can’t easily step debug on the server), and have to watch files/directories change full-speed.

Back to the drawing board, I guess.

In the meantime, please let me know if you’ve had any experience with large(r) files crashing your Railo/Lucee server.

Thanks,
Jamie

On Wednesday, September 23, 2015 at 10:20:02 AM UTC-4, Jamie Jackson wrote:
Hi Folks,

I’m having some problems where larger files are crashing my Lucee (4.5.1.023) server.

First question:

If I give fileUpload(All) a file in the 0-50MB range, it flushes all at once to the temp directory (full-sized file appears); however, if I give it a file greater than about 55MB, it flushes to the temp directory incrementally (file grows until it’s full-sized).

Can anyone explain this threshold/behavior?

I looked at the guts of fileUploadAll() https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/runtime/tag/FileTag.java?at=master&fileviewer=file-view-default#FileTag.java-813, but I didn’t spot anything.

Thanks,
Jamie

–
See Lucee at CFCamp Oct 22 & 23 2015 @ Munich Airport, Germany - Get your ticket NOW - http://www.cfcamp.org/ http://www.cfcamp.org/

You received this message because you are subscribed to the Google Groups “Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com <javascript:>.
To post to this group, send email to lu...@googlegroups.com <javascript:>.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/3f2cae40-d105-4a68-a2c3-9bfb00a0e340%40googlegroups.com https://groups.google.com/d/msgid/lucee/3f2cae40-d105-4a68-a2c3-9bfb00a0e340%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

–
See Lucee at CFCamp Oct 22 & 23 2015 @ Munich Airport, Germany - Get your ticket NOW - http://www.cfcamp.org/ http://www.cfcamp.org/

You received this message because you are subscribed to the Google Groups “Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+unsubscribe@googlegroups.com mailto:lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com mailto:lucee@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/b6bc302e-4e44-4080-867a-24593c5389e7%40googlegroups.com https://groups.google.com/d/msgid/lucee/b6bc302e-4e44-4080-867a-24593c5389e7%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

20

Contrary to some folks’ misconceptions, Tomcat doesn’t have a default max
file size–it’s unlimited by default. (I confirmed on the Tomcat mailing
list.
http://mail-archives.apache.org/mod_mbox/tomcat-users/201510.mbox/<560FE4E8.8050801%40christopherschultz.net>
)

Also, my questions still stand, but I’m still investigating–I’m not quite
ready to give up on stock Lucee uploads (I really want to find out what’s
going on).

I’ll keep you guys’ uploaders in mind if I can’t get to the bottom of this,
though, so thanks for mentioning them.

Thanks,
JamieOn Tue, Sep 29, 2015 at 9:14 AM, Jamie Jackson <@Jamie_Jackson> wrote:

I’m back to looking at this flushing behavior on Linux. I seem to have
been right, on Linux, files smaller than the magic threshold (~55MB) write
to the temp directory all at once, while larger files flush incrementally,
from the very beginning.

So, to sum up my questions. (I’ve got a new one, too.)

Flushing

  1. Why/how do files get written differently to the temp directory,
    depending on their size, on Linux JVMs? (Incrementally for files greater
    than ~55MB, and all at once for smaller files.)

Transfer Rate

  1. Why is a cURL upload 2.5x faster than a real form upload from a
    browser? Throughput Difference in Form File Upload vs. cURL Upload · GitHub (Using
    Chrome, FWIW.)

Thanks,

Jamie

On Thursday, September 24, 2015 at 1:57:35 PM UTC-4, Jamie Jackson wrote:

After doing some step-debugging in the Lucee core, I see that (on
Windows, at least), there is no difference in behavior between Lucee
(via, FileOutputStream)
https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/commons/io/IOUtil.java?at=master&fileviewer=file-view-default#IOUtil.java-278 writing
a large vs. small file. Both seem to write incrementally on each iteration
of out.write().

So, I’m not sure why I saw the behavior I’d previously seen/described (on
Linux), but I wasn’t as scientific with the previous experiment (since I
can’t easily step debug on the server), and have to watch files/directories
change full-speed.

Back to the drawing board, I guess.

In the meantime, please let me know if you’ve had any experience with
large(r) files crashing your Railo/Lucee server.

Thanks,
Jamie

On Wednesday, September 23, 2015 at 10:20:02 AM UTC-4, Jamie Jackson wrote:

Hi Folks,

I’m having some problems where larger files are crashing my Lucee
(4.5.1.023) server.

First question:

If I give fileUpload(All) a file in the 0-50MB range, it flushes all at
once to the temp directory (full-sized file appears); however, if I give it
a file greater than about 55MB, it flushes to the temp directory
incrementally (file grows until it’s full-sized).

Can anyone explain this threshold/behavior?

I looked at the guts of fileUploadAll()
https://bitbucket.org/lucee/lucee/src/30b941b832b54c5c4c877adc8fb21ca462001b90/lucee-java/lucee-core/src/lucee/runtime/tag/FileTag.java?at=master&fileviewer=file-view-default#FileTag.java-813,
but I didn’t spot anything.

Thanks,
Jamie

–
See Lucee at CFCamp Oct 22 & 23 2015 @ Munich Airport, Germany - Get your
ticket NOW - http://www.cfcamp.org/

You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/ffa8600b-a023-4236-abb6-1f84b9e309e0%40googlegroups.com
https://groups.google.com/d/msgid/lucee/ffa8600b-a023-4236-abb6-1f84b9e309e0%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.