Upload large files without timeout

I maintain a platform where people can upload and download files (between accountants and clients). It all works nice, but sometimes a user uploads a file that is not uploaded before the timeout (set to 60 seconds). I want to fix this, but not sure about the way to go. A possiblity is to make the timeout larger, but I already know we will hit that timeout in the future as well.

So I’m thinking about something that uploads asynchronous, but not sure if that route is full of problems as well (threads that have the potential to run for a long period of time, slowing down the server), the problem with how you tell users that they don’t see their file, but it is being uploaded etc.

So I just want to pick your brains. How would you go about working this pragmatically, but also with a solid solution in the end. Curious what you think!

Marcel

You might want to check this article our by @bennadel

It is 10 years old, but I have used this technique in the past with great success. Plupload has not been updated in a while, but still works well last time I looked.

The advantage of this technique is that you will not have long running threads and timeout issues. It breaks the file apart on the client, uploads it to the server in chunks, where it then gets put back together into the complete file. You also get a nice uploaded status as a result that you can build a progress bar from.

3 Likes

Another solid option is to have the upload go directly from the browser to an S3 bucket. This way, there’s basically no burden on your server during the upload process. Of course, then you have to get the file off S3; but, there are a variety of ways to do that if you go down that route.

1 Like

You are probably referring to this article: Chunking Amazon S3 File Uploads With Plupload And ColdFusion. Interesting read. You have given me a really good starting point here. I am wondering what happens during the upload from the user perspective. It seems the user stays on the page where the upload form was. But I’ll figure that out once I start implementing. Long story short: THANK YOU!

1 Like

There are also plenty of frontends that you can use other than plupload…
like Dropzone.js or
FileStack or
flowjs/flow.js or
FileDrop.js

or roll your own pure javascript :slight_smile:
How To Build A JavaScript File Uploader (filestack.com)

Using JavaScript FileReader to Upload Large Files in Chunks and Avoid Server Limits (deliciousbrains.com)

How to build a file upload service with vanilla JavaScript - LogRocket Blog
… so many more…

1 Like