URLDecoder timeout for very large POST requests

wait, so JSON values in form fields are getting automatically parsed by Lucee, but only in new pageContexts for threads?

I don’t think they’re being parsed like “JSON”, I think they’re just being decoded (like form-field encoding).

@bennadel I see some discussion above related to spawning threads, but I thought your original question was simply saying the initial request to Lucee was slow. Is it both?

I have defo complained about the overhead of cloning the full servlet request object and how it gets slower the more stuff is in there. MIcha has been resistant to consider any sort of changes to that process, but I do think it’s a real issue. Can you please enter a ticket for this Ben? Should be easy to whip up a standalone example of submitting a huge JSON form field.

And a final thought-- you mentioned this was faster when you sent a file-- I assume you meant attaching it as a separate MIME content section? What happens if you send the raw JSON in the request body with a JSON content type so it’s not URL encoded form fields?

Would be easy enough to tweak your test case to lead with invalid json characters.

So, my work-use-case deals with JSON. But, the test-case that I created this morning (linked above) just uses plain-text … 1.4MB of it. The slowness seems to happen in both places:

  1. The initial request handling.
  2. Any spawning of async threads within the request (cfthread). This seems to “repay” all of the initial cost of the request-handling.

I haven’t tried posting as a single JSON body. But, that’s a great suggestion. The code I happen to be working in is a few years old, and was built using form-fields; so, I was just jamming off of that. But, I am curious to see if the Body approach would be just as helpful.

Ah ok, so it’s really both places-- the initial request and then all the cloning that happens for every single thread you spawn (I REALLY hate that).

Whether or not changing up the content type helps will probably depend on whether the biggest overhead is

  • Simply dealing with that much data when cloning, etc
  • Or specifically the URL decoding of said data

Sending raw JSOn in the body with a JSON content type wold skip the second bullet, but not the first. I’m not sure right off if the URL Decoding happens when the pc is cloned or just once.

Ok, I just updated my local test and it seems that post in as the Body and then accessing it via getHttpRequestData().content works just as well as the file-based approach :smiley:

The Content-Type didn’t seem to matter (text/plain vs application/json). In both cases, the demo ran super fast!

True, if the URL Decoding is the big hold up, then the content type could be html or json or anything so long as it’s NOT application/x-www-form-urlencoded! That’s the specific request content type that tells the app server it needs to decode it first.

How fast is thread spawning when you have a large request body that’s using a text or JSON type?

Also don’t forget to put in a bug. Just because you find a workaround doesn’t mean Lucee can’t do better! :slight_smile:

So, when posting as application/x-www-form-urlencoded with 1.4MB of text, spawning a cfthread was taking ~ 33-seconds. When I switch to text/plain and post as the body, same exact demo takes ~ 4-ms to spawn threads. :exploding_head:

I’ll write this up, then post a ticket, probably tomorrow.

Wow, it defo seems that the URL Decoding is happening on every clone which seems incredibly unnecessary! Once Lucee has URL decoded a form post, there’s no need to do it again, even if cloning the HTTPServletRequest object.

Thats what the Fusion-Reactor stack-traces seem to indicate.

if you modify the form scope, then spawn a thread, does the thread get the original form scope or the modified one?

It appears to get the original one. Trying to trace through the Lucee code on GitHub, the duplication of the PageContext eventually calls form.getRaw(), which I think is only bound when the FormImpl is initialized.

It’s fun finding a CF bug/behavior and then finding yourself an hour later knee keep in Lucee’s source code tracking down where the code is right? :slight_smile:

Ha ha, all I can say is that I wish I had Fusion-Reactor installed months ago! Just got it yesterday!

1 Like

So, I just wanted to write-up my findings using Joe Gooch’s “file upload” and Brad Wood’s “request body” approach, both of which demonstrate a dramatic improvement over using a form-field:

I’ll write up a Lucee ticket for this as well.

2 Likes

@joe.gooch I just wanted to say that we deployed a File-based upload (instead of a Form-field based upload) yesterday and it has had dramatic improvements in our processing times. Thank you again for your suggestion!

nudge :slight_smile:

1 Like

I created a ticket, but somehow it ended up on the wrong JIRA board (Extension-Hibernate). I think someone created an EHIB ticket right before me, and JIRA used that board as my board? Strange.

https://luceeserver.atlassian.net/browse/EHIB-26