Lucee 5.2.x Java Heap Issues

With the release of 5.2.x there have been several reports of memory issues. In a bid to track down the problem I’ve merged reports into this single thread. If you have additional information please post here.

This issue is also being tracked in JIRA as:
https://luceeserver.atlassian.net/browse/LDEV-1640

We are running Lucee with Coldbox and are having a problem with Heap memory filling up. It appears that threads are being created and not removed. I watch them run on VisualVM and they start in a running state. When the request is complete the thread goes into a Park State and stays there. I have a heap dump and a thread dump right after running box and starting the server and then with Postman I issue a Login endpoint and issue a view Customer endpoint 10 times in a row.

It appears to create 1 thread for each request and that thread shows running and then Parked. It will stay in the Parked state for ever. All of these threads consume 383MB which then never is released.

If i run more the heap will run out of room and Java will lock up.

I have started this investigation on Version 5.2.1.9, tried it on 5.2.3.35 and now on 5.2.4.37.

I am running this is a dev environment on my Windows Laptop Running Win7 and an i7 with 16GB of RAM.

I have the 4 dump files, but as a new user I cannot upload them. the two heap dumps are 112MB each and the two thread dumps are 53 KB each. How can I send them to you?

@djprice Are you sure those are full heap dumps? If your JVM heap is 4GBs for example, I would expect the dump to be 4GBs. You can zip them, but they don’t compress that much. You can stick them on Google drive for now and just post the link here.

I actually decreased my heap size to 1024 for testing

I wouldn’t post any heap dump publicly.

If you have usernames/passwords/any-sensitive-information in memory then it will be available to … the public.

We’ve experienced memory leakage issues with both 4.5 and 5.2.3.35. Upgrading to the latest 5.2.5 snapshot build seems to have solved the problems in both cases. See these two discussions for more detail:

https://lucee.daemonite.io/t/upgrade-to-lucee-5-2-3-35-results-in-gc-overhead-limit-crashes/2766

https://lucee.daemonite.io/t/cfthread/2765

1 Like

@Julian_Halliwell Did you try 5.2.4.37? I’m still running 5.2.5.12 SNAP and it is working great on my cfthread heavy machines.

We loaded 5.2.4.37 on our Development server and my script appear to be running cleaner and faster. The Heap is not climbing like it was on my person laptop.
I believe that the initial problem was my person environment as well as an older version of Lucee.

Can I close this issue or does someone from the team do this? If so, please close this issue.

No we went straight for 5.2.5.20 It was hard to know which version contained the fix we needed so we just went for the latest to make sure we got it. Seems quite stable.

do we have a heap dump to look into?

@micstriit The OP sent a message with the heap dumps shared in Google drive and said he would delete the links after they had been downloaded. It looks like you’re late to the party :slight_smile: Foreseeing this possible outcome, I downloaded them the other day while the links were published and will send them to you now on Slack.

Yesterday I downgraded our production servers from Lucee 5.2.4.37 to 5.1.3.18. Production environment was stable for months on 5.1.3.18. After upgrading to 5.2.4.37 we had three outages in one week all related to memory problems. So I definitly things there is something wrong in this release…

You might want to try 5.2.5.20 which is currently the RC. We were having serious memory issues with earlier versions but this one has been very stable for a couple of months now.

3 Likes

I would have to agree that there is an issue with 5.2.4.37 - after upgrading, servers that ran stable for months are now consuming the heap and throwing GC errors.

Hoping 5.2.5 will resolve the issue when it’s released.

Any idea when 5.2.5 will be released. Since upgrading to 5.2.4.37 I’ve needed to add a scheduled task to restart Lucee every 72 hours to clear the Java Heap otherwise I run into GC errors and it crashes.

Or would I be better rolling back to 5.2.3.35?

T.

We found 5.2.3.35 to be unstable and were also having memory issues with 4.5. In the circumstances we thought it was worth giving 5.2.5 a try despite being just a SNAPSHOT at the time.

It’s turned out to be rock-solid over the last 2-3 months. No more memory/thread problems.

Your situation may prove to be different, but you might want to consider going straight to 5.2.5 even though it’s still a RC.

Thank Julian… others have made similar statements.

I had actually skipped 5.2.3.35 and went to the current from 5.2.2.71

Something to consider.

T.

I should just add we did a clean install of 5.2.5 having removed our previous 4.5 installation.

5.2.5.0 ran stable for 3 days with only moderate heap elevation. Decided to go to the 5.2.5.20-RC. Resulted in requiring two server restarts in 48 hours. Issue didn’t appear to be heap, but excessive CPU. The heap showed it had climbed to 83%, but the actual server memory usage was less than 2gb but the CPU running at 90%.

Nothing in the error logs regarding heap issues, only a couple of errors referencing a failed feed connection (twitter).

Any ideas? At this point I think I’m going back to 5.1.1.65 (the last stable version I was running).