We’ve got an open bug report here: [LDEV-2904] - Lucee but while we wait for it to hopefully be fixed,I’m wondering if anyone has any workaround ideas for us so we can remain updated without crashing.
In a nutshell, ever since Lucee 5.3.5.78, Lucee has been continually loading classes that it already has, which then fills up non-heap memory and crashes. ie: before updating, after 7 days, we had 21k classes loaded. After updating, after 7 hours, we had 70k classes loaded.
Is there any way to clear these out, or other ideas other than restarting?
Normally we’d just downgrade, but seems like some other components (Java Mail/S3) are now less happy being downgraded.
Still seems to happen in 6.x, though note the original issue was locked. I think the v6 version is : Jira
We’ve got a cluster of a few machines, and I keep catching them in the act of loading ~40k classes in the space of a few tens of seconds.
Code does not update on these machines, so something must be leaking.
I’ve yet to find anything in common from logs at these times, and there are no abnormal requests, so it’s not like “every time we run API /goo” there’s a spike in class loading. Stats are from DataDog’s generic Tomcat/JVM metrics dashboard
This was resolved in 5.4 with moving to having a single class loader per class - rather than mapping. Since 6.x (not sure exactly), this was refactored and the issue returned for us.
If I understand correctly, a dynamically loaded class will remain in the “loaded class” off-heap code cache(?) until its class loader no longer exists and is garbage collected. This is why the class loader per class approach was so effective - when Lucee decided to recreate a class from source, it would remove any old references and class loader so cleanup would happen.
I think that Micha would argue that the classloader per class approach was a patch and the real cause was Lucee reloading classes that did not need to be reloaded which is what I believe we’re seeing in 6.x.
After several attempts at resolving in 6, I’m testing applications in 7 to see if there is a significant change (as Micha has said that the class loading has been rewritten in 7 again). Initial signs show some promise. Classes are not being unloaded with the rate that they were in 5.4, BUT so far I have not seen a steady increase of class loading and it appears to be levelling out. Further testing and time will tell.
That would make sense, and explain why we see “classes loaded” ticking up on a sites that do not ever change their files, and as far as we can, avoid even createObject() (cfc or Java)…
Is it possible to fix in 6.x ? And if not (or it is but isn’t going to be) are there any code patterns to be avoided to stop Lucee reloading things ? Should we be look at setting long application timeouts or java loader watch intervals ? is there a setting to say “never check files for changes” already maybe ?
As a side note (I’m having these same issues with 6) I’ve downgraded my 6 install to 6.2.4.21-RC and this has improved things somewhat. The Class loading/meta space still climbs but at least the G1 Eden space (Normal Heap memory) is stable. I went form having to restart our server every 3 days to now going 1.5 to 2 weeks. And altho the non-heap memory is rising, my total server memory usage is stable (can’t answer why) We’re considering moving to 7, but will have to do some real testing to see if the problem is gone. This is all very frustrating as our old 5.4 server run flawlessly.
I don’t think we use dynamically loaded Java classes to the extent that’s being talked about here (and we haven’t noticed any memory leaks on 5.4.8.2).
We have 340 references to CreateObject(“java”…) throughout our application
