We recently upgraded from Lucee 5.3.3 to 5.4.3 and are now seeing our meta space (off-heap) gradually increase over time until this eventually causes problems. Inspecting the classes loaded using jcmd <pid> GC.class_stats shows a lot of duplicate classes for cfc and cfm templates. We have inspect templates never and this is in a containerised environment where the files never change during the life of the application running.
My suspicion here is that this is caused by the following ticket: [LDEV-2904] - Lucee. In particular, a change to MappingImpl where a seemingly abitrary, and hardcoded, limit was set individually on CFC and CFM templates in the class loader(s) for the mapping:
Old limit for sum of all templates: 6,783 (introduced in [LDEV-2105] - Lucee)
New limit(s): 3,000 cfc templates, 2,000 cfm templates stored in separate class loaders
Our application has a mapping with over 3,000 cfc templates and over 2,000 cfm templates. However, it has under 6783 in total which might explain why we did not see this issue with 5.3.3
Playing around with a local build I am able to reproduce this fairly quickly by hardcoding the max numbers of templates in the MappingImpl.java class to be something very small (i.e. 200 for both CFCs and CFMs). After loading the application and loading a handful of pages, I am able to see > 1,000 duplicate classes when running jcmd <PID> GC.class_stats. When I increase the limits to well over their existing values, I do not see these duplicates.
My question here I think is where do these limit numbers come from and could/should they be configurable?
This of course may not be the cause at all, but it looks like a very likely candidate given the numbers we see and the slowish rate at which we start to see these problems in our applications.
Edit: I have now created: [LDEV-4739] - Lucee as this is defo looking like a bug given the evidence since scraped.