Perm Gen Space

mysap3com · November 27, 2017, 8:23am

anyone have error message like this? my app is very slowly

2017-11-27 15:14:24.801  Free Perm Gen Space is less than 5% free: shrinking all template classloaders : consider increasing allocated Perm Gen Space

i have already set memory on tomcat/bin/setenv.sh

#! /bin/sh

export CATALINA_OPTS="$CATALINA_OPTS -Xms512m"
export CATALINA_OPTS="$CATALINA_OPTS -Xmx6024m"
export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxPermSize=1024m"

my physically memory is 8GB, my server is ppc64 using rhel and i install lucee using 5.2.3.35.war.

bdw429s · November 27, 2017, 4:51pm

Firstly, is there a reason you’re not on Java 8 yet? As for your error messages, sounds like something is loading Java classes over and over again.

For starters are you using Javaloader? Always always always make sure you cache your javaloader instance in the server scope and don’t recreate it. The next thing is are you using TestBox with heavy mocking/stubbing or WireBox AOP on transients. Either of those can sometimes create a large number of new class files due to the way they create .cfm templates and cfinclude them on the fly.

And finally, use a tool like FusionReactor to monitor your permgen space and see if the jumps correspond with any particular scheduled tasks or jobs on your server.

mysap3com · November 28, 2017, 3:30am

Firstly, is there a reason you’re not on Java 8 yet? As for your error messages, sounds like something is loading Java classes over and over again.

i’m using ibm-java-sdk-8.0-5.5-ppc-archive.bin

For starters are you using Javaloader? Always always always make sure you cache your javaloader instance in the server scope and don’t recreate it. The next thing is are you using TestBox with heavy mocking/stubbing or WireBox AOP on transients. Either of those can sometimes create a large number of new class files due to the way they create .cfm templates and cfinclude them on the fly.

I don’t know how to configure it. Can you give me sample, please…

And finally, use a tool like FusionReactor to monitor your permgen space and see if the jumps correspond with any particular scheduled tasks or jobs on your server.

ok, i’ll try to use this tool.

bdw429s · November 28, 2017, 4:03am

i’m using ibm-java-sdk-8.0-5.5-ppc-archive.bin

You may have that installed, but I’m pretty sure you wouldn’t be getting any logging related to perm gen space if you were on Java 8… Dump out the server scope in your app to confirm what version of java is being used.

I don’t know how to configure it. Can you give me sample, please…

Configure what? I listed 3 different libraries. Are you using any of those libraries?

mysap3com · November 28, 2017, 9:52am

This is my java information:
java_ibm

For library i’m using cfspreadsheet with inspect Once and for coding i’m never using Javaloder.

This log always show up until now, i don’t know what wrong with my lucee config

2017-11-27 15:14:24.801 Free Perm Gen Space is less than 5% free: shrinking all template classloaders : consider increasing allocated Perm Gen Space

bdw429s · November 28, 2017, 5:15pm

Ahh, I found where that error is coming from. It’s coming from Lucee itself, not the JVM which explains a lot. You have no perm gen space at all, but Lucee hasn’t been adjusted fully for Java 8 and it still has a permgen check in place. I’m honestly a little curious what the heck it’s measuring on your Java 8 system. From the looks of the SystemUtil class, it’s looking over a bunch of MemoryPoolMXBean classes for one that has a specific name in the title.

I put in this ticket:
https://luceeserver.atlassian.net/browse/LDEV-1598

What does FusionReactor show for your memory spaces?

Leftbower · December 29, 2018, 1:38am

I just started getting this error. Only thing I did was download the OpenJDK 11.0.1+13 from the adoptopenjdk.net site, unzip and point Tomcat to it. The stderr log fills up wit this error without ever even hitting Lucee.

I’d already been using Tomcat 9.0.12 and OpenJDK 11.0.1 GA from JDK 11 Releases on several other machines without trouble.

The only difference is that the machine having issues is a 32b Windows machine, the others are 64b Windows machines. AdoptOpenJDK has a Windows 32b version and other sites do not.

I am running Lucee 5.3.2.17-SNAPSHOT (not updated beyond that due to this issue:
Redirecting to Google Groups)

When upgrading Lucee to any SNAPSHOT above Lucee 5.3.2.17-SNAPSHOT, our site gets this error:
Type: java.io.IOException Messages: org.osgi.framework.BundleException:Bundle symbolic name and version are not unique: TOMCAT_9_0_12:0.0.0

Leftbower · January 2, 2019, 8:23pm

Is anybody else seeing this with OpenJDK 11.0.1+13?

Jordan_Michaels · January 7, 2019, 6:22pm

Gotta admit, I’m really surprised to see this. Any particular reason you’re still running 32b here? Supporting an additional bit type is a lot of work, there’s very little demand for it, and there’s no Adopt 32b Java 11 build for Linux. Due to these factors, I just stopped doing 32b builds for the installers.

Are you using ancient hardware or software that still requires 32b? Would be interested to know your reasons for continuing to run 32b in your environments.

Jordan_Michaels · January 7, 2019, 6:28pm

The current Lucee 5.3 installer builds are also running Tomcat 9 and Adopt Java 11. They passed my initial tests, but we haven’t used them too much yet in production environments. That said, we haven’t had issues with them either. My guess is that your issue is related to your 32b system or JRE.

Leftbower · January 7, 2019, 7:52pm

Thanks Jordan. The production servers are all 64-bit machines, so there’s no problem there. However, the company’s dev workstations are all 32b as the company’s CRM, ERP, etc. software are ancient DOS-based FoxPro applications. There’s no way to get off of the FoxPro app.

The irony is it’s a computer hardware manufacturer, so all hardware is beyond state of the art. The PC used by the average secretary for email could probably run a small 3rd world country! We are in the heart of Silicon Valley and you would be absolutely amazed at how many big tech hardware companies (all whose names you know quite well!) are running DOS-based software applications!

Anyways, there’re plenty of workarounds such as having separate dev workstations, etc. but I was trying to understand why this might be happening?

bdw429s · January 7, 2019, 7:55pm

Just to be clear, is this message in the logs actually causing you issues or are you just trying to remove it? The message

Free Perm Gen Space is less than 5% free: shrinking all template classloaders : consider increasing allocated Perm Gen Space

is really more of a warning than an error, and if you’re on Java 8+ plus you don’t even have a perm gen space so you can probably just ignore it. Analyze your memory spaces and if they are healthy (especially you rmetaspace) then I wouldn’t worry about it warning.

Leftbower · January 7, 2019, 10:54pm

I haven’t noticed a specific issue other than crashing Notepad++ when trying to load the std err log due to its size! That error fills up continuously for nearly a million lines.

The issue is that the tomcat9-stderr.2019-01-02.log (so created 5 days ago) is at 135MB on a machine where the Tomcat service is running, but no apps, etc. are hitting it, so it just seems to grow non-stop?

You said the error comes from Lucee itself, so is there a way to turn off that warning?

bdw429s · January 7, 2019, 11:19pm

Yes, log files of that size does seem like an issue. Looking at the logic, there does not appear to be any way you can disable the setting. There is a permGenCleanUpThreshold that starts at 60 but regardless of what you set it to, Luee will slowly decrease it by 5 until it reaches 5 at which point your message begins appearing in the logs. The output also bypasses any log4j settings which would allow you to disable the logging of the message because it prints directly to the system.out.

github.com

lucee/Lucee/blob/master/core/src/main/java/lucee/runtime/config/ConfigServerImpl.java#L519


      
          			}
          		}
          		else if (check && kbFreePermSpace < 2048) {
          			LogUtil.log(ThreadLocalPageContext.getConfig(this), Log.LEVEL_WARN, ConfigServerImpl.class.getName(),
          
          
					" Free Perm Gen Space is less than 2Mb (free:" + ((SystemUtil.getFreePermGenSpaceSize() / 1024)) + "kb), shrinking all template classloaders");
          			// first request a GC and then check if it helps
          			System.gc();
          			if ((SystemUtil.getFreePermGenSpaceSize() / 1024) < 2048) {
          				shrink();
          			}
          		}
          	}
          
          
	private void shrink() {
          		ConfigWeb[] webs = getConfigWebs();
          		int count = 0;
          		for (int i = 0; i < webs.length; i++) {
          			count += shrink((ConfigWebImpl) webs[i], false);
          		}
          		if (count == 0) {

I think your only recourse is to lobby for the tiket I posted above to get fixed. I am curious however if you do have a memory space that is running low and which on it is. it appears if you don’t have a memory space with a name that looks like perm gen, Lucee simply grabs the first non-heap space it can find and if that fails, it grabs any memory bean called “Class memory”.

github.com

lucee/Lucee/blob/master/core/src/main/java/lucee/commons/io/SystemUtil.java#L228-L269


      
          
          
	public static ClassLoader getCoreClassLoader() {
          		if (coreCL == null) coreCL = new ClassLoaderHelper().getClass().getClassLoader();
          		return coreCL;
          	}
          
          
	public static MemoryPoolMXBean getPermGenSpaceBean() {
          		java.util.List<MemoryPoolMXBean> manager = ManagementFactory.getMemoryPoolMXBeans();
          		MemoryPoolMXBean bean;
          		// PERM GEN
          		Iterator<MemoryPoolMXBean> it = manager.iterator();
          		while (it.hasNext()) {
          			bean = it.next();
          			if ("Perm Gen".equalsIgnoreCase(bean.getName()) || "CMS Perm Gen".equalsIgnoreCase(bean.getName())) {
          				return bean;
          			}
          		}
          		it = manager.iterator();
          		while (it.hasNext()) {
          			bean = it.next();

This file has been truncated. show original

Leftbower · January 8, 2019, 12:29am

Voted.

bdw429s · January 8, 2019, 5:31pm

On a related note, here is the ticket for Adopt to start providing LInux 32 bit builds for their OpenJDK builds. Freel free to vote on it if 32 bit Linux is a need for you.

bdw429s · January 8, 2019, 5:41pm

@Leftbower I almost forgot-- I searched the Lucee JIRA yesterday and I couldn’t find a ticket for this error:

When upgrading Lucee to any SNAPSHOT above Lucee 5.3.2.17-SNAPSHOT, our site gets this error:
Type: java.io.IOException Messages: org.osgi.framework.BundleException:Bundle symbolic name and version are not unique: TOMCAT_9_0_12:0.0.0

@micstriit 's rule is if a bug isn’t in JIRA, it doesn’t exist And he doesn’t read every post here so he possibly doesn’t even know about it. Can you enter a ticket?

Jordan_Michaels · January 8, 2019, 5:58pm

lol

It’s interesting reading some of the comments in that thread. If there’s still a healthy demand for the 32b builds, I’m okay with continuing to build and test 32b installers. I just didn’t want to invest the time if it wasn’t an actual need there.

Leftbower · January 8, 2019, 6:59pm

I will do this when I get in and link it here.

Leftbower · January 8, 2019, 9:44pm

Bug filed: LDEV-2132