Concurrent Modification Exception in queryExecute()

Recently we’ve switched from 6.2.0.321 to 6.2.1.122 and into single mode and following errors started popping up in the log:

java.util.ConcurrentModificationException

lucee.runtime.exp.NativeException: java.util.ConcurrentModificationException at 
java.base/java.util.HashMap$HashIterator.nextNode(HashMap.java:1605) at 
java.base/java.util.HashMap$ValueIterator.next(HashMap.java:1633) at 
lucee.transformer.library.tag.TagLib.getTag(TagLib.java:154) at 
lucee.runtime.tag.TagUtil.setAttributeCollection(TagUtil.java:95) at 
lucee.runtime.functions.query.QueryExecute.call(QueryExecute.java:64) at 
afcai.automation.aiprofiledataupdate_cfm$cf.call(/AfcAI/Automation/AIProfileDataUpdate.cfm:144) at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1082) at 
lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1006) at 
lucee.runtime.PageContextImpl.doInclude(PageContextImpl.java:987) at 
afcengine.cfc.onrequest_cfc$cf.udfCall(/AffinoMapping/AfcEngine/cfc/OnRequest.cfc:249) 

It started happening in random places but it’s always on queryExecute()
Sometimes query is cached, sometimes it’s not, sometimes it’s from DB, sometimes it’s QoQ.
It’s intermittent and I can’t can’t provide code example to replicate it, but it’s happening for example in this query:

local.qGetZone = queryExecute('
	SELECT "ZoneCode", "BotPermittedDomains"
	FROM "Zone"
	WHERE 
		"Active" = 1
		AND (
			"ZoneDomain" = :sZoneDomain1
			OR '','' + "ZoneOtherDomains" + '','' LIKE :sZoneDomain2
		)',
	{
		sZoneDomain1 = { cfsqltype="cf_sql_varchar", value=CGI.SERVER_NAME },
		sZoneDomain2 = { cfsqltype="cf_sql_varchar", value='%,#CGI.SERVER_NAME#,%' }
	}, {
		datasource   = Application.DB,
		cachedwithin = CreateTimeSpan(0,0,5,0)
	}
);

There’s no iteration here, CGI is set to be read only. The stack trace for this one is:

lucee.runtime.exp.NativeException: java.util.ConcurrentModificationException at 
java.base/java.util.HashMap$HashIterator.nextNode(HashMap.java:1605) at 
java.base/java.util.HashMap$ValueIterator.next(HashMap.java:1633) at 
lucee.transformer.library.tag.TagLib.getTag(TagLib.java:154) at 
lucee.runtime.tag.TagUtil.setAttributeCollection(TagUtil.java:95) at 
lucee.runtime.functions.query.QueryExecute.call(QueryExecute.java:64) at 

Does anyone have idea what could be causing it?

OS : Linux 6.8.0-1028-aws
Java Version : 21.0.6
Tomcat Version : 9.0.104
Lucee Version : 6.2.1.122

https://luceeserver.atlassian.net/browse/LDEV-5595

That’s a curly one! There isn’t a caused by at the bottom of the stack trace is there?

The problem is occurring whilst setting the attributeCollection on a cfquery tag, as queryExecute is a cfml wrapper around cfquery

Unfortunately no - all the stack traces end in

org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346) at 
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:397) at 
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) at 
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:935) at 
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1792) at 
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) at 
org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1189) at 
org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:658) at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: 
java.util.ConcurrentModificationException ... 59 more

Prod or local or both?

Any code doing config changes?

Cfadmin or configImport?

I did try to repro but couldn’t

Dev, staging and live servers - we don’t use local.

Cfadmin

Not that I’m aware of. Errors happen intermittently - when there’s traffic and middle of the night when only scheduled tasks are running.

In application.cfc we have an include that sets all the app settings:

component displayname="Application" output=false {
	this.name = "AppNameHere;
	
	include "ApplicationCheck.cfm";

	public void function OnApplicationStart() {....

Could that be affecting things?

Also - we didn’t have that issue on 6.2.0.321, started happening after we updated to 6.2.1.122

my rough working theory is that some cfadmin operations can trigger a full reload of config, which could cause the tag definitions to be reloaded, which might lead to the concurrent modification

I don’t know if it’s related, but started seeing following error - very rarely though:

lucee.runtime.exp.NativeException: Cannot invoke "lucee.transformer.library.tag.TagLib.getTag(java.lang.Class)" because "tld" is null at 
lucee.runtime.tag.TagUtil.setAttributeCollection(TagUtil.java:95) at 
lucee.runtime.functions.query.QueryExecute.call(QueryExecute.java:64) at 
afositeanalysis.automation.aggregatecontentview2_part1_cfm990$cf.call(/AfoSiteAnalysis/Automation/AggregateContentView2-part1.cfm:539) at 
lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1082) at 
lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1006) at 
lucee.runtime.PageContextImpl.doInclude(PageContextImpl.java:987) at 
afcengine.cfc.onrequest_cfc$cf.udfCall(/AffinoMapping/AfcEngine/cfc/OnRequest.cfc:249) at 
lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:112) at 
lucee.runtime.type.UDFImpl._call(UDFImpl.java:357) at 
lucee.runtime.type.UDFImpl.callWithNamedValues(UDFImpl.java:214) at 
lucee.runtime.ComponentImpl._call(ComponentImpl.java:679) at 
lucee.runtime.ComponentImpl._call(ComponentImpl.java:595) at 
lucee.runtime.ComponentImpl.callWithNamedValues(ComponentImpl.java:2120) at 
lucee.runtime.util.VariableUtilImpl.callFunctionWithNamedValues(VariableUtilImpl.java:858) at 
lucee.runtime.PageContextImpl.getFunctionWithNamedValues(PageContextImpl.java:2099) at 
application_cfc$cf$1g.udfCall(/Application.cfc:144) at 
lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:112) at 
lucee.runtime.type.UDFImpl._call(UDFImpl.java:357) at 
lucee.runtime.type.UDFImpl.call(UDFImpl.java:224) at 
lucee.runtime.ComponentImpl._call(ComponentImpl.java:678) at 
lucee.runtime.ComponentImpl._call(ComponentImpl.java:595) at 
lucee.runtime.ComponentImpl.call(ComponentImpl.java:2101) at 
lucee.runtime.listener.ModernAppListener.call(ModernAppListener.java:477) at 
lucee.runtime.listener.ModernAppListener._onRequest(ModernAppListener.java:212) at 
lucee.runtime.listener.MixedAppListener.onRequest(MixedAppListener.java:41) at 
lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2810) at 
lucee.runtime.PageContextImpl._execute(PageContextImpl.java:2797) at 
lucee.runtime.PageContextImpl.executeCFML(PageContextImpl.java:2768) at 
lucee.runtime.engine.Request.exe(Request.java:45) at 
lucee.runtime.engine.CFMLEngineImpl._service(CFMLEngineImpl.java:1109) at 
lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:1066) at 
lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:97) at 
lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:51) at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:623) at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:199) at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at 
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168) at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482) at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at 
org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:762) at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346) at 
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:397) at 
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) at 
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:935) at 
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1792) at 
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) at 
org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1189) at 
org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:658) 
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) at 
java.base/java.lang.Thread.run(Thread.java:1583) Caused by: 
java.lang.NullPointerException: Cannot invoke "lucee.transformer.library.tag.TagLib.getTag(java.lang.Class)" because "tld" is null ... 56 more

Theory confirmed, I created a repo, nicknamed “Torture”, available for birthdays, bah mitzvahs and weddings

I’ve had a look at your code and you’re using removeCacheConnection. We’re not doing anything like that, not using Adminstrator.cfc, not doing anything requiring admin password.

But we do use quite a lot cfschedule() - could it be causing issues?

Anything triggering a config reload could cause this issue

@Lucas can you try the 6.2.2.31 snapshot and report back?

We’ve installed it on the dev server now, but as we’re unable to replicate it may take some time to report back. I’ll keep an eye on the logs.

No errors so far - but we don’t have much traffic on the dev servers.

1 Like

We have been running the 6.2.2.31-SNAPSHOT for a couple of days now and error doesn’t happen any more. Thanks for sorting.

When is the next stable release planned?

Same here, seems to be working without issues. Thanks for the fix - was the major issue preventing us from upgrading to v6.x

RC coming out today

1 Like