Tomcat crashes using COM object repeatedly during long-running process

When processing ~500K database records, Apache Tomcat will unexpectedly exit. This is the error in Event Viewer: The Apache Tomcat 9.0 Lucee service terminated unexpectedly. It has done this XX time(s).

Nothing we can see in the Tomcat logs is pointing to the actual cause. We’ve tried updating Lucee, Apache, and the JRE, we’ve fiddled with memory settings, we’ve adjusted Lucee and IIS request timeouts, and we’re currently at a loss as to what is the ultimate culprit. Watching memory and CPU graphs while the process is running doesn’t show any unexpected spikes, and Tomcat never surpasses ~1.5GB of memory consumed.

The process dies at around the 400K mark. It doesn’t matter if we attempt to process 500K records at once, or 300K (which runs successfully) and then another 200K (which fails after about 100K iterations), leading us to believe it’s a memory leak somewhere.

Trying to isolate the issue, we’ve tried 500K iterations of calling the COM object with a hard-coded address, eliminating database calls altogether, and Apache Tomcat still quits in the middle of the job.

Here’s some semi-anonymized code.

<!--- create an object for looking up address information --->
<cfset zipObj = createObject("com","AddressObject.AddressCheck")>	

<cfloop condition="there are more records to be processed">
	<cfquery name="contactRecords" datasource="#dsn#">
		SELECT TOP 1000 
			ID,
			[multiple address fields]
		FROM Contacts
		WHERE AccountId = <cfqueryparam value="#AccountId#" cfsqltype="integer">
	</cfquery>
	<!--- find zipcodes for contact records and update if appropriate --->
	<cfloop query="contactRecords">
		
		<cfset fullZip = "">
		<cfscript>
			 // Clear then fill properties for lookup
			zipObj.clearProperties();
			// pass properties to zipObj
			zipObj.address = contactRecords.Address1;
			zipObj.address2 = contactRecords.Address2;
			zipObj.city = contactRecords.City;
			zipObj.state = contactRecords.State;
			zipObj.zip = contactRecords.Zip;
			// Perform zipcode lookup and record flag indicating success
			localvar.codable = zipObj.verifyAddress();
			// return ZIP+4 as fullZip
			if( localvar.codable and len(zipObj.plus4) ){
				fullZip = "#zipObj.zip#-#zipObj.plus4#";
			}
		</cfscript>
		<!--- if zipcode variable was filled, perform updates --->
		<cfif len(fullZip)>
			<!--- update zipcode in database --->
		</cfif>
		<!--- decide if there are more records to be processed --->
	</cfloop>
</cfloop>

Or, even more succinctly:

create COM object
call COM object 500,000 times
Tomcat dies before completing its job

Any thoughts? :worried:

OS: Windows 10 Pro 64-bit
Java Version: JRE from OpenJDK 11
Tomcat Version: 9.0.43
Lucee Version: 5.3.7.48

1 Like

Lucee is using a slightly older version (1.16) of jacob, the latest is 1.20, that would be the first step to rule out any problems in the upstream lib which have already been fixed.

I filed a bug [LDEV-3314] update jacob library to 1.20 - Lucee

you might find some useful info here JACOB - Java COM Bridge / Discussion / Help

But as Lucee’s COM support really isn’t a high priority, you may want to consider sponsoring a fix?

Thank you for the suggestion, Zac. I will investigate that approach and see if it helps with the issue.

While your suggestion did seem to help with the issue, it did not completely alleviate the behavior. I can now run approximately 850,000 iterations of the task before the server terminates, which is about double the previous amount, but still not suitable for our production purposes.

You suggested possibly sponsoring a fix. How would we go about doing that? Otherwise we will need to investigate a different approach to the problem.

Progress! Which suggestion exactly?

The contact form is here Lucee :: Consulting

Using the 1.20 version of JACOB.

Cool. Could you add a comment regarding that to the issue I created?