When processing ~500K database records, Apache Tomcat will unexpectedly exit. This is the error in Event Viewer: The Apache Tomcat 9.0 Lucee service terminated unexpectedly. It has done this XX time(s).
Nothing we can see in the Tomcat logs is pointing to the actual cause. We’ve tried updating Lucee, Apache, and the JRE, we’ve fiddled with memory settings, we’ve adjusted Lucee and IIS request timeouts, and we’re currently at a loss as to what is the ultimate culprit. Watching memory and CPU graphs while the process is running doesn’t show any unexpected spikes, and Tomcat never surpasses ~1.5GB of memory consumed.
The process dies at around the 400K mark. It doesn’t matter if we attempt to process 500K records at once, or 300K (which runs successfully) and then another 200K (which fails after about 100K iterations), leading us to believe it’s a memory leak somewhere.
Trying to isolate the issue, we’ve tried 500K iterations of calling the COM object with a hard-coded address, eliminating database calls altogether, and Apache Tomcat still quits in the middle of the job.
Here’s some semi-anonymized code.
<!--- create an object for looking up address information --->
<cfset zipObj = createObject("com","AddressObject.AddressCheck")>
<cfloop condition="there are more records to be processed">
<cfquery name="contactRecords" datasource="#dsn#">
SELECT TOP 1000
ID,
[multiple address fields]
FROM Contacts
WHERE AccountId = <cfqueryparam value="#AccountId#" cfsqltype="integer">
</cfquery>
<!--- find zipcodes for contact records and update if appropriate --->
<cfloop query="contactRecords">
<cfset fullZip = "">
<cfscript>
// Clear then fill properties for lookup
zipObj.clearProperties();
// pass properties to zipObj
zipObj.address = contactRecords.Address1;
zipObj.address2 = contactRecords.Address2;
zipObj.city = contactRecords.City;
zipObj.state = contactRecords.State;
zipObj.zip = contactRecords.Zip;
// Perform zipcode lookup and record flag indicating success
localvar.codable = zipObj.verifyAddress();
// return ZIP+4 as fullZip
if( localvar.codable and len(zipObj.plus4) ){
fullZip = "#zipObj.zip#-#zipObj.plus4#";
}
</cfscript>
<!--- if zipcode variable was filled, perform updates --->
<cfif len(fullZip)>
<!--- update zipcode in database --->
</cfif>
<!--- decide if there are more records to be processed --->
</cfloop>
</cfloop>
Or, even more succinctly:
create COM object
call COM object 500,000 times
Tomcat dies before completing its job
Any thoughts? 
OS: Windows 10 Pro 64-bit
Java Version: JRE from OpenJDK 11
Tomcat Version: 9.0.43
Lucee Version: 5.3.7.48