Tomcat 9 Upgrade Issues

dev support
1

We’ve been running our instance of Lucee on a 64-bit Windows 2022 server for quite a while now without any major hiccups. However, if we upgrade Tomcat from 9.0.85 to 9.0.95, our web app appears to run just fine, but then seems to cause Tomcat to crash (it spits out dump files). Of course, it’s intermittent and we can’t consistently reproduce the issue.

Lucee 6.0.0.585 (tried updating to 6.1.0.243 but same result)
Adoptium JRE 11.0.22
Boncode 1.0.42
mod_cfml 1.1.11

We’ve got a second set of Windows-based Lucee servers (running a completely different web app) that we updated Tomcat on, and that one seems happy. It just doesn’t seem to like something with the existing app. And of course, we’re not getting any useful log information anywhere that points a finger at the culprit (we’ve adjusted log levels, restarted services/server, etc…).

Has anyone run into this scenario or have thoughts on what we might want to chase down?

1 Like
2

You could gradually update to 86-94 and see where the issue is.

the complete changelog is here:
https://tomcat.apache.org/tomcat-9.0-doc/changelog.html

Maybe something in your code is affected by an update.

3

Do the dump files start with text lines indicating the error? If so, what is it? That might give us clues.

4

Example of the start of one of the dump files…

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000012201012d91, pid=3456, tid=3212
#
# JRE version: OpenJDK Runtime Environment Temurin-11.0.22+7 (11.0.22+7) (build 11.0.22+7)
# Java VM: OpenJDK 64-Bit Server VM Temurin-11.0.22+7 (11.0.22+7, mixed mode, tiered, compressed oops, g1 gc, windows-amd64)
# Problematic frame:
# C  [jacob-x64.dll+0x12d91]
#
# CreateCoredumpOnCrash turned off, no core file dumped
#
# If you would like to submit a bug report, please visit:
#   https://github.com/adoptium/adoptium-support/issues
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  S U M M A R Y ------------

Command Line: -Dcatalina.home=E:\Programs\tomcat -Dcatalina.base=E:\Programs\tomcat -Dignore.endorsed.dirs=E:\Programs\tomcat\endorsed -Djava.io.tmpdir=E:\Programs\tomcat\temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=E:\Programs\tomcat\conf\logging.properties -XX:-CreateCoredumpOnCrash --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED exit abort -Xms2048m -Xmx6144m 

Host: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz, 2 cores, 7G,  Windows Server 2022 , 64 bit Build 20348 (10.0.20348.2652)
Time: Thu Oct  3 09:17:22 2024 Central Daylight Time elapsed time: 297.018000 seconds (0d 0h 4m 57s)

---------------  T H R E A D  ---------------

Current thread (0x000001227e7be000):  JavaThread "ajp-nio-0.0.0.0-8889-exec-7" daemon [_thread_in_native, id=3212, stack(0x0000001b51200000,0x0000001b51300000)]

Stack: [0x0000001b51200000,0x0000001b51300000],  sp=0x0000001b512fce30,  free space=1011k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [jacob-x64.dll+0x12d91]

Oh, and yes, our Tomcat is installed to E:\ (to make sure runaway processes do not bring the OS itself to its knees).

5

We can certainly put it on our ToDo list to try out 9.0.86 to see if that has issues too, but we ran into this error with other versions lesser than 9.0.95 (but we didn’t go as far back as 9.0.86 when trying things).

6

Ok, and as you can see that says it’s an exception access violation, specifically in jacob-x64.dll. That’s not a common problem in Lucee, in my experience.

So I’d ask a few followups:

  • that Java 11.0.22 is what you’ve been running, both now and when it worked?
  • indeed, would you say “nothing else has changed” between it working and failing, beside the Tomcat upgrade?
  • how are you running lucee on tomcat? Did you deploy the lucee war? If so, are you installing a new tomcat and redeploying the war? Or are you trying to update the tomcar in-place?
  • Or is this the lucee express or installer, which itself implements a tomcat for us? And if so, again are you self-updating the Tomcat within it? What if instead you implemented a new version of that Lucee (and its tomcat)?
7

We try to use Puppet as much as we can on our servers, and configurations haven’t changed much at all on this particular Windows server. We also try to reduce the number of people that manually interact with the server to prevent unwanted changes as well.

We typically leverage the Lucee.jar file, and the Tomcat .ZIP file, when we update things. We’ve been running Lucee 6.0.0.585 and Tomcat 9.0.85 since 2/1/2024 without any blips, and have not made any updates/changes to Adoptium JRE 11.0.22 since 2/1/2024 either (it installs to its own directory outside of Tomcat and Lucee).

E:\Programs\Tomcat
E:\Programs\Lucee
E:\Programs\Java

Again, everything runs fine, but things fall apart after updating Tomcat (and only Tomcat).

We spent some time and upgraded Tomcat incrementally last week and found that things broke when upgrading from Tomcat 9.0.89 to 9.0.90. We weren’t able to find a culprit, but after bouncing around some things we ended up uninstalling/removing Lucee, Tomcat (9.0.95 this time), Boncode, Java, and mod_cfml entirely (removing their folders too), and then redeploying the newest versions fresh via Puppet.

After that, our AppDev was able to clean up some code and things seem to be working (at least, Tomcat doesn’t fall over randomly so far).

If we discover the root cause, or if our problem continues to crop up, I’ll try posting here again, but for now we’re going to assume the gremlin has been slain and move on to other fires…

1 Like