ORM issue / JVM dump | state

Now I realise that the vewrsion of Lucee we are running is a little old, now.
But the reason for my post is a two part one.

  1. To let you know about an incident and ask for help with it specifically - perhaps someone has a similar experience / thoughts?
  2. To ask how I might get “JVM” information / state at the time of an error.

We had an issue where we had an incomplete record saved and attempting to read it / display it - threw an exception that was “swallowed” by the JVM.

“Instantly” all running requests failed, complaining about ORM classes not being found.
(The error message was xxx was not found : here is a list of known ones…)

Our platform has “Customers”
Our Customers have “Users”.

There is no custromer / account / role overlap. a user of Customer “A”, cannot interact with Customer “B” resources.

  • All logged in users, irrespective of Customer Account were affected.

And : I dont know if I am recalling correctly or not - but I THINK our scheduled tasks also stopped working, too.

Anecdotedly - we believe the issue caused by accessing a database records by ORM - with incomplete data.
Perhaps there is a missing FK or PK or whatever…

We have a “long” process that creates and saves new customers to our platform.
There are a lot of moving parts to our application and a lot of data to be actioned / saved.
During the creation of some “demo” customer accounts the process did not complete fully - resulting in somethings being saved - others not at all or in an incomplete state.

When attempting to interact wirth the new customer accounts in the UI, we encountered the

Aside :

  • We have corrected the code that allowed the faulty state, in the first instance.
  • We have disabled the accounts to stop them being used in the UI.

At the time of the exception:

  • Running applicationStop() didn’t help.
  • Running ormReload() didn’t help.
  • Reloading Mach-II (though not expected to do anything) - didn’t help, either.

Restarting Lucee (tomcat) - was the path taken to get our production instance back online.

Extra considerations:

  • We’re using the Lucee 3.5.x ORM extension.
  • We can’t upgrade to the Lucee 5.x or the Ortus 5.x ORM extension, at the moment : though it is being worked on - with some help from the Ortus clan.
  • We DON’T have Fusion Reactor.

Part 2

  • The ONLY error that appears in any log is the “Class Not found” error.
  • ALL logs are set to DEBUG.

So there doesn’t seem like anywhere near enough noise in the logs for what we were seeing.
I am thinking that other erros were being swallowed by the JVM - or perhaps something else… Some sort of “panic” in the JVM?
I have no idea - of course - and thus;

How would I go about getting a current state / dump / whatever the correct terminology is of the JVM - in this instance?

As always - Thanks.

Don’t forget to tell us about your stack!
OS: AWS Linux 2 (CentOS 7)
Java Version: 11.x
Tomcat Version: 9.x
Lucee Version: