Now I realise that the vewrsion of Lucee we are running is a little old, now.
But the reason for my post is a two part one.
- To let you know about an incident and ask for help with it specifically - perhaps someone has a similar experience / thoughts?
- To ask how I might get “JVM” information / state at the time of an error.
We had an issue where we had an incomplete record saved and attempting to read it / display it - threw an exception that was “swallowed” by the JVM.
“Instantly” all running requests failed, complaining about ORM classes not being found.
(The error message was xxx was not found : here is a list of known ones…)
Our platform has “Customers”
Our Customers have “Users”.
There is no custromer / account / role overlap. a user of Customer “A”, cannot interact with Customer “B” resources.
- All logged in users, irrespective of Customer Account were affected.
And : I dont know if I am recalling correctly or not - but I THINK our scheduled tasks also stopped working, too.
Anecdotedly - we believe the issue caused by accessing a database records by ORM - with incomplete data.
Perhaps there is a missing FK or PK or whatever…
We have a “long” process that creates and saves new customers to our platform.
There are a lot of moving parts to our application and a lot of data to be actioned / saved.
During the creation of some “demo” customer accounts the process did not complete fully - resulting in somethings being saved - others not at all or in an incomplete state.
When attempting to interact wirth the new customer accounts in the UI, we encountered the
symptoms.
Aside :
- We have corrected the code that allowed the faulty state, in the first instance.
- We have disabled the accounts to stop them being used in the UI.
At the time of the exception:
- Running
applicationStop()didn’t help. - Running
ormReload()didn’t help. - Reloading Mach-II (though not expected to do anything) - didn’t help, either.
Restarting Lucee (tomcat) - was the path taken to get our production instance back online.
Extra considerations:
- We’re using the Lucee 3.5.x ORM extension.
- We can’t upgrade to the Lucee 5.x or the Ortus 5.x ORM extension, at the moment : though it is being worked on - with some help from the Ortus clan.
- We DON’T have Fusion Reactor.
Part 2
- The ONLY error that appears in any log is the “Class Not found” error.
- ALL logs are set to DEBUG.
So there doesn’t seem like anywhere near enough noise in the logs for what we were seeing.
I am thinking that other erros were being swallowed by the JVM - or perhaps something else… Some sort of “panic” in the JVM?
I have no idea - of course - and thus;
How would I go about getting a current state / dump / whatever the correct terminology is of the JVM - in this instance?
As always - Thanks.
Don’t forget to tell us about your stack!
OS: AWS Linux 2 (CentOS 7)
Java Version: 11.x
Tomcat Version: 9.x
Lucee Version: 5.3.10.97