Have had server and running fine for well over a year without issue. Today Lucee stopped and will not start. There have been no configuration changes recently.
We have tried rebooting the server, several attempts at restart lucee but nothing. We get the "It may take a few moments for Lucee… " message… but on /opt/lucee/lucee_status there is no PID.
See below for Catalina.out log. There is one error about SSL (The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib])… but nothing indicating failure to start or reason.
Catalina logs here (sorry… forgot to include in original post):
05-Oct-2021 05:25:54.113 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlValidation] to [false]
05-Oct-2021 05:25:54.118 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlNamespaceAware] to [false]
05-Oct-2021 05:25:54.167 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlValidation] to [false]
05-Oct-2021 05:25:54.168 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlNamespaceAware] to [false]
05-Oct-2021 05:25:54.169 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlValidation] to [false]
05-Oct-2021 05:25:54.169 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlNamespaceAware] to [false]
05-Oct-2021 05:25:54.170 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlValidation] to [false]
05-Oct-2021 05:25:54.170 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlNamespaceAware] to [false]
05-Oct-2021 05:25:54.172 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlValidation] to [false]
05-Oct-2021 05:25:54.172 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlNamespaceAware] to [false]
05-Oct-2021 05:25:54.173 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlValidation] to [false]
05-Oct-2021 05:25:54.173 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlNamespaceAware] to [false]
05-Oct-2021 05:25:54.174 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlValidation] to [false]
05-Oct-2021 05:25:54.175 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlNamespaceAware] to [false]
05-Oct-2021 05:25:54.176 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlValidation] to [false]
05-Oct-2021 05:25:54.176 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlNamespaceAware] to [false]
05-Oct-2021 05:25:54.177 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlValidation] to [false]
05-Oct-2021 05:25:54.177 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlNamespaceAware] to [false]
05-Oct-2021 05:25:54.178 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlValidation] to [false]
05-Oct-2021 05:25:54.179 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlNamespaceAware] to [false]
05-Oct-2021 05:25:54.180 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlValidation] to [false]
05-Oct-2021 05:25:54.180 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlNamespaceAware] to [false]
05-Oct-2021 05:25:54.181 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlValidation] to [false]
05-Oct-2021 05:25:54.181 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Engine/Host] failed to set property [xmlNamespaceAware] to [false]
05-Oct-2021 05:25:54.184 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name: Apache Tomcat/9.0.35
05-Oct-2021 05:25:54.187 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: May 5 2020 20:36:20 UTC
05-Oct-2021 05:25:54.187 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 9.0.35.0
05-Oct-2021 05:25:54.187 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux
05-Oct-2021 05:25:54.187 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 5.4.0-1057-aws
05-Oct-2021 05:25:54.187 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
05-Oct-2021 05:25:54.188 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /opt/lucee/jre
05-Oct-2021 05:25:54.188 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 11.0.7+10
05-Oct-2021 05:25:54.188 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: AdoptOpenJDK
05-Oct-2021 05:25:54.188 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: /opt/lucee/tomcat
05-Oct-2021 05:25:54.189 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: /opt/lucee/tomcat
05-Oct-2021 05:25:54.191 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
05-Oct-2021 05:25:54.191 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
05-Oct-2021 05:25:54.192 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
05-Oct-2021 05:25:54.192 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/opt/lucee/tomcat/conf/logging.properties
05-Oct-2021 05:25:54.192 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
05-Oct-2021 05:25:54.192 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
05-Oct-2021 05:25:54.192 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
05-Oct-2021 05:25:54.193 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
05-Oct-2021 05:25:54.193 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms5128m
05-Oct-2021 05:25:54.193 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx5128m
05-Oct-2021 05:25:54.193 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
05-Oct-2021 05:25:54.193 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/opt/lucee/tomcat
05-Oct-2021 05:25:54.194 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/opt/lucee/tomcat
05-Oct-2021 05:25:54.194 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/opt/lucee/tomcat/temp
05-Oct-2021 05:25:54.194 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]
05-Oct-2021 05:25:54.570 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler [“http-nio-8888”]
05-Oct-2021 05:25:54.604 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler [“ajp-nio-127.0.0.1-8009”]
05-Oct-2021 05:25:54.606 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [710] millisecondsstrong text
About a month ago a vulnerability was discovered in OPENSSL. According to the report it’s be there for years and will no doubt have industry wide implications (many things are based on it).
Like TLS1… the response may be to disable the component. Do you have the ability to IMAGE and rollback about a week?
Thanks Jay.MC. We haven’t done any updates server updates for quite some time… But are you saying this may have been an auto update to underlying OS or something?
Possibly… I’m still new to lucee and I don’t want to take shots in the dark sending possibly sending you in the wrong direction. The OpenSSL is what caught my attention.
If it’s what I suspect you may be able to manually change things off.
Do you have anything lucee related set to auto update?
Oooo… Just noticed your on linux… So I will probably just add more confusion. See if you can roll back and get a look at the versions … then compare with current.
This does look like something could be wide spread. I have created a new instance from another image running same version (Lucee 5.3.6) and same issue (lucee wont start). So the OpenSSL issue you mention may be impacting any 5.3.6 installs. We are also running most stuff on docker images with 5.3.8 and no issues there.
The plan at current is to upgrade to 5.3.8 in hope that resolves… but would be keen to explore your suggestion of manually turning things off… Can you elaborate on that one ?
Being a noob the only way I would know is to just swap the files. BUT… If the action is so aggressive you may be putting your self at risk (and probably are). With cyber attacks now a daily thing compounded by the OpenSSL suspicion… I can only say to backup your stuff more aggressively than before.
However, even with that… I believe that there are sleeper viruses in all areas at this point that have been slowly spreading around. So when people backup… they are also backing up the viruses. I got a light hit of ransomeware earlier in the year… I drilled the machine out with numerous virus scanners and everything came back clean. Yet only last week the same machine was hit this time the damage was total.
It was always a suspicious situation. In over 20 years of running servers this was my 1st Root level breech. So… I think the viruses are getting through the scanners… or at worst… being purposely let through so people can make money off it.
If you look at the history of viruses and So-Called virus protection… there has always been a mutual benefit.
I know this is pretty far off topic… But I think this is the new norm and soon someone will appear with some sort of solution to sell and attempt to get rich on .
As far as I understand, that’s not an error. It’s only saying that you haven’t apache tomcat native library installed. I’ve those log entries too on my 20.04LTS. I think there’s nothing wrong with that.
furhter:
The above is saying that your Tomcat has only been initialized(!!!) but not started. All the further log entries about server service start and context creation are missing. Here you can find some of those as an example:
05-Oct-2021 17:03:22.094 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
05-Oct-2021 17:03:22.096 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.40]
05-Oct-2021 17:03:22.127 INFO [main] org.apache.catalina.startup.HostConfig.deployDescriptor Deploying deployment descriptor [/opt/lucee/tomcat/conf/Catalina/xxx.xxx.xxx/ROOT.xml]
05-Oct-2021 17:03:22.180 WARNING [main] org.apache.catalina.startup.HostConfig.deployDescriptor Deployment of deployment descriptor [/opt/lucee/tomcat/conf/Catalina/xxx.xxx.xxx/ROOT.xml] with an external docBase means the directory [/opt/lucee/tomcat/webapps/ROOT] in the appBase will be ignored
...
...
Sometimes Tomcat simply needs a good time to start. Did you give Tomcat enough time to start? Under certain circumstances it may need more than a minute to create all contexts. Could you please recheck your logs and see if something is logged after that NFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [xxx] milliseconds?
Furher questions:
How did you install Lucee? Installer?
On what Java version Tomcat is running?