Lucee Site Stops Loading ColdFusion Pages - After A Few Weeks

dev support
1

My Lucee website stops loading any ColdFusion pages after 4 weeks or so of running. I need to reboot the server in order to get things running again.

Server Setup

This is a low volume website that allows users to search for information via a backend database.

  • Cloud Server 1 vCPU, 1 Gb Ram
  • Ubuntu 22.04 LTS
  • Apache Apache/2.4.52 (Ubuntu)
  • MySQL 8.0.34-0ubuntu0.22.04.1 for Linux on x86_64 ((Ubuntu))
  • Lucee lucee-5.4.3.15, Heap Size: Min (mb) [256], Max (mb) [512].

Exception Log

The exception log mostly has errors such as below.

"ERROR","http-nio-8888-exec-31","01/15/2024","23:26:14","","Statement cancelled due to timeout or client request;lucee.runtime.exp.DatabaseException: Statement cancelled due to timeout or client request
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:113)
...

Notes

  • Happens every 4 weeks or so.
  • I need to reboot the server in order to get things running again.
  • In ColdFusion Administrator, the heap size usage is typically listed at 60% or below.
  • Queries to MySQL seem to be slower on this server than my old 2015 ubuntu server with MySQL.
  • I do not have diagnostic tools such as Fusion Reactor.
  • I am a web developer, and not a server administrator, and so my server knowledge is limited.

Help

Your help and suggestions would be most welcome. Thank you.

2

could you please share the full stacktrace to repro?

3

Here is a stack trace of a typical error in the exception log.

“ERROR”,“http-nio-8888-exec-31”,“01/15/2024”,“23:26:14”,“”,“Statement cancelled due to timeout or client request;lucee.runtime.exp.DatabaseException: Statement cancelled due to timeout or client request
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:113)
at com.mysql.cj.jdbc.StatementImpl.checkCancelTimeout(StatementImpl.java:2167)
at com.mysql.cj.protocol.a.NativeProtocol.sendQueryPacket(NativeProtocol.java:1134)
at com.mysql.cj.protocol.a.NativeProtocol.sendQueryString(NativeProtocol.java:998)
at com.mysql.cj.NativeSession.execSQL(NativeSession.java:655)
at com.mysql.cj.jdbc.StatementImpl.executeInternal(StatementImpl.java:723)
at com.mysql.cj.jdbc.StatementImpl.execute(StatementImpl.java:648)
at lucee.runtime.type.util.QueryUtil.execute(QueryUtil.java:320)
at lucee.runtime.type.QueryImpl.execute(QueryImpl.java:287)
at lucee.runtime.type.QueryImpl.(QueryImpl.java:235)
at lucee.runtime.tag.Query.executeDatasoure(Query.java:1135)
at lucee.runtime.tag.Query._doEndTag(Query.java:700)
at lucee.runtime.tag.Query.doEndTag(Query.java:566)
at search.index.tags.increportexport_cfm$cf$1tx.call(/search/index/tags/IncReportExport.cfm:329)
at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1056)
at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:948)
at lucee.runtime.PageContextImpl.doInclude(PageContextImpl.java:929)
at search.index.tags.tagsearchresults_cfm$cf$1tc.call(/search/index/tags/TagSearchResults.cfm:13)
at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1056)
at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:948)
at lucee.runtime.PageContextImpl.doInclude(PageContextImpl.java:940)
at lucee.runtime.tag.CFTag.doInclude(CFTag.java:319)
at lucee.runtime.tag.CFTag.cfmlStartTag(CFTag.java:245)
at lucee.runtime.tag.CFTag.doStartTag(CFTag.java:179)
at search.results_cfm$cf$1tb.call(/search/results.cfm:196)
at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1056)
at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:948)
at lucee.runtime.listener.ClassicAppListener._onRequest(ClassicAppListener.java:65)
at lucee.runtime.listener.MixedAppListener.onRequest(MixedAppListener.java:45)
at lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2493)
at lucee.runtime.PageContextImpl._execute(PageContextImpl.java:2478)
at lucee.runtime.PageContextImpl.executeCFML(PageContextImpl.java:2449)
at lucee.runtime.engine.Request.exe(Request.java:45)
at lucee.runtime.engine.CFMLEngineImpl._service(CFMLEngineImpl.java:1216)
at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:1162)
at lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:97)
at lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:51)
…”

4

No answers for you yet, just more questions. I’m numbering them so you can reply with yes or no, referring to the numbers.

  1. Is the mysql running in the same box or another? That unclear from your original post
  2. If so, running lucee and mysql and apache in a 1g box is pushing it. But you seem to be saying it’s always worked–at least in a previous setup. Tell us what else has changed between the old and new setups.
  3. And I assume you’re saying this recurrent problem has happened “ever since” moving to the new setup, right?
  4. And I assume you’d say “nothing has changed” before the problems started, since being on the new setup, right?
  5. Finally, you say you have to reboot the box. Have you tried just restarting Lucee, and Apache if that doesn’t do it? And/or mysql, if indeed it’s on the same box?
5

Thank you for taking the time to help!

  1. Lucee, Apache, and MySQL are all running on the same server.

  2. My old server from 2015 was Ubuntu 14.04 and was running Railo, Apache, and MySQL. That server was also 1 vCPU and 1 Gb Ram, and never crashed. It was literally 1-2 years between reboots. For the new server, I set it up from scratch in October 2023 with the latest versions Ubuntu, Lucee, Apache, and MySQL. So all those are now running the latest versions, which is the main difference. No changes were made to the website code. The one thing that I noticed on the new server was that queries seemed to run slower than on the old server.

  3. Yes, the problems started on the new server. 4 weeks after it went live, the first crash occurred. I rebooted, and 6 weeks later the second crash occurred. The only exception that seems to occur is the one listed above.

  4. Nothing has changed on the new server since the launch.

  5. I have not tried restarting the apps individually. I just reboot, and then everything runs fine for a while.

6

I have not tried restarting the apps individually. I just reboot, and then everything runs fine for a while.

If next time the error occurs, you should check the CPU usage/ Memory usage and which service may stopped working or using to much ressources. Otherwise its really hard to help you, because it could be everything.
When i see the error you posted at the top it could be that the jdbc connection broke or the mysql service is not responding.
So you should also verify if the mysql server is running and working. Its maybe a problem with your mysql server and not directly with lucee, so you can also check the mysql logs for errors.

Due the hardware requirements 1GB RAM seem to be not enough. Thats the minumum requirement for ubuntu 22.04 and you also installed Apache, Lucee(Tomcat) and MySQL 8. Maybe also some other services are running in the background… That probably really on the limit.
You techstack was really old (ubuntu 14.04 etc.) before so the requirments may not match that setting like before. I personally would recommend to update to at least 2GB if possible.

1 Like
7

OH MY GOD!!! SOMEONE ELSE!!! Oh my god it’s not just me!

This sounds very similar to the issue that has been dragging me for MONTHS. I was on Ubuntu 18, MySQL 5.7, Nginx and Lucee 5.4.4.38 on a cloud VM with 2GB RAM and 2 vCPUs.

Ubuntu and MySQL were at EOL, so it was time to move, so I went to the same 2GB RAM/2 vCPU, but went with Ubuntu 22.04 (and 20.04, same problems), MySQL 8, Apache (and Nginx! no difference) and Lucee 5.4.4.38. No changes to my code, no changes to traffic, no changes whatsoever and my server would run for a few hours, randomly, and then, with no identifiable or common triggers whatsoever would completely crash, halt all network traffic and require me to do a force stop and start of the cloud server. My server logs told me I was having “Out of Memory” events, which were tied to Lucee.

I installed FusionReactor, which didn’t help with performance or uptime but did tell me that there was a ceaseless, never-ending increase in metaspace and non-heap memory allocation and usage. I looked into memory leaks (tried to, anyway) and couldn’t find anything definitive, other than “classes” being opened and then rarely closed.

I was told by other posters that the relentless, never-ending, pathological, ever-increasing memory usage was, apparently, no big deal! Nothing to worry about and totally normal (which seemed, you know, on-its-face crazy to me? But, I don’t know anything about java memory stuff, after all). But I still think this memory thing is an issue, even if everybody else keeps blowing it off and doesn’t care. Perhaps this is also happening to you? Perhaps the effect is most pronounced in our somewhat unique environments and combinations of software?

In the hours and hours and hours and hours of my short life on this beautiful planet I’ve spent working on this issue, I did run across this article mentioning that MySQL 8 absolutely consumes more memory than 5.7 and this is most noticeable on smaller servers. Additionally, this article mentions that you can turn off the “Performance Schema” monitoring tools on small instances to free up some memory. I tried doing that, but it didn’t really help, it just added a few more hours of life before the inevitable server meltdown.

So – I so very much wish I had advice to offer, but what I can mostly say is – it’s not just you! This really is a problem! It is! There is something going on with Lucee + MySQL 8.0. It seems very possible, to me, that the combination of the two puts some kind of unique strains on server memory management.

1 Like
8

This is not a flame war, nor a knock to your distro. What it is, is a factual take on it. Enterprise organizations that care about downtime and ROI do not use Ubuntu, or even Debian for that matter.
I highly suggest using a RHEL clone such as Alma Linux or RHEL if you want the commercial support.

Ubuntu is based upon Debian’s bleeding channel. Its not stable nor as secure as Debian, which has adopted a slower adoption model for stable, secure software.

if you drop your code down to the most basic Debian instance, you will see that you save 120-340 mb on just starting up your basic lucee instance and mariadb or mysql.

same code on a RHEL clone, 8 or 9 will run approximately 240-260 mb smaller on the same hardware/ same config

If you want to make Ubuntu work more like RHEL, you have to recompile the kernel to remove most of the non-foss driver hooks (not needed anyways in most cases) - remove all the experimental features, remove the modules for old and non related processors you do not have. Ubuntu’s kernel is nearly monolithic in scope.

After you have that cleaned up, remove app armor, its bloated and unix permissions are far better than this bloat.

Then use diskpart to sclice out a swap partition of at least 1GB per Processor plus 1 GB per 10 GB of memory up to 1GB per 1 GB of memory. You’re running out of memory as their kernel will still have an unmittigated memory leak, which will cause a temp file to slowly grow until you run out of resources. to speed up your box, allocate swap space.

Then install a clean install of apache or nginx from source. Yes, SOURCE as ubuntu and debian both will hav emany extra’s setup in their default binary that will slow your box down.

Then, install your lucee instance, a supported version of java.
then give java path rights to be executed by your lucee user
remove UFW, and just use iptables. Seriously, iptables -input -p TCP -dport 80 -j ACCEPT, repeat for any other ports such as 443, or allowing loopback. – There are far more optimizations needed to make your UBUNTU box work, or

install RHEL / RHEL CLone
setenforce 0 (this is temp until install complests)
yum update -y
yum install httpd mod_ssl mariadb mariadb-server (or mysql mysql-server) java java-jdk , wget, rsync,
wget pathtoluceebinary
tar -zxf luceedownloaded binary
chmod +x luceebinary
sh ./luceebinary
follow the prompts and then go to /opt/lucee/ and fire up your box.