River_bender, how have things gone since you brought this up? If you’re still having problems, I have some thoughts from looking across the whole thread of replies.
First, I will say that the error you report is an indication not that Lucee (or Tomcat or the JVM) is “out of memory” but rather that your box was, at the time, so that in fact it could not give Lucee/Tomcat/the JVM the memory is had been told it could have (via the xmx). And indeed you indicated later that IIS workers were using up most of the OS memory.
Then as for your sense that your settings of XMX were being ignored, it may have to do with where you were setting those. You don’t indicate that at first, same for when you said that with a set of jvm args Terry offered “lucee wouldn’t start”. Where were you putting those?
Then later Andreas proposed you put them in the java options tab of the tomcat9w.exe. You said you put them there and the service would not start. To be clear, those args need to each be on their own line (in that “java options” tab, as opposed to about anywhere else). Also, in that UI, note that the min and max heap are to be set in a separate field offered, rather than as normal java args.
Finally, in case anyone finds this thread in the future, and they find that that tomcat9w.exe fails to open (reporting the specified service does not exist as an installed service. unable to open the service ‘tomcat9’), look for a luceew.exe instead and try that.
Let us know if any of this gets you going, River_bender, or if perhaps you were already set (in which case I leave this as info for future readers).