Lucee unstable under assault?

We have a new centos 6.7 install with the Lucee packaged installer (Lucee
4.5.1.024 on Tomcat 8.0.24). Its running our ex-Railo app quite happily
under normal load but when we point an external PCI compliance tester at
it, Lucee becomes unresponsive. There’s nothing in catalina.out except
possibly this:

Exception in thread “http-nio-8888-exec-10”
java.lang.IllegalMonitorStateException
at
java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:151)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261)
at
java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:449)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

The process is still there when it becomes unresponsive:

“ps wauxf” gives us:

lucee 22865 3.7 3.8 2529420 312460 pts/0 Sl 15:40 0:30
/opt/lucee/jdk/jre/bin/java
-Djava.util.logging.config.file=/opt/lucee/tomcat/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms256m
-Xmx2048m -XX:MaxPermSize=512m -javaagent:lib/lucee-inst.jar
-Djava.endorsed.dirs=/opt/lucee/tomcat/endorsed -classpath
/opt/lucee/tomcat/bin/bootstrap.jar:/opt/lucee/tomcat/bin/tomcat-juli.jar
-Dcatalina.base=/opt/lucee/tomcat -Dcatalina.home=/opt/lucee/tomcat
-Djava.io.tmpdir=/opt/lucee/tomcat/temp
org.apache.catalina.startup.Bootstrap start

not spectacularly large. We’ve set extra memory in the Java startup as you
can see, together with extra threads in lucee/tomcat/conf/server.xml:

<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
    maxThreads="1500" minSpareThreads="10"/>




but tbh I’m shooting in the dark here. Any ideas please?

Hi Tim,

Which:

  1. web server and version
  2. Java and version
  3. Which OS and version

Kind regards,

Andrew
about.me http://about.me/andrew_dixon - mso http://www.mso.net - Lucee
Association Member http://lucee.orgOn 9 November 2015 at 16:03, Tim Clarke <@Tim_Clarke> wrote:

We have a new centos 6.7 install with the Lucee packaged installer (Lucee
4.5.1.024 on Tomcat 8.0.24). Its running our ex-Railo app quite happily
under normal load but when we point an external PCI compliance tester at
it, Lucee becomes unresponsive. There’s nothing in catalina.out except
possibly this:

Exception in thread “http-nio-8888-exec-10”
java.lang.IllegalMonitorStateException
at
java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:151)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261)
at
java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:449)
at
org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

The process is still there when it becomes unresponsive:

“ps wauxf” gives us:

lucee 22865 3.7 3.8 2529420 312460 pts/0 Sl 15:40 0:30
/opt/lucee/jdk/jre/bin/java
-Djava.util.logging.config.file=/opt/lucee/tomcat/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms256m
-Xmx2048m -XX:MaxPermSize=512m -javaagent:lib/lucee-inst.jar
-Djava.endorsed.dirs=/opt/lucee/tomcat/endorsed -classpath
/opt/lucee/tomcat/bin/bootstrap.jar:/opt/lucee/tomcat/bin/tomcat-juli.jar
-Dcatalina.base=/opt/lucee/tomcat -Dcatalina.home=/opt/lucee/tomcat
-Djava.io.tmpdir=/opt/lucee/tomcat/temp
org.apache.catalina.startup.Bootstrap start

not spectacularly large. We’ve set extra memory in the Java startup as you
can see, together with extra threads in lucee/tomcat/conf/server.xml:

<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
    maxThreads="1500" minSpareThreads="10"/>




but tbh I’m shooting in the dark here. Any ideas please?


Love Lucee? Become a supporter and be part of the Lucee project today! -
http://lucee.org/supporters/become-a-supporter.html

You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/2588f0d7-358d-4fda-a7ce-fb6f4eec7763%40googlegroups.com
https://groups.google.com/d/msgid/lucee/2588f0d7-358d-4fda-a7ce-fb6f4eec7763%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

More questions… are you connecting via AJP or HTTP? and how are you doing it? What’s your apache config? is it just apache AJP connector or using mod_cfml?

Regards

MD> On 9 Nov 2015, at 16:26, Tim Clarke <@Tim_Clarke> wrote:

On Monday, 9 November 2015 16:17:37 UTC, Andrew Dixon wrote:
Hi Tim,

Which:

  1. web server and version
  2. Java and version
  3. Which OS and version

Kind regards,

Andrew
about.me http://about.me/andrew_dixon - mso http://www.mso.net/ - Lucee Association Member http://lucee.org/
On 9 November 2015 at 16:03, Tim Clarke <hey...@gmail.com <javascript:>> wrote:
We have a new centos 6.7 install with the Lucee packaged installer (Lucee 4.5.1.024 on Tomcat 8.0.24). Its running our ex-Railo app quite happily under normal load but when we point an external PCI compliance tester at it, Lucee becomes unresponsive. There’s nothing in catalina.out except possibly this:

Exception in thread “http-nio-8888-exec-10” java.lang.IllegalMonitorStateException
at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:151)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261)
at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:449)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

The process is still there when it becomes unresponsive:

“ps wauxf” gives us:

lucee 22865 3.7 3.8 2529420 312460 pts/0 Sl 15:40 0:30 /opt/lucee/jdk/jre/bin/java -Djava.util.logging.config.file=/opt/lucee/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms256m -Xmx2048m -XX:MaxPermSize=512m -javaagent:lib/lucee-inst.jar -Djava.endorsed.dirs=/opt/lucee/tomcat/endorsed -classpath /opt/lucee/tomcat/bin/bootstrap.jar:/opt/lucee/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/lucee/tomcat -Dcatalina.home=/opt/lucee/tomcat -Djava.io.tmpdir=/opt/lucee/tomcat/temp org.apache.catalina.startup.Bootstrap start

not spectacularly large. We’ve set extra memory in the Java startup as you can see, together with extra threads in lucee/tomcat/conf/server.xml:

<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
    maxThreads="1500" minSpareThreads="10"/>




but tbh I’m shooting in the dark here. Any ideas please?


Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html http://lucee.org/supporters/become-a-supporter.html

You received this message because you are subscribed to the Google Groups “Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com <javascript:>.
To post to this group, send email to lu...@googlegroups.com <javascript:>.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/2588f0d7-358d-4fda-a7ce-fb6f4eec7763%40googlegroups.com https://groups.google.com/d/msgid/lucee/2588f0d7-358d-4fda-a7ce-fb6f4eec7763%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

Centos 6.7 as I said plus java version “1.8.0_65” and apache httpd Apache/2.2.15


Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html http://lucee.org/supporters/become-a-supporter.html

You received this message because you are subscribed to the Google Groups “Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+unsubscribe@googlegroups.com mailto:lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com mailto:lucee@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/ab946851-7d85-4bb6-84c7-ef17cc783d6e%40googlegroups.com https://groups.google.com/d/msgid/lucee/ab946851-7d85-4bb6-84c7-ef17cc783d6e%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

Questions are good:)

… AJP or HTTP? tbh I’m not sure how they are doing it - as I said its an
external PCI compliance test.
…apache config attachedOn Monday, 9 November 2015 16:28:17 UTC, Mark Drew wrote:

More questions… are you connecting via AJP or HTTP? and how are you doing
it? What’s your apache config? is it just apache AJP connector or using
mod_cfml?

Regards

MD

TEMP.tar.gz (12.6 KB)

directly on a new Centos 6.7 install, not a VM

So, this is dedicated hardware?

The only reason I mention it is because kernel virtualization software
can restrict the threading of the host OS. I was simply shooting for
ideas of what might be causing issues with threading. It’s quite
possible that it’s not relevant to your situation. I wouldn’t know
without doing more research.

Warm Regards,
Jordan Michaels

It is dedicated, Jordan, thanks for trying to help. I hope someone else
chimes in.On Monday, 9 November 2015 23:54:10 UTC, Jordan Michaels wrote:

directly on a new Centos 6.7 install, not a VM

So, this is dedicated hardware?

The only reason I mention it is because kernel virtualization software
can restrict the threading of the host OS. I was simply shooting for
ideas of what might be causing issues with threading. It’s quite
possible that it’s not relevant to your situation. I wouldn’t know
without doing more research.

Warm Regards,
Jordan MichaelsOn 11/09/2015 03:00 PM, Tim Clarke wrote:

On Monday, 9 November 2015 20:09:08 UTC, Jordan Michaels wrote:

If you're using a default install of Lucee done via the installer,
then you're connecting Apache to Tomcat over HTTP using
mod_proxy_http. This is further evident due to the fact that you're
hitting an error on Tomcat's 8888 port - which is the default HTTP
port in an installer build.

In your stack trace, no Lucee classes are being hit, so the problem
is directly related to something Tomcat itself is doing.

I did a couple quick reads on the "IllegalMonitorStateException"
error, and it looks like you're hitting some kind of runtime error
with relation to threading:
http://examples.javacodegeeks.com/java-basics/exceptions/java-lang-illegalmonitorstateexception-how-to-solve-illegalmonitorstateexception/
<http://examples.javacodegeeks.com/java-basics/exceptions/java-lang-illegalmonitorstateexception-how-to-solve-illegalmonitorstateexception/>


Does your server limit threading in any way? For example, are you on
a Virtual Machine or some other kind of service that can impact
threading? There may also be connector settings that can be adjusted
to  correct for this error, but we won't know what settings to
adjust until we know why this error is being thrown in the first place.

If I were you, I'd start my search there.

Kind regards,
Jordan Michaels

Exactly right, Jordan, and I found Mr Maneas’ excellent tutorial
earlier. Unfortunately it doesn’t help me since the problem is occurring
inside Tomcat. This Lucee/Tomcat install is running directly on a new
Centos 6.7 install, not a VM. I have no idea where one might “limit
server threading” inside that set-up, we have increased the Tomcat
maxThreads settings in server.xml as already detailed above.

tbh I have no idea where to search since debugging Tomcat itself is
quite out of my league.


Love Lucee? Become a supporter and be part of the Lucee project today! -
http://lucee.org/supporters/become-a-supporter.html

You received this message because you are subscribed to the Google
Groups “Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to lucee+unsubscribe@googlegroups.com
mailto:lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com
mailto:lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/fab7461d-96a4-4400-b2b5-6168e1746b0c%40googlegroups.com
https://groups.google.com/d/msgid/lucee/fab7461d-96a4-4400-b2b5-6168e1746b0c%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

If you’re using a default install of Lucee done via the installer, then you’re connecting Apache to Tomcat over HTTP using mod_proxy_http. This is further evident due to the fact that you’re hitting an error on Tomcat’s 8888 port - which is the default HTTP port in an installer build.

In your stack trace, no Lucee classes are being hit, so the problem is directly related to something Tomcat itself is doing.

I did a couple quick reads on the “IllegalMonitorStateException” error, and it looks like you’re hitting some kind of runtime error with relation to threading:

Does your server limit threading in any way? For example, are you on a Virtual Machine or some other kind of service that can impact threading? There may also be connector settings that can be adjusted to correct for this error, but we won’t know what settings to adjust until we know why this error is being thrown in the first place.

If I were you, I’d start my search there.

Kind regards,
Jordan Michaels----- Original Message -----
From: “Tim Clarke” <@Tim_Clarke>
To: “Lucee” lucee@googlegroups.com
Sent: Monday, November 9, 2015 9:48:09 AM
Subject: Re: [Lucee] lucee unstable under assault?

On Monday, 9 November 2015 16:59:37 UTC, Tim Clarke wrote:

On Monday, 9 November 2015 16:33:07 UTC, Andrew Dixon wrote:

Hi Tim

Firstly I would remove -XX:MaxPermSize=512m as this is no longer valid
in Java 1.8 and not required, don’t think this will help but it’s good to
tidy up.

Also I would change the -Xms and -Xmx to be the same value, so -Xms2048m
-Xmx2048m, this saves it from doing too much garbage collection, which
might be the problem once you are under load.

I would then install FusionReactor and see what that shows you when you
are under load, as it will probably help point you in the direction of the
issue.

Also see Mark’s questions.

Kind regards,

Andrew

I’ve put those suggestions in place and am re-running, thanks Andrew.

The re-run has stopped with the process not responding again. The last
section of catalina.out is:

Exception in thread “http-nio-8888-exec-6” Exception in thread
“http-nio-8888-exec-7” java.lang.IllegalMonitorStateException
at
java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:151)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261)
at
java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:449)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
java.lang.IllegalMonitorStateException
at
java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:151)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261)
at
java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:449)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)


Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html

You received this message because you are subscribed to the Google Groups “Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/7c50cf20-7fc7-45b0-a377-0b879c4afe42%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I’ve put those suggestions in place and am re-running, thanks Andrew.On Monday, 9 November 2015 16:33:07 UTC, Andrew Dixon wrote:

Hi Tim

Firstly I would remove -XX:MaxPermSize=512m as this is no longer valid in
Java 1.8 and not required, don’t think this will help but it’s good to tidy
up.

Also I would change the -Xms and -Xmx to be the same value, so -Xms2048m
-Xmx2048m, this saves it from doing too much garbage collection, which
might be the problem once you are under load.

I would then install FusionReactor and see what that shows you when you
are under load, as it will probably help point you in the direction of the
issue.

Also see Mark’s questions.

Kind regards,

Andrew

Hi Tim

Firstly I would remove -XX:MaxPermSize=512m as this is no longer valid
in Java 1.8 and not required, don’t think this will help but it’s good to
tidy up.

Also I would change the -Xms and -Xmx to be the same value, so -Xms2048m
-Xmx2048m, this saves it from doing too much garbage collection, which
might be the problem once you are under load.

I would then install FusionReactor and see what that shows you when you
are under load, as it will probably help point you in the direction of the
issue.

Also see Mark’s questions.

Kind regards,

Andrew

I’ve put those suggestions in place and am re-running, thanks Andrew.

The re-run has stopped with the process not responding again. The last
section of catalina.out is:

Exception in thread “http-nio-8888-exec-6” Exception in thread
“http-nio-8888-exec-7” java.lang.IllegalMonitorStateException
at
java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:151)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261)
at
java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:449)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
java.lang.IllegalMonitorStateException
at
java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:151)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261)
at
java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:449)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)On Monday, 9 November 2015 16:59:37 UTC, Tim Clarke wrote:

On Monday, 9 November 2015 16:33:07 UTC, Andrew Dixon wrote:

Exactly right, Jordan, and I found Mr Maneas’ excellent tutorial earlier.
Unfortunately it doesn’t help me since the problem is occurring inside
Tomcat. This Lucee/Tomcat install is running directly on a new Centos 6.7
install, not a VM. I have no idea where one might “limit server threading”
inside that set-up, we have increased the Tomcat maxThreads settings in
server.xml as already detailed above.

tbh I have no idea where to search since debugging Tomcat itself is quite
out of my league.On Monday, 9 November 2015 20:09:08 UTC, Jordan Michaels wrote:

If you’re using a default install of Lucee done via the installer, then
you’re connecting Apache to Tomcat over HTTP using mod_proxy_http. This is
further evident due to the fact that you’re hitting an error on Tomcat’s
8888 port - which is the default HTTP port in an installer build.

In your stack trace, no Lucee classes are being hit, so the problem is
directly related to something Tomcat itself is doing.

I did a couple quick reads on the “IllegalMonitorStateException” error,
and it looks like you’re hitting some kind of runtime error with relation
to threading:

http://examples.javacodegeeks.com/java-basics/exceptions/java-lang-illegalmonitorstateexception-how-to-solve-illegalmonitorstateexception/

Does your server limit threading in any way? For example, are you on a
Virtual Machine or some other kind of service that can impact threading?
There may also be connector settings that can be adjusted to correct for
this error, but we won’t know what settings to adjust until we know why
this error is being thrown in the first place.

If I were you, I’d start my search there.

Kind regards,
Jordan Michaels

An observation to illustrate my point: you’ve got over 18,000 404’s (which PCI testing will produce). Do you have a site wide error handler in place for missing templates that might be the culprit?On November 10, 2015 at 1:59:01 PM, Jon Clausen (@Jon_Clausen) wrote:

Those that have chimed in already are much better with the JVM, Tomcat and Lucee server than I am, but the screenshots that I’m seeing (and what’s described in the rest of this thread) suggest that it’s not a JVM or Tomcat issue, but is rather something in the application, itself, that’s causing a race condition. Are there any writes to the application scope or locks which take place every request or I/O actions such as logging that fire on errors which might have permissions issues which trigger additional errors, additional loging attempts, etc? Those can produce these kinds of lock-ups, in my experience. If you have a high request timeout setting, conditions causing the race will not resolve themselves and the server will eventually stop responding.

In many early CFML libraries, before OOP practices were widely implemented in CFML, locks were used quite a bit, so it’s possible that, if this has a legacy code base, there’s a old third party library that’s the culprit.

Jon

On November 10, 2015 at 1:48:29 PM, Tim Clarke (@Tim_Clarke) wrote:

On Tuesday, 10 November 2015 09:39:17 UTC, Mark Drew wrote:
I would second the use of FusionReactor in this instance by the way. You will be able to monitor what is going on a lot better and see what requests are making the server inactive. It could be buffer overflow tests that the PCI company is running for example.

Without this you are just taking stabs I the dark.

If not. Try Newrelic, although the FR is much better.

I am not at my desk (but will be later to copy the setting) but you also want to do heap dump on out of memory errors (google it) setting in your setenv.sh to see what was in memory. It might help if you are actually crashing the server.

Mark Drew

fusionreactor installed, server restarted and a new PCI scan initiated.

Picture “Screenshot from 2015-11-10 18-13-50.png” was caught as the freeze started, the server in fact did respond, just very slowly. A page of our application without CSS could be loaded to a browser. “Screenshot from 2015-11-10 18-14-06.png” contains the detailed lower stats from the page. Now I hope I’m dumping something informative here lol

After about 10 minutes only this can be loaded:

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /.

Reason: Error reading from remote server

Apache/2.2.15 (CentOS) Server at myserver.com Port 443

fusionreactor still continues to update, however, and I can see memory usage (top right graph) slowly increasing then dropping, presumably as garbage collection is performed. The PCI scan fails still but only on one item: “CGI Generic Cross-Site Scripting” (grrr why can’t I turn off italics?)

Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html

You received this message because you are subscribed to the Google Groups “Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/c08e9dc9-c9b5-4dbe-825b-439cc7e755d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Tim,

That’s a start,then. First off, I would set a default value for “returnTo”, so that login redirects don’t throw 500’s. Secondly, a lot of those PCI URLS being passed are for other language file extensions. Do you have those being handled by the web server, or is Lucee forced to handle those as well? You can very easily handle those in your rewrite rules, if you have them.

Since your exception log i/o actions are firing on 500’s, rather than the web server’s log appender, it’s best to deal with those 404’s at the web server level and save yourself the expense on the app server. With 18,000 404’s in a short period of time, that will take an instant load off of your application server.

JonOn November 10, 2015 at 2:07:05 PM, Tim Clarke (@Tim_Clarke) wrote:

Ty for the thought, Jon, but since the application begins with a login page, the PCI scan is unable to access anything of our code except that. I’ve scanned the errors caused by the scan in our application log and you can see attempt-by-attempt these categories:

page not found
…has no remote function with name…
The parameter returnTo to function logIn is required but was not passed in
invalid url [empty string] for attribute url
component [controllers.LocalizationController] has no remote function with name

I’m sure that’s not exhaustive but it certainly covers that last 200+ error lines. Looks like they can’t get in to our app?

Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html

You received this message because you are subscribed to the Google Groups “Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/CAGC6_%2BoC2vsGkZY19iA0G0stsUrhdV9%3DiUyBhJqRPOxT%2BahhOw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

A clue possibly in this thread graph?On Tuesday, 10 November 2015 18:48:23 UTC, Tim Clarke wrote:

fusionreactor installed, server restarted and a new PCI scan initiated.

Picture “Screenshot from 2015-11-10 18-13-50.png” was caught as the freeze
started, the server in fact did respond, just very slowly. A page of our
application without CSS could be loaded to a browser. “Screenshot from
2015-11-10 18-14-06.png” contains the detailed lower stats from the page.
Now I hope I’m dumping something informative here lol

After about 10 minutes only this can be loaded:

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /
https://demo.manifest.info/
.

Reason: Error reading from remote server

Apache/2.2.15 (CentOS) Server at myserver.com Port 443

fusionreactor still continues to update, however, and I can see memory
usage (top right graph) slowly increasing then dropping, presumably as
garbage collection is performed. The PCI scan fails still but only on one
item: “CGI Generic Cross-Site Scripting” (grrr why can’t I turn off
italics?)

I would second the use of FusionReactor in this instance by the way. You will be able to monitor what is going on a lot better and see what requests are making the server inactive. It could be buffer overflow tests that the PCI company is running for example.

Without this you are just taking stabs I the dark.

If not. Try Newrelic, although the FR is much better.

I am not at my desk (but will be later to copy the setting) but you also want to do heap dump on out of memory errors (google it) setting in your setenv.sh to see what was in memory. It might help if you are actually crashing the server.

Mark Drew

  • Sent by typing with my thumbs.> On 9 Nov 2015, at 23:57, Tim Clarke <@Tim_Clarke> wrote:

On Monday, 9 November 2015 23:54:10 UTC, Jordan Michaels wrote:

directly on a new Centos 6.7 install, not a VM

So, this is dedicated hardware?

The only reason I mention it is because kernel virtualization software
can restrict the threading of the host OS. I was simply shooting for
ideas of what might be causing issues with threading. It’s quite
possible that it’s not relevant to your situation. I wouldn’t know
without doing more research.

Warm Regards,
Jordan Michaels

It is dedicated, Jordan, thanks for trying to help. I hope someone else chimes in.

Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html

You received this message because you are subscribed to the Google Groups “Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/c94ca271-b87a-4e41-b5c2-915f7556b9dc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Those that have chimed in already are much better with the JVM, Tomcat and

Lucee server than I am, but the screenshots that I’m seeing (and what’s
described in the rest of this thread) suggest that it’s not a JVM or Tomcat
issue, but is rather something in the application, itself, that’s causing a
race condition. Are there any writes to the application scope or locks
which take place every request or I/O actions such as logging that fire on
errors which might have permissions issues which trigger additional errors,
additional loging attempts, etc? Those can produce these kinds of
lock-ups, in my experience. If you have a high request timeout setting,
conditions causing the race will not resolve themselves and the server will
eventually stop responding.

In many early CFML libraries, before OOP practices were widely implemented
in CFML, locks were used quite a bit, so it’s possible that, if this has a
legacy code base, there’s a old third party library that’s the culprit.

Jon

Ty for the thought, Jon, but since the application begins with a login
page, the PCI scan is unable to access anything of our code except that.
I’ve scanned the errors caused by the scan in our application log and you
can see attempt-by-attempt these categories:

page not found
…has no remote function with name…
The parameter returnTo to function logIn is required but was not passed in
invalid url [empty string] for attribute url
component [controllers.LocalizationController] has no remote function with
name

I’m sure that’s not exhaustive but it certainly covers that last 200+ error
lines. Looks like they can’t get in to our app?On 10 November 2015 at 18:59, Jon Clausen <@Jon_Clausen> wrote:

fusionreactor installed, server restarted and a new PCI scan initiated.

Picture “Screenshot from 2015-11-10 18-13-50.png” was caught as the freeze
started, the server in fact did respond, just very slowly. A page of our
application without CSS could be loaded to a browser. “Screenshot from
2015-11-10 18-14-06.png” contains the detailed lower stats from the page.
Now I hope I’m dumping something informative here lol

After about 10 minutes only this can be loaded:

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /
https://demo.manifest.info/
.

Reason: Error reading from remote serverOn Tuesday, 10 November 2015 09:39:17 UTC, Mark Drew wrote:

I would second the use of FusionReactor in this instance by the way. You
will be able to monitor what is going on a lot better and see what requests
are making the server inactive. It could be buffer overflow tests that the
PCI company is running for example.

Without this you are just taking stabs I the dark.

If not. Try Newrelic, although the FR is much better.

I am not at my desk (but will be later to copy the setting) but you also
want to do heap dump on out of memory errors (google it) setting in your
setenv.sh to see what was in memory. It might help if you are actually
crashing the server.

Mark Drew


Apache/2.2.15 (CentOS) Server at myserver.com Port 443

fusionreactor still continues to update, however, and I can see memory
usage (top right graph) slowly increasing then dropping, presumably as
garbage collection is performed. The PCI scan fails still but only on one
item: “CGI Generic Cross-Site Scripting” (grrr why can’t I turn off
italics?)

Those that have chimed in already are much better with the JVM, Tomcat and Lucee server than I am, but the screenshots that I’m seeing (and what’s described in the rest of this thread) suggest that it’s not a JVM or Tomcat issue, but is rather something in the application, itself, that’s causing a race condition. Are there any writes to the application scope or locks which take place every request or I/O actions such as logging that fire on errors which might have permissions issues which trigger additional errors, additional loging attempts, etc? Those can produce these kinds of lock-ups, in my experience. If you have a high request timeout setting, conditions causing the race will not resolve themselves and the server will eventually stop responding.

In many early CFML libraries, before OOP practices were widely implemented in CFML, locks were used quite a bit, so it’s possible that, if this has a legacy code base, there’s a old third party library that’s the culprit.

Jon On November 10, 2015 at 1:48:29 PM, Tim Clarke (@Tim_Clarke) wrote:

On Tuesday, 10 November 2015 09:39:17 UTC, Mark Drew wrote:
I would second the use of FusionReactor in this instance by the way. You will be able to monitor what is going on a lot better and see what requests are making the server inactive. It could be buffer overflow tests that the PCI company is running for example.

Without this you are just taking stabs I the dark.

If not. Try Newrelic, although the FR is much better.

I am not at my desk (but will be later to copy the setting) but you also want to do heap dump on out of memory errors (google it) setting in your setenv.sh to see what was in memory. It might help if you are actually crashing the server.

Mark Drew

fusionreactor installed, server restarted and a new PCI scan initiated.

Picture “Screenshot from 2015-11-10 18-13-50.png” was caught as the freeze started, the server in fact did respond, just very slowly. A page of our application without CSS could be loaded to a browser. “Screenshot from 2015-11-10 18-14-06.png” contains the detailed lower stats from the page. Now I hope I’m dumping something informative here lol

After about 10 minutes only this can be loaded:

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /.

Reason: Error reading from remote server

Apache/2.2.15 (CentOS) Server at myserver.com Port 443

fusionreactor still continues to update, however, and I can see memory usage (top right graph) slowly increasing then dropping, presumably as garbage collection is performed. The PCI scan fails still but only on one item: “CGI Generic Cross-Site Scripting” (grrr why can’t I turn off italics?)

Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html

You received this message because you are subscribed to the Google Groups “Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/c08e9dc9-c9b5-4dbe-825b-439cc7e755d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Tim,

That’s a start,then. First off, I would set a default value for
“returnTo”, so that login redirects don’t throw 500’s. Secondly, a lot of
those PCI URLS being passed are for other language file extensions. Do you
have those being handled by the web server, or is Lucee forced to handle
those as well? You can very easily handle those in your rewrite rules, if
you have them.

Amended returnTo parameter.

apache blocks in place (I’m really unhappy with this kind of blacklisting -
I’ve just stabbed at a broad range that leaves our service happily doing
its thing but blocks a high number of intrustions):

RewriteEngine On
RewriteRule ^/amazon - [F]
RewriteRule ^/biz - [F]
RewriteRule ^/certsrv - [F]
RewriteRule ^/CFIDE - [F]
RewriteRule ^/cgi - [F]
RewriteRule ^/dev - [F]
RewriteRule ^/fusetalk - [F]
RewriteRule ^/flatnuke - [F]
RewriteRule ^/flex - [F]
RewriteRule ^/forum - [F]
RewriteRule ^/guestbook - [F]
RewriteRule ^/gbook - [F]
RewriteRule ^/gb - [F]
RewriteRule ^/idealbb - [F]
RewriteRule ^/ifx - [F]
RewriteRule ^/insertorder.cfm - [F]
RewriteRule ^/jffnms - [F]
RewriteRule ^/officescan - [F]
RewriteRule ^/php - [F]
RewriteRule ^/railo - [F]
RewriteRule ^/texis.exe - [F]
RewriteRule ^/script - [F]
RewriteRule ^/shop - [F]
RewriteRule ^/sitebuilder - [F]
RewriteRule ^/store - [F]
RewriteRule ^/viewcvs.cgi - [F]
RewriteRule ^/w - [F]
RewriteRule ^/x - [F]
RewriteRule ^/y - [F]
RewriteRule ^/z - [F]

Since your exception log i/o actions are firing on 500’s, rather than the
web server’s log appender, it’s best to deal with those 404’s at the web
server level and save yourself the expense on the app server. With 18,000
404’s in a short period of time, that will take an instant load off of your
application server.

Jon

Am running PCI test again now.On Tuesday, 10 November 2015 19:16:14 UTC, Jon Clausen wrote:

Yesyes! The service now stays up under the PCI test loading.

Sincerest props to Jon Clausen. On to the PCI test results. Oo-er… :/On Thursday, 12 November 2015 16:09:09 UTC, Gert Franz wrote:

Hey Tim,

Any update on this? Did it work now?

Sincerely
Gert Franz

Sweet! Glad I could help. I think 99.9% of all issues I’ve ever had with an ACF, Railo, or Lucee server becoming unresponsive have had something to do with exception handling and/or log files. :slight_smile:

Yesyes! The service now stays up under the PCI test loading.

Sincerest props to Jon Clausen. On to the PCI test results. Oo-er… :/–
Love Lucee? Become a supporter and be part of the Lucee project today! - http://lucee.org/supporters/become-a-supporter.html

You received this message because you are subscribed to the Google Groups “Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/1b562ac8-b581-43be-8024-d00c63588d30%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.