Lucee Performance/caching

andrew · May 11, 2016, 4:41pm

@brad - would that be this ticket:

https://luceeserver.atlassian.net/browse/LDEV-638

Kind regards,

Andrew
about.me http://about.me/andrew_dixon - mso http://www.mso.net - Lucee
Association Member http://lucee.orgOn 11 May 2016 at 17:40, Scott Conklin <@Scott_Conklin> wrote:

@andrew
sure…

OS
*Windows Server 2012 R2 (6.3) 64bit *
8 GB RAM

Servlet Container
*Apache Tomcat/8.0.28 *

Java
*1.8.0_77 (Oracle Corporation) 64bit *

Architecture
*64bit *

Database
MS SQL Server 2014

Db Driver
jTDS Type 4 JDBC Driver for MS SQL Server

Java args:

-Xms4096m -Xmx4096m

On Monday, May 9, 2016 at 5:30:38 PM UTC-5, Scott Conklin wrote:

On a recent upgrade from ACF to Lucee we have noticed a situation where
performance becomes an issue that we did not have on ACF.
We notice that under normal operations, our sites run very fast, faster
than they did on an equivalent machine running ACF 9.01.

Our application architecture consists of two parts, and application
builder and the applications built by the builder.
The application builder reads our standard templates (about 160 or so)
into memory and replaces various mnemonics with client-customized snippets
of codes and writes the .cfm pages to the webroot for each of our 55 sites
running on our Lucee 4.5 installation. (win 2012)
The builder can loop over all 55 clients and rewrite all the pages of
those sites in about 4 or 5 minutes.

Over the years, we have always done this throughout regular business
hours without any issue…sometimes several times a day.

In Lucee, we have found that while the application builder is running,
access to the sites comes to almost a screeching halt …taking sometimes
20, 30 or 40 seconds for pages to complete. and on some occasions the
pages timeout … with (sometimes) the following errors appearing the
log files.

java.lang.ThreadDeath
at java.lang.Thread.stop(Unknown Source)
at lucee.commons.io.StopThread.run(SystemUtil.java:1091)
Exception in thread “Thread-53889” java.lang.NullPointerException
at lucee.commons.io.StopThread.run(SystemUtil.java:1068)

Exception in thread “ajp-nio-8009-exec-14”
java.lang.IllegalMonitorStateException
at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(Unknown
Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(Unknown
Source)
at java.util.concurrent.locks.ReentrantLock.unlock(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)

It does recover eventually… maybe 3 or 4 minutes after the application
builder completes, but not before someone complains of performance issues
by our clients.

I have tried all 3 settings for *Inspect Templates (CFM/CFC) *setting
(always, once, never)
The problem occurs exactly the same for always and once.
Obviously, when set to never it the issue goes away but as soon as I
click “Clear page cache Pool” to make the latest rebuild changes
persistent, the issue shows itself . This leads me to speculate that it
must have something to do with the algorithms used to populate/refresh the
cache or the comparison routine for large-scale wholesale changes in the
file timestamps?

Hate to bring up the fact that it the behavior was different in ACF but
this is something we need to be able to do a regular basis without
interference to our clients
ability to work.

Has anyone else ever had such an experience?
Any thoughts on why this happens? and what we can do about it?

–
Love Lucee? Become a supporter and be part of the Lucee project today! -
http://lucee.org/supporters/become-a-supporter.html

You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/d90975ec-0be0-488c-a06d-bec5ecc52a0d%40googlegroups.com
https://groups.google.com/d/msgid/lucee/d90975ec-0be0-488c-a06d-bec5ecc52a0d%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

Scott_Conklin · May 11, 2016, 5:22pm

@mark-
Another quick question…

When I follow your instructions for modifying the Setenv.sh

I end up with this

CATALINA_OPTS=“-Xms256m -Xmx512m”;

CATALINA_OPTS=“-Xms256m -Xmx512m -javaagent:lib/lucee-inst.jar”;

this is what you meant? correct?
curious what these xms and xmx setting here are when i have
these in my java args for Lucee

-Xms4096m -Xmx4096m

I am assuming that i will need to cycle Lucee to make this change
persist?
(is so, I may have to wait until after business hours to try this out)

Scott_Conklin · May 16, 2016, 9:23pm

sorry for talking so long to respond on my findings on this issue. I was
out last week.

I am happy to report that implementing the suggestion made by @brad ;
(removing this -javaagent:C:\lucee\tomcat\lib\lucee-inst.jar from the java
arguments)
completely solved this issue…
All through the application building process the site loads times remain
quick and unaffected by rewriting all the files of all the websites to
their respective webroots.

AND under Lucee, the application builder itself is now running faster than
I have ever seen it averaging under a second for each venue (website) (took
about 3 seconds under ACF)

Thanks so much for everyone’s input on this.On Monday, May 9, 2016 at 5:30:38 PM UTC-5, Scott Conklin wrote:

On a recent upgrade from ACF to Lucee we have noticed a situation where
performance becomes an issue that we did not have on ACF.
We notice that under normal operations, our sites run very fast, faster
than they did on an equivalent machine running ACF 9.01.

Our application architecture consists of two parts, and application
builder and the applications built by the builder.
The application builder reads our standard templates (about 160 or so)
into memory and replaces various mnemonics with client-customized snippets
of codes and writes the .cfm pages to the webroot for each of our 55 sites
running on our Lucee 4.5 installation. (win 2012)
The builder can loop over all 55 clients and rewrite all the pages of
those sites in about 4 or 5 minutes.

Over the years, we have always done this throughout regular business hours
without any issue…sometimes several times a day.

In Lucee, we have found that while the application builder is running,
access to the sites comes to almost a screeching halt …taking sometimes
20, 30 or 40 seconds for pages to complete. and on some occasions the
pages timeout … with (sometimes) the following errors appearing the
log files.

java.lang.ThreadDeath
at java.lang.Thread.stop(Unknown Source)
at lucee.commons.io.StopThread.run(SystemUtil.java:1091)
Exception in thread “Thread-53889” java.lang.NullPointerException
at lucee.commons.io.StopThread.run(SystemUtil.java:1068)

Exception in thread “ajp-nio-8009-exec-14”
java.lang.IllegalMonitorStateException
at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(Unknown
Source)
at java.util.concurrent.locks.ReentrantLock.unlock(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)

It does recover eventually… maybe 3 or 4 minutes after the application
builder completes, but not before someone complains of performance issues
by our clients.

I have tried all 3 settings for *Inspect Templates (CFM/CFC) *setting
(always, once, never)
The problem occurs exactly the same for always and once.
Obviously, when set to never it the issue goes away but as soon as I
click “Clear page cache Pool” to make the latest rebuild changes
persistent, the issue shows itself . This leads me to speculate that it
must have something to do with the algorithms used to populate/refresh the
cache or the comparison routine for large-scale wholesale changes in the
file timestamps?

Hate to bring up the fact that it the behavior was different in ACF but
this is something we need to be able to do a regular basis without
interference to our clients
ability to work.

Has anyone else ever had such an experience?
Any thoughts on why this happens? and what we can do about it?

Brad_Wood · May 21, 2016, 3:31pm

It allows for Lucee to rewrite the bytecode inside a class file on the fly
instead of just creating new class files when a file changes. This keeps
from using up class files in permgen. The java agent is optional though
and Lucee will work fine without it. I’ve actually hit a couple bugs
recently related to compilation errors and performance issues in some
scenarios with the java agent so I remove it sometimes anyway. With Java
8, there’s not a concern with running out of permgen space anyway. Micha
can chime in if he wants with additional info about that jar.

Thanks!

~Brad

ColdBox Platform Evangelist
*Ortus Solutions, Corp *

E-mail: brad@coldbox.org
ColdBox Platform: http://www.coldbox.org
Blog: http://www.codersrevolution.comOn Sat, May 21, 2016 at 10:19 AM, David Eurenius < @David_Eurenius> wrote:

What exactly does the -javaagent:C:\lucee\tomcat\lib\lucee-inst.jar do?
In which scenarios would it be safe to remove the argument?

–David

On Monday, May 16, 2016 at 11:23:41 PM UTC+2, Scott Conklin wrote:

sorry for talking so long to respond on my findings on this issue. I was
out last week.

I am happy to report that implementing the suggestion made by @brad ;
(removing this -javaagent:C:\lucee\tomcat\lib\lucee-inst.jar from the java
arguments)
completely solved this issue…
All through the application building process the site loads times remain
quick and unaffected by rewriting all the files of all the websites to
their respective webroots.

AND under Lucee, the application builder itself is now running faster
than I have ever seen it averaging under a second for each venue (website)
(took about 3 seconds under ACF)

Thanks so much for everyone’s input on this.

On Monday, May 9, 2016 at 5:30:38 PM UTC-5, Scott Conklin wrote:

On a recent upgrade from ACF to Lucee we have noticed a situation where
performance becomes an issue that we did not have on ACF.
We notice that under normal operations, our sites run very fast, faster
than they did on an equivalent machine running ACF 9.01.

Our application architecture consists of two parts, and application
builder and the applications built by the builder.
The application builder reads our standard templates (about 160 or so)
into memory and replaces various mnemonics with client-customized snippets
of codes and writes the .cfm pages to the webroot for each of our 55 sites
running on our Lucee 4.5 installation. (win 2012)
The builder can loop over all 55 clients and rewrite all the pages of
those sites in about 4 or 5 minutes.

Over the years, we have always done this throughout regular business
hours without any issue…sometimes several times a day.

In Lucee, we have found that while the application builder is running,
access to the sites comes to almost a screeching halt …taking sometimes
20, 30 or 40 seconds for pages to complete. and on some occasions the
pages timeout … with (sometimes) the following errors appearing the
log files.

java.lang.ThreadDeath
at java.lang.Thread.stop(Unknown Source)
at lucee.commons.io.StopThread.run(SystemUtil.java:1091)
Exception in thread “Thread-53889” java.lang.NullPointerException
at lucee.commons.io.StopThread.run(SystemUtil.java:1068)

Exception in thread “ajp-nio-8009-exec-14”
java.lang.IllegalMonitorStateException
at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(Unknown
Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(Unknown
Source)
at java.util.concurrent.locks.ReentrantLock.unlock(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)

It does recover eventually… maybe 3 or 4 minutes after the application
builder completes, but not before someone complains of performance issues
by our clients.

I have tried all 3 settings for *Inspect Templates (CFM/CFC) *setting
(always, once, never)
The problem occurs exactly the same for always and once.
Obviously, when set to never it the issue goes away but as soon as I
click “Clear page cache Pool” to make the latest rebuild changes
persistent, the issue shows itself . This leads me to speculate that it
must have something to do with the algorithms used to populate/refresh the
cache or the comparison routine for large-scale wholesale changes in the
file timestamps?

Hate to bring up the fact that it the behavior was different in ACF but
this is something we need to be able to do a regular basis without
interference to our clients
ability to work.

Has anyone else ever had such an experience?
Any thoughts on why this happens? and what we can do about it?

–
Love Lucee? Become a supporter and be part of the Lucee project today! -
http://lucee.org/supporters/become-a-supporter.html

You received this message because you are subscribed to a topic in the
Google Groups “Lucee” group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/lucee/56WHZlWc9rI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/b9b669bf-b44c-4b78-829d-aa317bbc8fdb%40googlegroups.com
https://groups.google.com/d/msgid/lucee/b9b669bf-b44c-4b78-829d-aa317bbc8fdb%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

David_Eurenius · May 21, 2016, 3:19pm

What exactly does the -javaagent:C:\lucee\tomcat\lib\lucee-inst.jar do?
In which scenarios would it be safe to remove the argument?

–DavidOn Monday, May 16, 2016 at 11:23:41 PM UTC+2, Scott Conklin wrote:

sorry for talking so long to respond on my findings on this issue. I was
out last week.

I am happy to report that implementing the suggestion made by @brad ;
(removing this -javaagent:C:\lucee\tomcat\lib\lucee-inst.jar from the java
arguments)
completely solved this issue…
All through the application building process the site loads times remain
quick and unaffected by rewriting all the files of all the websites to
their respective webroots.

AND under Lucee, the application builder itself is now running faster than
I have ever seen it averaging under a second for each venue (website) (took
about 3 seconds under ACF)

Thanks so much for everyone’s input on this.

On Monday, May 9, 2016 at 5:30:38 PM UTC-5, Scott Conklin wrote:

On a recent upgrade from ACF to Lucee we have noticed a situation where
performance becomes an issue that we did not have on ACF.
We notice that under normal operations, our sites run very fast, faster
than they did on an equivalent machine running ACF 9.01.

Our application architecture consists of two parts, and application
builder and the applications built by the builder.
The application builder reads our standard templates (about 160 or so)
into memory and replaces various mnemonics with client-customized snippets
of codes and writes the .cfm pages to the webroot for each of our 55 sites
running on our Lucee 4.5 installation. (win 2012)
The builder can loop over all 55 clients and rewrite all the pages of
those sites in about 4 or 5 minutes.

Over the years, we have always done this throughout regular business
hours without any issue…sometimes several times a day.

In Lucee, we have found that while the application builder is running,
access to the sites comes to almost a screeching halt …taking sometimes
20, 30 or 40 seconds for pages to complete. and on some occasions the
pages timeout … with (sometimes) the following errors appearing the
log files.

java.lang.ThreadDeath
at java.lang.Thread.stop(Unknown Source)
at lucee.commons.io.StopThread.run(SystemUtil.java:1091)
Exception in thread “Thread-53889” java.lang.NullPointerException
at lucee.commons.io.StopThread.run(SystemUtil.java:1068)

Exception in thread “ajp-nio-8009-exec-14”
java.lang.IllegalMonitorStateException
at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(Unknown
Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(Unknown
Source)
at java.util.concurrent.locks.ReentrantLock.unlock(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)

It does recover eventually… maybe 3 or 4 minutes after the application
builder completes, but not before someone complains of performance issues
by our clients.

I have tried all 3 settings for *Inspect Templates (CFM/CFC) *setting
(always, once, never)
The problem occurs exactly the same for always and once.
Obviously, when set to never it the issue goes away but as soon as I
click “Clear page cache Pool” to make the latest rebuild changes
persistent, the issue shows itself . This leads me to speculate that it
must have something to do with the algorithms used to populate/refresh the
cache or the comparison routine for large-scale wholesale changes in the
file timestamps?

Hate to bring up the fact that it the behavior was different in ACF but
this is something we need to be able to do a regular basis without
interference to our clients
ability to work.

Has anyone else ever had such an experience?
Any thoughts on why this happens? and what we can do about it?