Lucee 5 crashing (Exact issue happening in 4.5 also)

I could really use some help on this…

I have been plagued for a long time (years) with my services just stopping
taking requests. It have gone as far as writing a service monitor that will
restart my app when it become unresponsive. Recently I upgraded to Lucee 5
hoping things would be better as I cannot find anything in my code that
could be causing this (but most likely it is the code). I have
fusionreactor installed and once the service became unresponsive I started
my investigation. It turned out that all the threads were in a “blocked”
state. I have found that 5-7% of my requests get the error below.

Unlike other errors in my logs when it is a .cfm or .cfc file from my code,
I can see that in the error log. In this case it appears to be something
either in jetty or lucee getting concurrent modification errors then never
releasing the thread block.

java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429)
at java.util.HashMap$KeyIterator.next(HashMap.java:1453)
at java.util.Collections$3.nextElement(Collections.java:5216)
at
lucee.runtime.net.http.HTTPServletRequestWrap.disconnect(HTTPServletRequestWrap.java:364)
at lucee.runtime.PageContextImpl.release(PageContextImpl.java:550)
at
lucee.runtime.CFMLFactoryImpl.releaseLuceePageContext(CFMLFactoryImpl.java:199)
at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:852)
at
lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:103)
at lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at
com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doNext(FusionReactorRequestHandler.java:806)
at
com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doHttpServletRequest(FusionReactorRequestHandler.java:339)
at
com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doFusionRequest(FusionReactorRequestHandler.java:215)
at
com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.handle(FusionReactorRequestHandler.java:839)
at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at
com.intergral.fusionreactor.agent.filter.FusionReactorStaticServlet.service(FusionReactorStaticServlet.java:50)
at
com.intergral.fusionreactor.agent.pointcuts.ServletPointCut$1.invoke(ServletPointCut.java:41)
at javax.servlet.http.HttpServlet.service(HttpServlet.java)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:245)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
at
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)

Hi Lyle,

I have similar issues, see
https://luceeserver.atlassian.net/browse/LDEV-947

I added a comment with your stacktrace.

-HarryVon: lucee@googlegroups.com [mailto:lucee@googlegroups.com] Im Auftrag von Lyle Karstensen
Gesendet: Montag, 1. August 2016 02:42
An: Lucee lucee@googlegroups.com
Betreff: [Lucee] Lucee 5 crashing (Exact issue happening in 4.5 also)

I could really use some help on this…

I have been plagued for a long time (years) with my services just stopping taking requests. It have gone as far as writing a service monitor that will restart my app when it become unresponsive. Recently I upgraded to Lucee 5 hoping things would be better as I cannot find anything in my code that could be causing this (but most likely it is the code). I have fusionreactor installed and once the service became unresponsive I started my investigation. It turned out that all the threads were in a “blocked” state. I have found that 5-7% of my requests get the error below.

Unlike other errors in my logs when it is a .cfm or .cfc file from my code, I can see that in the error log. In this case it appears to be something either in jetty or lucee getting concurrent modification errors then never releasing the thread block.

java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429)
at java.util.HashMap$KeyIterator.next(HashMap.java:1453)
at java.util.Collections$3.nextElement(Collections.java:5216)
at lucee.runtime.net.http.HTTPServletRequestWrap.disconnect(HTTPServletRequestWrap.java:364)
at lucee.runtime.PageContextImpl.release(PageContextImpl.java:550)
at lucee.runtime.CFMLFactoryImpl.releaseLuceePageContext(CFMLFactoryImpl.java:199)
at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:852)
at lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:103)
at lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doNext(FusionReactorRequestHandler.java:806)
at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doHttpServletRequest(FusionReactorRequestHandler.java:339)
at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doFusionRequest(FusionReactorRequestHandler.java:215)
at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.handle(FusionReactorRequestHandler.java:839)
at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.intergral.fusionreactor.agent.filter.FusionReactorStaticServlet.service(FusionReactorStaticServlet.java:50)
at com.intergral.fusionreactor.agent.pointcuts.ServletPointCut$1.invoke(ServletPointCut.java:41)
at javax.servlet.http.HttpServlet.service(HttpServlet.java)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:245)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)

Get 10% off of the regular price for this years CFCamp in Munich, Germany (Oct. 20th & 21st) with the Lucee discount code Lucee@cfcamp. 189€ instead of 210€. Visit CFCamp 2016

You received this message because you are subscribed to the Google Groups “Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+unsubscribe@googlegroups.commailto:lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.commailto:lucee@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/3be860a6-a5aa-452b-9dda-691f733db306%40googlegroups.comhttps://groups.google.com/d/msgid/lucee/3be860a6-a5aa-452b-9dda-691f733db306%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

The problem occurs for me only when multiple requests occur in the same millisecond, e.g. for very fast includes.
I guess could be a temporary (but very ugly) fix.

-HarryVon: lucee@googlegroups.com [mailto:lucee@googlegroups.com] Im Auftrag von Lyle Karstensen
Gesendet: Montag, 1. August 2016 17:07
An: Lucee lucee@googlegroups.com
Betreff: Re: [Lucee] Lucee 5 crashing (Exact issue happening in 4.5 also)

This all starts to make sense. My framework uses for every request.

I wonder if using will be a temporary fix?

On Monday, August 1, 2016 at 12:39:03 AM UTC-7, Harry Klein wrote:
Hi Lyle,

I have similar issues, see
https://luceeserver.atlassian.net/browse/LDEV-947

I added a comment with your stacktrace.

-Harry

Von: lu...@googlegroups.com<javascript:> [mailto:lu...@googlegroups.com<javascript:>] Im Auftrag von Lyle Karstensen
Gesendet: Montag, 1. August 2016 02:42
An: Lucee <lu...@googlegroups.com<javascript:>>
Betreff: [Lucee] Lucee 5 crashing (Exact issue happening in 4.5 also)

I could really use some help on this…

I have been plagued for a long time (years) with my services just stopping taking requests. It have gone as far as writing a service monitor that will restart my app when it become unresponsive. Recently I upgraded to Lucee 5 hoping things would be better as I cannot find anything in my code that could be causing this (but most likely it is the code). I have fusionreactor installed and once the service became unresponsive I started my investigation. It turned out that all the threads were in a “blocked” state. I have found that 5-7% of my requests get the error below.

Unlike other errors in my logs when it is a .cfm or .cfc file from my code, I can see that in the error log. In this case it appears to be something either in jetty or lucee getting concurrent modification errors then never releasing the thread block.

java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429)
at java.util.HashMap$KeyIterator.next(HashMap.java:1453)
at java.util.Collections$3.nextElement(Collections.java:5216)
at lucee.runtime.net.http.HTTPServletRequestWrap.disconnect(HTTPServletRequestWrap.java:364)
at lucee.runtime.PageContextImpl.release(PageContextImpl.java:550)
at lucee.runtime.CFMLFactoryImpl.releaseLuceePageContext(CFMLFactoryImpl.java:199)
at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:852)
at lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:103)
at lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doNext(FusionReactorRequestHandler.java:806)
at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doHttpServletRequest(FusionReactorRequestHandler.java:339)
at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doFusionRequest(FusionReactorRequestHandler.java:215)
at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.handle(FusionReactorRequestHandler.java:839)
at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.intergral.fusionreactor.agent.filter.FusionReactorStaticServlet.service(FusionReactorStaticServlet.java:50)
at com.intergral.fusionreactor.agent.pointcuts.ServletPointCut$1.invoke(ServletPointCut.java:41)
at javax.servlet.http.HttpServlet.service(HttpServlet.java)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:245)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)

Get 10% off of the regular price for this years CFCamp in Munich, Germany (Oct. 20th & 21st) with the Lucee discount code Lucee@cfcamp. 189€ instead of 210€. Visit CFCamp 2016

You received this message because you are subscribed to the Google Groups “Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com<javascript:>.
To post to this group, send email to lu...@googlegroups.com<javascript:>.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/3be860a6-a5aa-452b-9dda-691f733db306%40googlegroups.comhttps://groups.google.com/d/msgid/lucee/3be860a6-a5aa-452b-9dda-691f733db306%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

Get 10% off of the regular price for this years CFCamp in Munich, Germany (Oct. 20th & 21st) with the Lucee discount code Lucee@cfcamp. 189€ instead of 210€. Visit CFCamp 2016

You received this message because you are subscribed to the Google Groups “Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+unsubscribe@googlegroups.commailto:lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.commailto:lucee@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/d1489d56-a219-4624-98e5-f7ab538a87e0%40googlegroups.comhttps://groups.google.com/d/msgid/lucee/d1489d56-a219-4624-98e5-f7ab538a87e0%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

This all starts to make sense. My framework uses
for every request.

I wonder if using will be a temporary fix?On Monday, August 1, 2016 at 12:39:03 AM UTC-7, Harry Klein wrote:

Hi Lyle,

I have similar issues, see

[LDEV-947] - Lucee

I added a comment with your stacktrace.

-Harry

Von: lu...@googlegroups.com <javascript:> [mailto:lu...@googlegroups.com
<javascript:>] *Im Auftrag von *Lyle Karstensen
Gesendet: Montag, 1. August 2016 02:42
An: Lucee <lu...@googlegroups.com <javascript:>>
Betreff: [Lucee] Lucee 5 crashing (Exact issue happening in 4.5 also)

I could really use some help on this…

I have been plagued for a long time (years) with my services just stopping
taking requests. It have gone as far as writing a service monitor that will
restart my app when it become unresponsive. Recently I upgraded to Lucee 5
hoping things would be better as I cannot find anything in my code that
could be causing this (but most likely it is the code). I have
fusionreactor installed and once the service became unresponsive I started
my investigation. It turned out that all the threads were in a “blocked”
state. I have found that 5-7% of my requests get the error below.

Unlike other errors in my logs when it is a .cfm or .cfc file from my
code, I can see that in the error log. In this case it appears to be
something either in jetty or lucee getting concurrent modification errors
then never releasing the thread block.

java.util.ConcurrentModificationException

        at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429)

        at java.util.HashMap$KeyIterator.next(HashMap.java:1453)

        at java.util.Collections$3.nextElement(Collections.java:5216)

        at 

lucee.runtime.net.http.HTTPServletRequestWrap.disconnect(HTTPServletRequestWrap.java:364)

        at 

lucee.runtime.PageContextImpl.release(PageContextImpl.java:550)

        at 

lucee.runtime.CFMLFactoryImpl.releaseLuceePageContext(CFMLFactoryImpl.java:199)

        at 

lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:852)

        at 

lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:103)

        at 

lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:62)

        at 

javax.servlet.http.HttpServlet.service(HttpServlet.java:848)

        at 

com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doNext(FusionReactorRequestHandler.java:806)

        at 

com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doHttpServletRequest(FusionReactorRequestHandler.java:339)

        at 

com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doFusionRequest(FusionReactorRequestHandler.java:215)

        at 

com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.handle(FusionReactorRequestHandler.java:839)

        at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown 

Source)

        at 

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:483)

        at 

com.intergral.fusionreactor.agent.filter.FusionReactorStaticServlet.service(FusionReactorStaticServlet.java:50)

        at 

com.intergral.fusionreactor.agent.pointcuts.ServletPointCut$1.invoke(ServletPointCut.java:41)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java)

        at 

org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)

        at 

org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)

        at 

org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)

        at 

org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)

        at 

org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)

        at 

org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)

        at 

org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)

        at 

org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)

        at 

org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)

        at 

org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)

        at 

org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:245)

        at 

org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)

        at 

org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)

        at org.eclipse.jetty.server.Server.handle(Server.java:370)

        at 

org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)

        at 

org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)

        at 

org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)

        at 

org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)

        at 

org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)

        at 

org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)

        at 

org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)

        at 

org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)

        at 

org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)

        at 

org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)

        at java.lang.Thread.run(Thread.java:745)


Get 10% off of the regular price for this years CFCamp in Munich, Germany
(Oct. 20th & 21st) with the Lucee discount code Lucee@cfcamp. 189€ instead
of 210€. Visit CFCamp 2016

You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+un...@googlegroups.com <javascript:>.
To post to this group, send email to lu...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/3be860a6-a5aa-452b-9dda-691f733db306%40googlegroups.com
https://groups.google.com/d/msgid/lucee/3be860a6-a5aa-452b-9dda-691f733db306%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

BTW… THANKS…On Monday, August 1, 2016 at 12:39:03 AM UTC-7, Harry Klein wrote:

Hi Lyle,

I have similar issues, see

[LDEV-947] - Lucee

I added a comment with your stacktrace.

-Harry

Von: lu...@googlegroups.com <javascript:> [mailto:lu...@googlegroups.com
<javascript:>] *Im Auftrag von *Lyle Karstensen
Gesendet: Montag, 1. August 2016 02:42
An: Lucee <lu...@googlegroups.com <javascript:>>
Betreff: [Lucee] Lucee 5 crashing (Exact issue happening in 4.5 also)

I could really use some help on this…

I have been plagued for a long time (years) with my services just stopping
taking requests. It have gone as far as writing a service monitor that will
restart my app when it become unresponsive. Recently I upgraded to Lucee 5
hoping things would be better as I cannot find anything in my code that
could be causing this (but most likely it is the code). I have
fusionreactor installed and once the service became unresponsive I started
my investigation. It turned out that all the threads were in a “blocked”
state. I have found that 5-7% of my requests get the error below.

Unlike other errors in my logs when it is a .cfm or .cfc file from my
code, I can see that in the error log. In this case it appears to be
something either in jetty or lucee getting concurrent modification errors
then never releasing the thread block.

java.util.ConcurrentModificationException

        at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429)

        at java.util.HashMap$KeyIterator.next(HashMap.java:1453)

        at java.util.Collections$3.nextElement(Collections.java:5216)

        at 

lucee.runtime.net.http.HTTPServletRequestWrap.disconnect(HTTPServletRequestWrap.java:364)

        at 

lucee.runtime.PageContextImpl.release(PageContextImpl.java:550)

        at 

lucee.runtime.CFMLFactoryImpl.releaseLuceePageContext(CFMLFactoryImpl.java:199)

        at 

lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:852)

        at 

lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:103)

        at 

lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:62)

        at 

javax.servlet.http.HttpServlet.service(HttpServlet.java:848)

        at 

com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doNext(FusionReactorRequestHandler.java:806)

        at 

com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doHttpServletRequest(FusionReactorRequestHandler.java:339)

        at 

com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doFusionRequest(FusionReactorRequestHandler.java:215)

        at 

com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.handle(FusionReactorRequestHandler.java:839)

        at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown 

Source)

        at 

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:483)

        at 

com.intergral.fusionreactor.agent.filter.FusionReactorStaticServlet.service(FusionReactorStaticServlet.java:50)

        at 

com.intergral.fusionreactor.agent.pointcuts.ServletPointCut$1.invoke(ServletPointCut.java:41)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java)

        at 

org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)

        at 

org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)

        at 

org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)

        at 

org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)

        at 

org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)

        at 

org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)

        at 

org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)

        at 

org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)

        at 

org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)

        at 

org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)

        at 

org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:245)

        at 

org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)

        at 

org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)

        at org.eclipse.jetty.server.Server.handle(Server.java:370)

        at 

org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)

        at 

org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)

        at 

org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)

        at 

org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)

        at 

org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)

        at 

org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)

        at 

org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)

        at 

org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)

        at 

org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)

        at 

org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)

        at java.lang.Thread.run(Thread.java:745)


Get 10% off of the regular price for this years CFCamp in Munich, Germany
(Oct. 20th & 21st) with the Lucee discount code Lucee@cfcamp. 189€ instead
of 210€. Visit CFCamp 2016

You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+un...@googlegroups.com <javascript:>.
To post to this group, send email to lu...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/3be860a6-a5aa-452b-9dda-691f733db306%40googlegroups.com
https://groups.google.com/d/msgid/lucee/3be860a6-a5aa-452b-9dda-691f733db306%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.