Anyone else having major hanging issues with sessionCluster/clientCluster?

frinky · November 22, 2017, 10:32am

Hi all,

I recently upgraded a few clustered servers to Lucee 5.2.4.37 (latest stable a.f.a.i.k.), from 5.0.x. They are using a datasource for session and client scope storage, with “sessionCluster” and “clientCluster” enabled.
Since I did the Lucee upgrade, I am having major problems with requestTimeouts almost daily. If the situation occurs, all requests in a few minutes will timeout, with the stacktrace underneath. That means tens to hundreds of errors. 2 times, Tomcat needed a restart to overcome this (it kept going at 100% cpu).

Looking at the stacktrace, timeouts occur in

github.com

lucee/Lucee/blob/5.2.4/core/src/main/java/lucee/runtime/type/scope/storage/IKStorageScopeSupport.java#L213


      
          		try {
          			return getInstance(scope, handler, appName, name, pc,existing, log);
          		}
          		catch (PageException e) {}
          		return defaultValue;
          	}
          
          
	
          	public synchronized static boolean hasInstance(int scope, IKHandler handler, String appName, String name, PageContext pc) {
          		try {
          			if(Scope.SCOPE_SESSION==scope)		return handler.loadData(pc, appName,name, "session",Scope.SCOPE_SESSION, null)!=null;
          			else if(Scope.SCOPE_CLIENT==scope)	return handler.loadData(pc, appName,name, "client",Scope.SCOPE_CLIENT, null)!=null;
          			return false;
          		} 
          		catch (PageException e) {
          			return false;
          		}
          	}
          	
          	@Override
          	public void touchBeforeRequest(PageContext pc) {

which is a “synchronized” method, meaning (if I recall correctly), only one request can execute it at any given time.
I also found the stacktrace of the first erroring request, which was one step further, at

github.com

lucee/Lucee/blob/5.2.4/core/src/main/java/lucee/runtime/type/scope/storage/IKHandlerDatasource.java#L36


      
          
          
public class IKHandlerDatasource implements IKHandler {
          	
          	public static final String PREFIX = "cf";
          	
          	@Override
          	public IKStorageValue loadData(PageContext pc, String appName, String name, String strType,
          			int type, Log log) throws PageException {
          		ConfigImpl config = (ConfigImpl)ThreadLocalPageContext.getConfig(pc);
          		DatasourceConnectionPool pool = config.getDatasourceConnectionPool();
          		DatasourceConnection dc=pool.getDatasourceConnection(config,pc.getDataSource(name),null,null);
          		SQLExecutor executor=SQLExecutionFactory.getInstance(dc);
          		Query query;
          		
          		try {
          			if(!dc.getDatasource().isStorage()) 
          				throw new ApplicationException("storage usage for this datasource is disabled, you can enable this in the Lucee administrator.");
          			query = executor.select(config,pc.getCFID(),pc.getApplicationContext().getName(), dc, type,log, true);
          		} 
          		catch (SQLException se) {
          			throw Caster.toPageException(se);

That stacktrace is also underneath, and indicates the request is waiting for “DatasourceConnectionPool.getDatasourceConnection”.

Meaning, if the datasource is unavailable for a specific request, all other requests will hang untill a request timeout occurs.

Is anyone else having these issues? I already created a ticket for it, but no me-too’s yet

Kind regards, Paul Klinkenberg

Stacktrace of all error pages, except the first request:
lucee.runtime.exp.RequestTimeoutException: request /index.cfm (/var/www/clientname/www/index.cfm) has run into a timeout (50 seconds) and has been stopped. at lucee.runtime.type.scope.storage.IKStorageScopeSupport.hasInstance(IKStorageScopeSupport.java:213) at lucee.runtime.type.scope.ScopeContext.hasExistingCFSessionScope(ScopeContext.java:549) at lucee.runtime.type.scope.ScopeContext.hasExistingSessionScope(ScopeContext.java:487) at lucee.runtime.PageContextImpl.initApplicationContext(PageContextImpl.java:3030) at lucee.runtime.listener.ModernAppListener._onRequest(ModernAppListener.java:119) at lucee.runtime.listener.MixedAppListener.onRequest(MixedAppListener.java:43) at lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2405) at lucee.runtime.PageContextImpl._execute(PageContextImpl.java:2395) at lucee.runtime.PageContextImpl.executeCFML(PageContextImpl.java:2363) at lucee.runtime.engine.Request.exe(Request.java:46) at lucee.runtime.engine.CFMLEngineImpl._service(CFMLEngineImpl.java:1084) at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:1032) at lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:102) at lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:51) at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:193) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

** Stacktrace of the first request**
stop thread (66) because run into a timeout path: /var/www/myclientname/www/index.cfm (/var/www/myclientname/www/Application.cfc).;java.lang.Throwable;java.lang.Throwable at lucee.runtime.db.DatasourceConnectionPool.getDatasourceConnection(DatasourceConnectionPool.java:77) at lucee.runtime.type.scope.storage.IKHandlerDatasource.loadData(IKHandlerDatasource.java:36) at lucee.runtime.type.scope.storage.IKStorageScopeSupport.hasInstance(IKStorageScopeSupport.java:213) at lucee.runtime.type.scope.ScopeContext.hasExistingCFSessionScope(ScopeContext.java:549) at lucee.runtime.type.scope.ScopeContext.hasExistingSessionScope(ScopeContext.java:487) at lucee.runtime.PageContextImpl.initApplicationContext(PageContextImpl.java:3030) at lucee.runtime.listener.ModernAppListener._onRequest(ModernAppListener.java:119) at lucee.runtime.listener.MixedAppListener.onRequest(MixedAppListener.java:43) at lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2405) at lucee.runtime.PageContextImpl._execute(PageContextImpl.java:2395) at lucee.runtime.PageContextImpl.executeCFML(PageContextImpl.java:2363) at lucee.runtime.engine.Request.exe(Request.java:46) at lucee.runtime.engine.CFMLEngineImpl._service(CFMLEngineImpl.java:1084) at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:1032) at lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:102) at lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:51) at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:193) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

psarin · November 22, 2017, 10:52am

Although the scenario is different, the error looks the same as my issue surrounding DB connections not being closed in the DB pool:

https://lucee.daemonite.io/t/more-database-woes-with-newest-lucee-versions/2979/9

Michael_Sprague1 · November 22, 2017, 2:01pm

Yes, I have run into major issues with using database session storage since 5.2.x. I had some back-and-forth with the engineers about it, but could not pinpoint the exact issue in order to open a bug report. We had lockups and heap overflows. However, it all runs great on 5.1.x.

Apparently, something changed with the serialization of the session scope in one of the releases with 5.2.x. If the scope is large, it cannot handle it. Reducing the session timeout seemed to help a little, but was not a complete solution.

For now, we had to abandon using a database for shared sessions. It’s not ideal, but at least our websites stay up.

isapir · November 22, 2017, 10:21pm

Is there a ticket for that?

@IamSigmund Are you aware of such a regression?

sbleon · November 29, 2017, 4:41pm

I have found 5.2.4.37 to be very unstable. My site locks up (requests just hang) within a day on that version. Version 5.2.1.9 is very stable by comparison.

I’m not using clustering. I’m using memory storage for sessions. So, I think there may be a larger problem than this thread described. I added my #metoo to Lucee Development - Issues - Lucee anyway.

IamSigmund · November 30, 2017, 2:31am

Igal, we don’t have an official regression identified, but as we look into it I think the evidence it building. FYI to everyone watching this one, it will be a top priority in our end-of-year planning work. Stay tuned. @sbleon

Ivan_McAvinchey · December 4, 2017, 2:07pm

We had an issue with it and had to stop using it, but in our case the problem was there in Railo as well. I didn’t conclusively get to the bottom of it but I suspect in our case it came down to serialization failing on objects (i.e CFCs or Java objects) that were stored in the session scope.

We ended up “rolling our own”- our main concern was just auto-login rather than preserving an entire session, so storing a token in a cookie and matching that against a user account to recreate a login.

sbleon · December 13, 2017, 4:25pm

Any news on this, @IamSigmund? We’re now frozen on 5.2.1.9 until this issue gets resolved.

Redtopia · December 15, 2017, 5:55pm

I am also seeing hanging threads that never timeout which appear to be related to jdbc connections never timing out and hanging forever. I have to restart my server daily. I’m running Lucee 5.2.4.37 on Windows x64 and sql server.

I created a bug report here: [LDEV-1622] - Lucee

IamSigmund · December 17, 2017, 5:07pm

Hey Leon. No official plans to report on this one, yet, but, we have gotten similar reports from other users, so it’s very high on the priority list as we get ready to start 2018 sprints. Ideally, we’ll have this fixed in the next RC, which should be out end-of-January. More soon!

IamSigmund · December 17, 2017, 5:13pm

Sorry, I just realized this wasn’t very clear. I meant we hope to have the memory/session/etc. problems addressed by the next next RC. That is, the current RC is 5.2.5-RC, which will come out end-of-Dec./beginning-of-Jan. But that likely won’t contain any new fixes. The 5.2.6-RC (slated for end-of-January) would be the next RC where we could get this addressed.

sbleon · December 18, 2017, 7:28pm

How can we help to make sure the instability gets fixed? I know that it may
be hard to reproduce the problem, and so far I haven’t been able to get any
good info on the state of Lucee when it has become unresponsive. Any
suggestions on gathering more useful data?

Leon

IamSigmund · December 18, 2017, 9:11pm

Hi Leon. Thanks for the offer to help. I’ll ponder that. Creating valid test cases/environments is easily one of our biggest challenges, given the huge range of use cases that Lucee solves.

Nick_S · January 23, 2018, 1:39pm

We’re running into the same issue. Working on a new environment on AWS with clustered sessions and the behavior described in the original post happens within seconds of traffic hitting the backend on version 5.2.3.35.

I have saved the stacktraces in FR-CLOUD. There are about 100 threads waiting on the following request:


java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242)

java.io.File.exists(File.java:819)

sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1282)

sun.misc.URLClassPath.getResource(URLClassPath.java:239)

java.net.URLClassLoader$1.run(URLClassLoader.java:365)

java.net.URLClassLoader$1.run(URLClassLoader.java:362)

java.security.AccessController.doPrivileged(Native method)

java.net.URLClassLoader.findClass(URLClassLoader.java:361)

java.lang.ClassLoader.loadClass(ClassLoader.java:424)
- locked <0x1C357766> (a java.lang.Object)

java.lang.ClassLoader.loadClass(ClassLoader.java:357)

org.apache.felix.framework.Felix.loadBundleClass(Felix.java:1922)

org.apache.felix.framework.BundleImpl.loadClass(BundleImpl.java:959)

org.apache.felix.framework.Felix.loadClass(Felix.java:106)

lucee.runtime.osgi.OSGiUtil.loadClass(OSGiUtil.java:304)

lucee.commons.lang.ClassUtil$OSGiBasedClassLoading.loadClass(ClassUtil.java:844)

lucee.commons.lang.ClassUtil$OSGiBasedClassLoading.loadClass(ClassUtil.java:849)

lucee.commons.lang.ClassUtil._loadClass(ClassUtil.java:284)

lucee.commons.lang.ClassUtil.loadClass(ClassUtil.java:172)

lucee.transformer.library.ClassDefinitionImpl.getClazz(ClassDefinitionImpl.java:79)

lucee.runtime.cache.CacheConnectionImpl.<init>(CacheConnectionImpl.java:53)

lucee.runtime.listener.ModernApplicationContext.toCacheConnection(ModernApplicationContext.java:898)

lucee.runtime.listener.ModernApplicationContext.toCacheConnection(ModernApplicationContext.java:876)

lucee.runtime.listener.ModernApplicationContext._initCache(ModernApplicationContext.java:829)

lucee.runtime.listener.ModernApplicationContext._initCache(ModernApplicationContext.java:821)

lucee.runtime.listener.ModernApplicationContext.initCache(ModernApplicationContext.java:792)

lucee.runtime.listener.ModernApplicationContext.getCacheConnection(ModernApplicationContext.java:740)

lucee.runtime.PageContextImpl.getCacheConnection(PageContextImpl.java:3426)

lucee.runtime.cache.CacheUtil.getCacheConnection(CacheUtil.java:136)

lucee.runtime.type.scope.storage.IKHandlerCache.getCache(IKHandlerCache.java:90)

lucee.runtime.type.scope.storage.IKHandlerCache.loadData(IKHandlerCache.java:23)

lucee.runtime.type.scope.storage.IKStorageScopeSupport.hasInstance(IKStorageScopeSupport.java:213)
- locked <0x4E91E337> (a java.lang.Class)

lucee.runtime.type.scope.ScopeContext.hasExistingCFSessionScope(ScopeContext.java:557)

lucee.runtime.type.scope.ScopeContext.hasExistingSessionScope(ScopeContext.java:487)

lucee.runtime.PageContextImpl.initApplicationContext(PageContextImpl.java:3030)

lucee.runtime.listener.ModernAppListener._onRequest(ModernAppListener.java:119)

lucee.runtime.listener.MixedAppListener.onRequest(MixedAppListener.java:43)

lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2405)

lucee.runtime.PageContextImpl._execute(PageContextImpl.java:2395)

lucee.runtime.PageContextImpl.executeCFML(PageContextImpl.java:2363)

lucee.runtime.engine.Request.exe(Request.java:46)

lucee.runtime.engine.CFMLEngineImpl._service(CFMLEngineImpl.java:1048)

lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:996)

lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:102)

lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:51)

javax.servlet.http.HttpServlet.service(HttpServlet.java:729)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)

org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)

org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176)

org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)

org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)

org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:394)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)

sun.reflect.GeneratedMethodAccessor140.invoke(Unknown source)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:134)

com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doNext(FusionReactorRequestHandler.java:763)

com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doHttpServletRequest(FusionReactorRequestHandler.java:351)

com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doFusionRequest(FusionReactorRequestHandler.java:214)

com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.handle(FusionReactorRequestHandler.java:800)

com.intergral.fusionreactor.j2ee.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilter.java:36)

sun.reflect.GeneratedMethodAccessor137.invoke(Unknown source)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:71)

sun.reflect.GeneratedMethodAccessor136.invoke(Unknown source)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

com.intergral.fusionreactor.agent.filter.FusionReactorStaticFilter.doFilter(FusionReactorStaticFilter.java:54)

com.intergral.fusionreactor.agent.pointcuts.NewFilterChainPointCut$1.invoke(NewFilterChainPointCut.java:41)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java)

org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)

org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:94)

org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)

org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)

org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)

org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:620)

org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:676)

org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)

org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:502)

org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1132)

org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)

org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:2458)
- locked <0x1E0B1FD> (a org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

java.lang.Thread.run(Thread.java:748)

Locked ownable synchronizers: 
- java.util.concurrent.ThreadPoolExecutor$Worker@67852C19
- java.util.concurrent.locks.ReentrantLock$FairSync@1107D2CC

Here’s a stack trace from one of the hanging threads:

- waiting on <0x4E91E337> (a java.lang.Class)

lucee.runtime.type.scope.ScopeContext.hasExistingCFSessionScope(ScopeContext.java:557)

lucee.runtime.type.scope.ScopeContext.hasExistingSessionScope(ScopeContext.java:487)

lucee.runtime.PageContextImpl.initApplicationContext(PageContextImpl.java:3030)

lucee.runtime.listener.ModernAppListener._onRequest(ModernAppListener.java:119)

lucee.runtime.listener.MixedAppListener.onRequest(MixedAppListener.java:43)

lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2405)

lucee.runtime.PageContextImpl._execute(PageContextImpl.java:2395)

lucee.runtime.PageContextImpl.executeCFML(PageContextImpl.java:2363)

lucee.runtime.engine.Request.exe(Request.java:46)

lucee.runtime.engine.CFMLEngineImpl._service(CFMLEngineImpl.java:1048)

lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:996)

lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:102)

lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:51)

javax.servlet.http.HttpServlet.service(HttpServlet.java:729)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)

org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)

org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176)

org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)

org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)

org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:394)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)

sun.reflect.GeneratedMethodAccessor140.invoke(Unknown source)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:134)

com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doNext(FusionReactorRequestHandler.java:763)

com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doHttpServletRequest(FusionReactorRequestHandler.java:351)

com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doFusionRequest(FusionReactorRequestHandler.java:214)

com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.handle(FusionReactorRequestHandler.java:800)

com.intergral.fusionreactor.j2ee.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilter.java:36)

sun.reflect.GeneratedMethodAccessor137.invoke(Unknown source)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:71)

sun.reflect.GeneratedMethodAccessor136.invoke(Unknown source)

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

java.lang.reflect.Method.invoke(Method.java:498)

com.intergral.fusionreactor.agent.filter.FusionReactorStaticFilter.doFilter(FusionReactorStaticFilter.java:54)

com.intergral.fusionreactor.agent.pointcuts.NewFilterChainPointCut$1.invoke(NewFilterChainPointCut.java:41)

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java)

org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)

org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:94)

org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)

org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)

org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)

org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:620)

org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:676)

org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)

org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:502)

org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1132)

org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)

org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:2458)
- locked <0x5EEE6FB3> (a org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

java.lang.Thread.run(Thread.java:748)

Locked ownable synchronizers: 
- java.util.concurrent.ThreadPoolExecutor$Worker@2443C6C8
- java.util.concurrent.locks.ReentrantLock$FairSync@6B05B81F

Trying to downgrade now. If I can provide any other details let me know.

Thanks,
-Nick

micstriit · January 24, 2018, 7:25am

ATM it is not clear to me, if this is a single issue people experience or multiple that looks as they are related. there are 3 ways to solve this (#1 is the best way and #3 the worst).

reproduce the issue on a test system and then figuring out exactly which version cause the issue.
reproduce the issue and then figuring out what the problem causing this.
based on existing data (stacktraces) try to figuring out what could possible cause this.

for #1 and #2 i need to know if and how to reproduce this. So i will start with #3, if you have any further input please add to the ticket from Paul.
https://luceeserver.atlassian.net/projects/LDEV/issues/LDEV-1587
i will post further information in this ticket and if this lead to more than one issue i will point to other tickets in that ticket.