Hi,
Our servers crash when under load. It appears to be related to the DB
connection but I am unsure of this is caused by the DB or by Lucee / Tomcat.
We see a very large number of stack trace like this in our logs.
Thread.stop(Throwable) is not supported by this JVM and failed with
UnsupportedOperationException;java.lang.Throwable;java.lang.Throwable
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1782)
at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:4881)
at com.microsoft.sqlserver.jdbc.TDSReader.nextPacket(IOBuffer.java:4791)
at com.microsoft.sqlserver.jdbc.TDSReader.ensurePayload(IOBuffer.java:4767)
at com.microsoft.sqlserver.jdbc.TDSReader.peekTokenType(IOBuffer.java:4961)
at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:50)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1510)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.processResults(SQLServerStatement.java:1157)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.processBatch(SQLServerStatement.java:1145)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.processExecuteResults(SQLServerStatement.java:1137)
at
com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.processResponse(SQLServerPreparedStatement.java:357)
at com.microsoft.sqlserver.jdbc.TDSCommand.close(IOBuffer.java:5781)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.discardLastExecutionResults(SQLServerStatement.java:94)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.closeInternal(SQLServerStatement.java:584)
at
com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.closeInternal(SQLServerPreparedStatement.java:170)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.close(SQLServerStatement.java:596)
…
We have updated one of the servers to use the most recent patch for Lucee
and in this server we no longer see the above in the stderr log but we see
this in the stdout log.
java.lang.StackTraceElement[]{java.net.SocketInputStream.socketRead0(Native
Method)
,java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
,java.net.SocketInputStream.read(SocketInputStream.java:170)
,java.net.SocketInputStream.read(SocketInputStream.java:141)
,com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1782)
,com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:4881)
,com.microsoft.sqlserver.jdbc.TDSReader.nextPacket(IOBuffer.java:4791)
,com.microsoft.sqlserver.jdbc.TDSReader.ensurePayload(IOBuffer.java:4767)
,com.microsoft.sqlserver.jdbc.TDSReader.peekTokenType(IOBuffer.java:4961)
,com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:50)
,com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1510)
,com.microsoft.sqlserver.jdbc.SQLServerStatement.processResults(SQLServerStatement.java:1157)
,com.microsoft.sqlserver.jdbc.SQLServerStatement.processBatch(SQLServerStatement.java:1145)
,com.microsoft.sqlserver.jdbc.SQLServerStatement.processExecuteResults(SQLServerStatement.java:1137)
,com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.processResponse(SQLServerPreparedStatement.java:357)
,com.microsoft.sqlserver.jdbc.TDSCommand.close(IOBuffer.java:5781)
,com.microsoft.sqlserver.jdbc.SQLServerStatement.discardLastExecutionResults(SQLServerStatement.java:94)
,com.microsoft.sqlserver.jdbc.SQLServerStatement.closeInternal(SQLServerStatement.java:584)
,com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.closeInternal(SQLServerPreparedStatement.java:170)
,com.microsoft.sqlserver.jdbc.SQLServerStatement.close(SQLServerStatement.java:596)
,com.intergral.fusionreactor.jdbc.jdbc42.PreparedStatementSurrogate2.close(PreparedStatementSurrogate2.java:157)
,lucee.runtime.db.driver.StatementProxy.close(StatementProxy.java:189)
,lucee.commons.db.DBUtil.closeEL(DBUtil.java:88)
…
The server specs are:
Windows Server 2012, Java 8
One with Lucee 4.5.4.017 final and the other running Lucee 4.5.2.018 final.
We have Fusion Reactor installed and it is very clear what is going wrong
in the logs, I am just unsure how to prevent it.
In the thread state log (image attached) you can see that suddenly we get
lots of blocked threads. We crashed at this time and the servers needed
restarting.
In the resource logs we see a similar pattern.
So. I can see the problem, a big jump in requests caused a big jump in
JDBC. Something chocked, the threads get blocked etc.
I am not sure where to start with the fix.
It feels like my DB has enough resource. My suspicion is that this is
something to do with thread management.
Any pointers very much appreciated.