Version Lucee 5.3.8.179-RC
Servlet Container Apache Tomcat/9.0.35
Java 11.0.11 (AdoptOpenJDK) 64bit
Host Name 127.0.0.1
OS Windows Server 2019 (10.0) 64bit
Architecture 64bit
I’m experiencing a strange issue on a default Windows 2019 Server installation. Sporadically, but several times a day, I get an timeout exception on certaint unpredictable requests, which are later viewable in Microsoft Event Viewer with the source
ASP.NET 4.0.30319.0
EVENT-ID: 1309
One example output is:
<System>
<Provider Name="ASP.NET 4.0.30319.0" />
<EventID Qualifiers="32768">1309</EventID>
<Level>3</Level>
<Task>3</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2021-06-21T11:56:43.607510900Z" />
<EventRecordID>328388</EventRecordID>
<Channel>Application</Channel>
<Computer>somedomain.tld</Computer>
<Security />
</System>
Event code: 3001
Event message: Die Anforderung wurde abgebrochen.
Event time: 21.06.2021 13:56:43
Event time (UTC): 21.06.2021 11:56:43
Event ID: 5b9466197ae54e0d8145993c7b72d3cc
Event sequence: 6
Event occurrence: 4
Event detail code: 0
Application information:
Application domain: /LM/W3SVC/2/ROOT-1-132687146831469662
Trust level: Full
Application Virtual Path: /
Application Path: E:\wwwroot_net\
Machine name: TPX
Process information:
Process ID: 5340
Process name: w3wp.exe
Account name: IIS APPPOOL\global
Exception information:
Exception type: HttpException
Exception message: Zeit für Anforderung überschritten.
Request information:
Request URL: https://somedomain.tld/some-path-1/some-path-2
Request path: /some-path-1/some-path-2
User host address: 89.x.x.x
User:
Is authenticated: False
Authentication Type:
Thread account name: IIS APPPOOL\global
Thread information:
Thread ID: 117
Thread account name: IIS APPPOOL\global
Is impersonating: False
Stack trace:
Custom event details:
As soon as this issue occurs for that request, the issue persists for every consecuting request of that same path/link.
There are no Stack Trace or Custom event details printed. This kind of Event-IDs are usually caused by timeout on Tomcat. I first thought it could be Lucee timing out, but if I directly try it through port 8888 the page is displayed instantly, while on IIS port 443 the issue persists (BUT ONLY FOR THAT LINK). So my first thought was an Boncode AJP issue.
So I’ve started logging Boncode on loglevel 1 and I was getting this type of error for the links throwing that event-id:
Boncode Error:
- 1.0.41 ERROR
Stream reading problem (2)(5), we stopped waiting on Tomcat response. You may have shutdown Tomcat unexpectedly
Der Thread wurde abgebrochen.
bei BonCodeAJP13.BonCodeAJP13ServerConnection.ReadStream(Byte[]& receivedPacketBuffer, String readOrigin)
bei BonCodeAJP13.BonCodeAJP13ServerConnection.ComunicateWithTomcat()
- 1.0.41 ERROR
TCP Client level -- Server/Port:localhost/8009
Der Thread wurde abgebrochen.
bei BonCodeAJP13.BonCodeAJP13ServerConnection.ComunicateWithTomcat()
bei BonCodeAJP13.BonCodeAJP13ServerConnection.HandleConnection()
bei BonCodeAJP13.BonCodeAJP13ServerConnection.p_CreateConnection(BonCodeAJP13PacketCollection packetsToSend)
What is strange is, I’m NOT getting that typical Boncode General Communication error and Tomcat is still fully available, because everything else seems fine and looks to be working ok: If I open the same page through port 8888, everything is instantly displayed. But opening on IIS the error keeps displaying.
So my first thing was trying a greater packetsize as described in Boncodes documentation and with the AJP Connector setting in server.xml:
<Connector protocol="AJP/1.3"
port="8009"
secretRequired="false"
redirectPort="8443"
address="::1"
packetSize="16384"/>
but that also didn’t work. Also, Lucee and Tomcat don’t throw any exception at all.
Further, as soon as I restart Tomcat the links causing the issue work again normally. But after some time, another link will popup in Windows Event Viewer with the same issue.
I don’t know what to try further, thus I’m asking here if others might be experiencing the same issue and know what would be the next step to try out? @Bilal, maybe you can tell what would you try next?