Version Lucee 220.127.116.11-RC
Servlet Container Apache Tomcat/9.0.35
Java 11.0.11 (AdoptOpenJDK) 64bit
Host Name 127.0.0.1
OS Windows Server 2019 (10.0) 64bit
I’m experiencing a strange issue on a default Windows 2019 Server installation. Sporadically, but several times a day, I get an timeout exception on certaint unpredictable requests, which are later viewable in Microsoft Event Viewer with the source
ASP.NET 4.0.30319.0 EVENT-ID: 1309
One example output is:
<System> <Provider Name="ASP.NET 4.0.30319.0" /> <EventID Qualifiers="32768">1309</EventID> <Level>3</Level> <Task>3</Task> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2021-06-21T11:56:43.607510900Z" /> <EventRecordID>328388</EventRecordID> <Channel>Application</Channel> <Computer>somedomain.tld</Computer> <Security /> </System> Event code: 3001 Event message: Die Anforderung wurde abgebrochen. Event time: 21.06.2021 13:56:43 Event time (UTC): 21.06.2021 11:56:43 Event ID: 5b9466197ae54e0d8145993c7b72d3cc Event sequence: 6 Event occurrence: 4 Event detail code: 0 Application information: Application domain: /LM/W3SVC/2/ROOT-1-132687146831469662 Trust level: Full Application Virtual Path: / Application Path: E:\wwwroot_net\ Machine name: TPX Process information: Process ID: 5340 Process name: w3wp.exe Account name: IIS APPPOOL\global Exception information: Exception type: HttpException Exception message: Zeit für Anforderung überschritten. Request information: Request URL: https://somedomain.tld/some-path-1/some-path-2 Request path: /some-path-1/some-path-2 User host address: 89.x.x.x User: Is authenticated: False Authentication Type: Thread account name: IIS APPPOOL\global Thread information: Thread ID: 117 Thread account name: IIS APPPOOL\global Is impersonating: False Stack trace: Custom event details:
As soon as this issue occurs for that request, the issue persists for every consecuting request of that same path/link.
There are no Stack Trace or Custom event details printed. This kind of Event-IDs are usually caused by timeout on Tomcat. I first thought it could be Lucee timing out, but if I directly try it through port 8888 the page is displayed instantly, while on IIS port 443 the issue persists (BUT ONLY FOR THAT LINK). So my first thought was an Boncode AJP issue.
So I’ve started logging Boncode on loglevel 1 and I was getting this type of error for the links throwing that event-id:
- 1.0.41 ERROR Stream reading problem (2)(5), we stopped waiting on Tomcat response. You may have shutdown Tomcat unexpectedly Der Thread wurde abgebrochen. bei BonCodeAJP13.BonCodeAJP13ServerConnection.ReadStream(Byte& receivedPacketBuffer, String readOrigin) bei BonCodeAJP13.BonCodeAJP13ServerConnection.ComunicateWithTomcat() - 1.0.41 ERROR TCP Client level -- Server/Port:localhost/8009 Der Thread wurde abgebrochen. bei BonCodeAJP13.BonCodeAJP13ServerConnection.ComunicateWithTomcat() bei BonCodeAJP13.BonCodeAJP13ServerConnection.HandleConnection() bei BonCodeAJP13.BonCodeAJP13ServerConnection.p_CreateConnection(BonCodeAJP13PacketCollection packetsToSend)
What is strange is, I’m NOT getting that typical Boncode General Communication error and Tomcat is still fully available, because everything else seems fine and looks to be working ok: If I open the same page through port 8888, everything is instantly displayed. But opening on IIS the error keeps displaying.
So my first thing was trying a greater packetsize as described in Boncodes documentation and with the AJP Connector setting in server.xml:
<Connector protocol="AJP/1.3" port="8009" secretRequired="false" redirectPort="8443" address="::1" packetSize="16384"/>
but that also didn’t work. Also, Lucee and Tomcat don’t throw any exception at all.
Further, as soon as I restart Tomcat the links causing the issue work again normally. But after some time, another link will popup in Windows Event Viewer with the same issue.
I don’t know what to try further, thus I’m asking here if others might be experiencing the same issue and know what would be the next step to try out? @Bilal, maybe you can tell what would you try next?