I have been doing some testing under a load and about 15% of the time I am getting a 502 error; Generic Connector Communication Error:
I can’t figure out what is wrong with my setup that I am having so many errors the CPU never goes over about 85% during the tests and memory is only 50%. Under a load I can bring the site up in a browser and if I keep hitting refresh I eventually get the Generic Connector error.
Does anyone have any ideas for my to get things working a bit more stable? Below are some to the things I have tried.
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009" redirectPort="8443" protocol="AJP/1.3" maxThreads="1000" keepAliveTimeout="-1"/>
And BonCodeAJP13.settings I changed max connections <MaxConnections>0</MaxConnections>
I have also enabled additional loggin to the event viewer by running
eventcreate /ID 1 /L APPLICATION /T INFORMATION /SO BonCodeConnector /D “Source Created”
which is giving me this information:
Connection error 1: Only one usage of each socket address (protocol/network address/port) is normally permitted [::1]:8009 at System.Net.Sockets.TcpClient…ctor(String hostname, Int32 port)
at BonCodeIIS.BonCodeCallHandler.ProcessRequest(HttpContext context)
In Luce-Tomcat service control I have 512MB Initial Memory and 4096 in the Max pool, I am a 4cpu/13GB memory VM running Windows2016 if that matters
Based on everything I have read the whole BonCode/threads/mem allocation/heap size combo is a bit of a mystery and black magic any everything I have read talks about test different configs but it seems everything I try either makes no difference or makes things worse. I just need to be pointed in the right direction on this one.
The documentation at BonCode Apache Tomcat AJP 1.3 Connector is thorough. You need to be very careful with your server.xml and BonCodeAJP13.settings files… for example, you’d want <MaxConnections>1000</MaxConnections> to match your server.xml. We were caught out once by an incorrect capitalisation. Check every character carefully and you should be right.
Thank you Simon for the feed back. I have read through that doc a couple of times and I think I am missing something. I did make sure that the capitalization was correct and I have set my MaxConnections & maxThreads both to 500. i also increased the Initial Memory Pool to 1024 and the Maximum Memory Pool to 8192 (Thred Stack Size was empty and I left it that way not sure what it should be). Ran another load test and 16% of the hit were to Generic Connector Error (status 502) and my event log filled up with this error:
Connection error 1: Only one usage of each socket address (protocol/network address/port) is normally permitted [::1]:8009 at System.Net.Sockets.TcpClient…ctor(String hostname, Int32 port)
at BonCodeIIS.BonCodeCallHandler.ProcessRequest(HttpContext context)
Changing the Dynamic Port Range
Open regedit.
Open key HKLM\System\CurrentControlSet\Services\Tcpip\Parameters
Edit (or create as DWORD) the MaxUserPort value.
Set it to a higher number. (i.e. 65534)
Changing the TIME_WAIT delay
Open regedit.
Open key HKLM\System\CurrentControlSet\Services\Tcpip\Parameters
Edit (or create as DWORD) the TcpTimedWaitDelay.
Set it to a lower number. Value is in seconds. (i.e. 60 for 1 minute delay)
The problem is 14% is still pretty high error rate. There must be some other setting I am missing.
Since you are doing load testing,can we validate the Tomcat is handling the load. Run your load tester directly against the Tomcat Webport (8888 on Lucee standard installs). Is everything working without issues?
Nonetheless, If I interpret your message correctly it may be based on port contention. Given general TCP/IP protocol stack rules, you have 65,535 ports per IP available (2-byte maximum). I am referring to source ports here, as the destination is always 8009.
Even if you call internally (localhost).
You seem to be getting into a situation where we are trying to address from a port that is either still open or has not been discarded by Windows TCP/IP yet. Even if the connector closes the port there is a life-cylcle process Windows has to walk through that keeps ports reserved for a bit after and releases it in concert with Tomcat. You can build contention when you have rapid connections to the machine and they are immediately discarded by connector (MaxConenction=0). Tomcat and Windows will work to clean up after the connections and it takes additional time.
Also if for some reason the Tomcat stack is still keeping the connection open the port will not be usable until that one is closed.
I am not sure how your test are structured but if you need to simulate a large number of users hitting one machine, you will have to spread traffic across multiple internal IPs or allow a little more time between requests.
You can use the TCPView tool to see the connections in detail:
As far as connector, you should attempt to reuse already established connections, so the should be greater than zero as you have it in your last setup. This way no new ports have to established and broken down.
You seem to be following setting the connector for maximum performance as documented in the section “Configuring the Connector for Performance”. This is a good setup generally.