Performance issue with Lucee, AJP, and IIS

Junior · January 11, 2024, 4:31pm

I’m migrating my system from ACF to Lucee. I’m finding it amazing to have some new possibilities, considering I’ve been on ACF 2016. However, I’ve been facing an issue that I can’t solve for more than 15 days. Currently, my ACF server runs on a Windows server using IIS. We have individual connectors for each site, and for over 4 years, I haven’t encountered any performance problems; it works perfectly. As ACF has a very high cost, we decided to switch to Lucee. We’ve been making small adjustments to our code for about 6 months to make it run 100%, and we succeeded! =]

The problem is that we are unable to get the Lucee server to respond at the same speed as we do with ACF. I’ll include performance screenshots below to help you understand what I’m talking about.

I’m running a test with a CFM file containing:

<cfset = “test”>
<cfoutput>
    #variables.test#
</cfoutput>

Conducting a test through K6 simulating 1000 VUs for 1 minute.

Windows - ACF 2016 - IIS

Checks: 217.386
http_req_waiting: 276.37ms

Windows - Lucee - IIS

Checks: 21.464
http_req_waiting: 2.81s

Windows - IIS (HTML text)

Checks: 337.681
http_req_waiting: 177.74ms

Windows - Lucee - without the IIS, consequently without the connector

Checks: 204.170
http_req_waiting: 293.63ms

For me, as a beginner with Lucee, it becomes evident that when there is a connection from Lucee to IIS, the bottleneck causing delays is in the connector. Below are the server.xml configurations

<Server port="8005" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
  <GlobalNamingResources>
    <Resource name="UserDatabase" auth="Container" type="org.apache.catalina.UserDatabase" description="User database that can be updated and saved" factory="org.apache.catalina.users.MemoryUserDatabaseFactory" pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>
  <Service name="Catalina">
    <Connector port="8888" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" />
    <Connector 
		protocol="AJP/1.3" 
		port="8009" 
		packetSize="65536" 
		secret="xxxx" 
		secretRequired="true" 
		redirectPort="8443" 
		address="::1" 
		maxConnections="10000" 
		MaxThreads="5000" 
	/>
    <Engine name="Catalina" defaultHost="127.0.0.1">
      <Realm className="org.apache.catalina.realm.LockOutRealm">
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase"/>
      </Realm>
      <Host name="127.0.0.1"  appBase="webapps" unpackWARs="true" autoDeploy="true">
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="localhost_access_log" suffix=".txt" pattern="%h %l %u %t &quot;%r&quot; %s %b" />
		<Valve className="mod_cfml.core" loggingEnabled="false" maxContexts="200" timeBetweenContexts="2000" scanClassPaths="false" responseCode="307" sharedKey="xxxx" />
      </Host>
    </Engine>
  </Service>
</Server>

BonCodeAJP13.settings

<Settings>
<Server>localhost</Server>
<Port>8009</Port>
<EnableRemoteAdmin>True</EnableRemoteAdmin>
<EnableHeaderDataSupport>True</EnableHeaderDataSupport>
<ForceSecureSession>False</ForceSecureSession>
<AllowEmptyHeaders>False</AllowEmptyHeaders>
<ModCFMLSecret>xxx</ModCFMLSecret>
<RequestSecret>xxxx</RequestSecret>
<MaxConnections>1000</MaxConnections>
<PacketSize>65536</PacketSize>
<EnableAggressiveGC>true</EnableAggressiveGC>
<FlushThreshold>1</FlushThreshold>
</Settings>

“The question is, what should be done to achieve results similar to ACF?”

Thanks

OS: Windows Server 2022 Standard
Java Version: 11.0.20.1
Tomcat Version: 9.0.80
Lucee Version: 6.0.0.585

TonyMonast · January 11, 2024, 9:23pm

HI,

Could it be related to this?

github.com/Bilal-S/iis2tomcat

Requests much slower than expected through IIS/AJP connector

opened 05:30PM - 07 May 20 UTC

closed 07:19PM - 07 May 20 UTC

bhartsfield

I have a Windows Server 2016 VM with IIS and a couple of instances of Lucee 5.3.…5+92 (using Commandbox). I am using the 1.0.41 connector to connect two IIS sites to their respective commandbox/Lucee instances. What I am seeing with that setup is that going directly to the underlying commandbox instances is much faster than sending the request through IIS and the AJP connector. On average, a request directly to the underlying server is about 300ms while, on average through IIS/AJP is about 1.4 seconds. The boncode settings are minimal. ``` <Settings> <Server>localhost</Server> <Port>8009</Port> <EnableRemoteAdmin>False</EnableRemoteAdmin> <EnableHeaderDataSupport>False</EnableHeaderDataSupport> <ForceSecureSession>False</ForceSecureSession> <AllowEmptyHeaders>False</AllowEmptyHeaders> <ResolveRemoteAddrFrom>HTTP_X_FORWARDED_FOR</ResolveRemoteAddrFrom> <PacketSize>65531</PacketSize> </Settings> ``` As are the web.config settings. ``` <handlers> <add name="BonCode-Tomcat-CFC-Handler" path="*.cfc" verb="*" type="BonCodeIIS.BonCodeCallHandler" preCondition="integratedMode" /> <add name="BonCode-Tomcat-CFM-Handler" path="*.cfm" verb="*" type="BonCodeIIS.BonCodeCallHandler" preCondition="integratedMode" /> </handlers> ``` I have zero proof the the slowness is being caused by the connector or that it is being caused by IIS but it seems to be one of them (or a combination of both). Any thoughts on how to speed this up or track down the actual cause?

mikechy · January 12, 2024, 12:01am

OPen up your server.xml file and in the connector node (the one with protocol=“AJP/1.3”, edit your address line to this

address=“::1” rather than 0

and restart the server.

Try that

Junior · January 12, 2024, 2:32am

I tried, it does improve the time indeed, but it didn’t solve it. I’m using this solution, but the difference compared to ACF remains immense.

Junior · January 12, 2024, 2:33am

I had changed it to 0 for testing, but currently, it’s set to 1.

carehart · January 12, 2024, 5:21am

First and foremost, in order to know WHAT that address value should be, do a ping localhost on the server. What does it resolve to? If ::1, then put that there. If perhaps 127.0.0.1, put that there. And you MUST restart Lucee/tomcat, of course, for that change to take effect. Had you done that with your last change attempt?

Note also that you could duplicate the address line, with each having the different address (in case something may change what localhost resolves to on your machine). That’s a surprisingly simple solution for many.

As for your vast difference between lucee and acf, it’s not clear: are they both on this same machine? If not, that could be the source of some difference. If you remain stumped and you’re NOT running that acf deployment on the machine, you may want to implement it just for this simple load test, as a sanity check.

There’s much more we might consider, which may vary based on your responses to all the above.

andreas · January 12, 2024, 5:26am

Hi Junior, I’m also alemão (in the cfmlbrazil group), glad you got some of the issues isolated down to the connector. So its not a Lucee issue as you assumed before. Looks like its the issue that came with the ghostcat patch.

I hope you are setting the boncode settings in the correct file: this is located in the windows directory and NOT in the installer directory. When I had this problem I was always changing the wrong file.Just checking.

Junior · January 13, 2024, 4:35pm

I asked for help on GitHub from Bilal (iis2tomcat), and he requested me to make an adjustment in BonCode and change the value of the “Server.” I set the value to “::1” because that is what responds when I ping localhost.

<Settings>
<Server>::1</Server>
<Port>8009</Port>
<EnableRemoteAdmin>True</EnableRemoteAdmin>
<EnableHeaderDataSupport>True</EnableHeaderDataSupport>
<ModCFMLSecret>xxxx</ModCFMLSecret>
<RequestSecret>xxxx</RequestSecret>
<PacketSize>65536</PacketSize>
<MaxConnections>4000</MaxConnections>
</Settings>

Server.xml

<Connector 
		protocol="AJP/1.3" 
		port="8009" 
		packetSize="65536" 
		secret="xxx" 
		secretRequired="true" 
		redirectPort="8443" 
		address="::1"
		enableLookups="false"
		keepAliveTimeout="-1"
		maxThreads="4000"
	/>

The information that made all the difference is that the “MaxConnections” in BonCode must be the same value as the “maxThreads” in the server.xml. Along with keepAliveTimeout=“-1”. These adjustments completely changed the performance. I will include a screenshot of the test.

Checks: 478.436
http_req_waiting: 116.76ms

Thank you to everyone who contributed to the discussion and took the time to respond to me. I hope this thread helps other people who, like me, are struggling to improve performance.

carehart · January 13, 2024, 5:01pm

Very glad to hear that you solved it. And kudos to Bilal, of course, who’s a wonderful human.

But since you changed the 3 things and feel that the maxconnections “made all the difference”, are you saying that if you now set it back to your original 1000, you’d get your original poor performance? Or is it better (with the other two changes) but still most affected by this? (Sorry, I’m writing on a phone so can’t just go setup the same test myself. Also, differences in your setup may influence results.)

It’s just that capturing such experience and knowledge transfer is really valuable–but more so when it’s clear what the impacts of specific changes are. Again, sometimes it’s not the same for everyone, or not as valuable alone as in conjunction with other changes.

Junior · January 13, 2024, 5:50pm

Hi, Carehart

Thanks for the prompt. I ran the test and set it to 1000, and to my surprise, the result remained the same (minimal variation). However, equalizing the values prevented any connection errors. Previously, I was getting numerous connection attempts with errors.

So, in that sense, I believe it helped stabilize the communication.

After all this headache, I’m starting to enjoy the switch from ACF to Lucee again.

Charlesr · January 14, 2024, 8:01am

Thank you for this very valuable information. I never knew about any of this. I must say, I have been a little dissatisfied with the performance of Lucee, as compared to ACF, but because I cannot afford an ACF Commercial Licence, I have just put up with this issue.
I am now very much looking forward to reassessing my connection settings and changing them accordingly.
Once again, this is a great thread and extremely valuable.

Charlesr · January 14, 2024, 8:34am

@Junior Just out of interest. Do we know what the default maxThreads value is? I found that this was NOT set in my server.xml AJP connector config? In BonCode, MaxConnections was already set to 200? I am wondering whether this matches Lucee’s AJP maxThreads value, by default?
And also do we know what the default keepAliveTimeout & address values, are set to, in server.xml? Again, these were not set, in my configuration.
I have pinged localhost and it responds with ::1
I am wondering now, whether I need to change anything?
Perhaps, by default, everything is already optimised?

And here is my server.xml as it was, by default:

<Connector protocol="AJP/1.3"
			port="8009"
			secret="[secret value]"
			secretRequired="true"
			redirectPort="8443"
/>

And my BoncodeAJP13.settings, before I changed the MaxConnections setting:

<Settings>
  <Port>8009</Port>
  <Server>localhost</Server>   
  <MaxConnections>200</MaxConnections>
  <LogLevel>0</LogLevel>
  <LogDir>c:\temp</LogDir>
  <FlushThreshold>0</FlushThreshold>
  <EnableRemoteAdmin>False</EnableRemoteAdmin>
</Settings>

Junior · January 14, 2024, 2:25pm

Hello, Carlos! It’s great to see that my shared experiences are proving helpful. I’d like to express my sincere gratitude to the forum and also to Bilal for the valuable support in the context of the AJP connector. The collaboration of everyone was crucial in overcoming challenges and finding solutions, making the journey smoother and more productive. Thank you all for your contribution and support!

It’s worth noting that by default, Lucee is not optimized for heavy loads, initially configured to cater to sites with low to medium traffic. Similar to my previous experience with ACF, manual adjustments are necessary to achieve faster results in robust environments.

The default value for maxThreads is 200, and in the case of BonCode, MaxConnections is also set to 200, indicating a standard for serving low to medium loads on the server.

The keepAliveTimeout has a default value of 20 seconds. If set to -1, the connection will be closed without traffic, allowing for faster connection recycling.

As for the address attribute, it’s not defined by default. However, research suggests that if not specified, the server may conduct searches, resulting in slow responses and a delay of up to 1 second. Setting the address to ::1 can optimize performance, a recommendation also valid for BonCode.

I recommend making adjustments in the test environment to assess the differences. In my case, the improvement was significant, making access to the site much faster. Give it a try and see how these settings can positively impact your environment.

Afterward, come back here and let me know if it changed the performance for you. It’s good to find out if this will also make a difference in different environments.

andreas · January 14, 2024, 9:04pm

This is a very valuable post @Junior really, thank you for posting all you experience. Muito obrigado!!!

tradecraft.asia · January 16, 2024, 8:22am

All I want to say is thank everyone for the contribution. I learned things I did not know for sure, more than you’d know. Thanks everyone.

aspirenet · February 21, 2024, 7:42pm

Windows Server 2022/IIS10
Java 11.0.21
Tomcat Version 9.0
Lucee 6.0.0.585
MS SQL 2022

I appear to potentially have the same issue. I started a new thread as I had no found this one until @TonyMonast kindly made me aware of it.

My CFM pages are taking about 4 seconds to load, however if I test them locally on port 8888 I would not describe them as lightening fast, which I was hoping for locally, but it’s certainly a lot quicker and not 4 seconds.

I am not a guru quite the opposite and I am new to Lucee. I’ve read through the above and here’s where I am up to, hopefully somebody can point me in the right direction

I pinged localhost and it goes to :11
I opened c:\windows\BonCodeAJP13.settings and made a change
Changed this
localhost
To this
::1

I then looked for server.xml, which I found in c:\lucee\tomcat\conf\server.xml

It has a lot of comments in it, and I was unable to find a reference to address=

There is this

I am wondering if I have to add address=“::1” here? I was under the impression I was changing it, so I’m concerned I might have the wrong file. This would give me

I did stop/start the service to see if it made any difference with the one change that I made, and it didn’t.

Could somebody please point me in the right direction

Thanks
Mark

Junior · February 21, 2024, 7:58pm

In server.xml you can search for protocol=“AJP/1.3” and add the parameter.

<Service name="Catalina">
    <Connector port="8888" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" />
    <Connector 
		protocol="AJP/1.3" 
		port="8009" 
		packetSize="65536" 
		secret="xxxx" 
		secretRequired="true" 
		redirectPort="8443" 
		address="::1" 
		maxConnections="10000" 
		MaxThreads="5000" 
	/>

After that you need to restart Lucee and when changing BinCode you need to restart IIS

aspirenet · February 21, 2024, 11:04pm

Here’s what I’ve found when I modified the BonCodeAJP13.setttings and the Server.xml files

Both original files - 4 second lag on page loads
Updated both of them - no lag
Updated server.xml only, keeping BonCodeAJP13.setttings as the original - no lag
Updated BonCodeAJP13.setttings but left server.xml - Server errors (see screenshot)

I’ve attached a screenshot of which files I changed, the location of them and what I changed, along with the test results.

It should be noted I did see a few intermittent lags but there were only a few, and I happened to notice that Microsoft antimalware service executable had decided to grab a lot of the CPU resource at that particular time, it was tricky to spot it, it happened a little quick, but I’m confident it was happening at the same time. It was a test VM with 1 core, so I cranked up the cores to 8 and gave it a few gig of extra RAM and it seemed to help. ChatGPT has given me some clear instructions on how to disable it, which is very tempting.

Again thanks to everybody for your help, VERY much appreciated!

aspirenet · February 21, 2024, 11:06pm

I should add, I went with original BonCodeAJP13.setttings and updated Server.xml . It seems to work. Does anybody see a potential issue with that?

jhonatanjacinto · April 4, 2024, 7:25pm

Amazing! Thanks everyone (specially @Junior) for this tip in solving the performance problem when Lucee is integrated with IIS.

These seconds of difference were really annoying and really bothered me! Now, with these adjustments, the performance boost is extremely perceptible.

Once again, thanks everybody!