TOMCAT CVE-2020-1938: Ghostcat (AJP)

you need to define a secret!

1 Like

We’re on Tomcat 9.x rather than 8.x, but having applied the patch the default settings is now to require a secret. That means you’ll need to set one both in server.xml and in your BonCode settings file:

Server.xml

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" maxThreads="300" packetSize="65536" secret="XXXX" />

BonCodeAJP13.settings

<RequestSecret>XXXX</RequestSecret>

1 Like

Apache 2.5 supports secrets…mod_cfml isn’t available for 2.5 / VC16…

Only Apache 2.4 is widely available for Windows, a backport has been requested
http://apache-http-server.18135.x6.nabble.com/mod-ajp-adding-quot-secret-xxx-quot-parameter-to-config-yields-syntax-error-td5051515.html

The secret is sent when the secret=secret_keyword parameter is used in ProxyPass or BalancerMember directives. The backend needs to support secret and the values must match. request.secret or requiredSecret are documented in the AJP configuration of the Apache Tomcat.

https://httpd.apache.org/docs/trunk/mod/mod_proxy_ajp.html

Incidentally, here are some straightforward instructions for upgrading Tomcat to the latest point release on Windows without having to re-install:

1 Like

Hi Julian and Zac, thanks for your post, advice and links. I did that all, but I am getting now a strange IIS “403 forbidden” page. I did nothing to the webroot, it is absolutely the same because it residest outside the lucee installation folder. To me it looks that boncode is not giving the secret to tomcat. Besides that, I am having this error:

23-Feb-2020 12:02:39.752 WARNING [127.0.0.1-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Thread-566] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.net.SocketInputStream.socketRead0(Native Method)
 java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
 java.net.SocketInputStream.read(SocketInputStream.java:171)
 java.net.SocketInputStream.read(SocketInputStream.java:141)
 org.apache.http.impl.conn.LoggingInputStream.read(LoggingInputStream.java:87)
 org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
 org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
 org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
 org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
 org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
 org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
 org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
 org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
 org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
 org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
 org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
 org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
 org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
 org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
 org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
 org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
 lucee.commons.net.http.httpclient.HTTPEngine4Impl._invoke(HTTPEngine4Impl.java:271)
 lucee.commons.net.http.httpclient.HTTPEngine4Impl.get(HTTPEngine4Impl.java:113)
 lucee.commons.net.http.HTTPEngine.get(HTTPEngine.java:91)
 lucee.runtime.schedule.ExecutionThread.execute(ExecutionThread.java:109)
 lucee.runtime.schedule.ExecutionThread.run(ExecutionThread.java:59)

Which Boncode and Lucee versions are you using?

I’m pretty sure that error is unrelated, it’s due to problems with the scheduler [LDEV-2712] - Lucee and linked tasks

So for now we are back online normally running with:

Version Lucee 5.2.9.31
Betriebssystem Windows Server 2012 R2 (6.3) 64bit
Servlet Container Apache Tomcat/8.5.51
Java 1.8.0_242 (Azul Systems, Inc.) 64bit
Architecture 64bit

but I coulnd’t get AJP and BondCode-Connector to work with secretRequired and requiredSecret=“xxxx” like @Julian_Halliwell advised (thanks by the way for ultra fast reply Julian). What I did finally was to setup secretRequired=“false” and that did it.

Our BoncodeConnector is Version 1.0.25, and setting RequestSecret to support Tomcat requiredSecret setup has been added to boncode in 1.0.24. However

server.xml

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" requiredSecret="thisismysecret" secretRequired="true" />

and

BonCodeAJP13.settings

<Settings>
  <Port>8009</Port>
  <Server>localhost</Server>   
  <MaxConnections>200</MaxConnections>
  <LogLevel>0</LogLevel>
  <LogDir>c:\temp</LogDir>
  <FlushThreshold>0</FlushThreshold>
  <EnableRemoteAdmin>False</EnableRemoteAdmin>
  **<RequestSecret>thisismysecret</RequestSecret>**
</Settings>

Always gave a 403 Forbidden. There is a hint to that link at this link here (see “RequestSecret”):
http://boncode.net/connector/webdocs/Tomcat_Connector.htm#_Toc520189759

But all that didn’t work.

Does the Apache Tomcat/Lucee windows service account have access to the tomcat installation folder? Perhaps permissions were overwritten when you upgraded?

This may be your problem. In the patched versions, this attribute should be secret, not requiredSecret (which is what the pre-patch versions used).

secret="thisismysecret"

In the patched versions you don’t need secretRequired="true" because that’s now the default.

I’ve tried that, but neither secret=“thisismysecret” nor requiredSecret=“thisismysecret” worked. I had to explicitly set secretRequired=“false”, and that made it for now. This wouldn’t cause a big impact in our configuration/setup, because only the admin have access to the server and webroots. This is OK for now.

That shouldn’t be the case. All I am doing now is changing server.xml and boncodeAJP13.settings. Maybe secretRequired=“true” creates/levels up to higher strict file permissions… really can’t say right know. Will try/error more about this soon. For now lots of thanks to you and all here helping out of this.

That’s right because the default has changed in the patched version. This solves the problem but obviously isn’t ideal - but as long as you’ve locked down direct access to Tomcat over AJP should be ok I guess.

Well you’ve also upgraded Tomcat, surely? But I wouldn’t expect file permissions to have changed, it’s just that’s what 403 errors are usually caused by hence the suggestion to check.

Last thing, did you restart Lucee/Tomcat? Sorry if that’s obvious, but it’s required to pick up changes to server.xml.

Yes, I did. No need to worry about questioning what ever comes to your mind. Any idea/hint about what ever could be going on is always very appreciated and warmly welcomed.

I can live with that configuration for now, because it isn’t any different from before. Of course enhancing security on AJP with secret key would be much better. Going to dig into this issue as soon as I can. Still glad you’ve identified the problem pretty fast and shared it here. Thanks!

Have you tried updating your boncode connector to the current version (1.0.41)?

1 Like

Definitely worth trying. I had one site which was using an old Boncode version by mistake and that did produce the 403 Tomcat error. Updating to the latest fixed it.

2 Likes

Good news on the Apache Httpd side of things

mod_proxy_ajp: patch to set worker secret passed to tomcat
https://bz.apache.org/bugzilla/show_bug.cgi?id=53098

Backported today to 2.4.x as r1874456, will be part of 2.4.42

This is totally embarrassing to post, but I got to do this: I was adding the settings in the BonCodeAJP13.settings of the installation files all the time, AND NOT in C:\windows\BonCodeAJP13.settings. Changed it, restarted IIS and Lucee and all worked. Sigh… frustrating… Really, sorry for bothering you with my stupidity. Zillions of apologies ( @martin , @Zackster, @Julian_Halliwell and every one else here)

4 Likes

If you run Tomcat on Linux and depending on which version you have been before be aware that the file permissions are different with 8.5. This has impact on file uploads with CF.

You might want to check out this and set UMASK accordingly:

|Version|Lucee 5.2.9.31|

|Betriebssystem|Windows Server 2012 R2 (6.3) 64bit|
|Servlet Container|Apache Tomcat/8.5.51|
|Java|1.8.0_242 (Azul Systems, Inc.) 64bit|
|Architecture|64bit|
IIS 8.5.96.00

We have seen a significant impact in TTFB (Server Response Time viewed in Googled Dev Tool Network) after the patch. A blank cfm page over port 8888 was responding in a couple of milliseconds. None cached images over IIS https about 25ms. But when retrieving the same blank cfm page over IIS (AJP with Boncode server.xml connector attribute secretRequired=“false”) all requests need over 1.03 Seconds (always). Downgraded to Tomcat Apache Tomcat/8.0.28 for testing and TTFB came back to 6ms. This performance impact is very probably related to the patch. Can somebody confirm that?

yeah, I’m seeing that same problem, with 1.03s per request with 8.5.51, via Apache not IIS

Bug filed here 64182 – 1.03s delay on every request via ajp

I also encountered a problem with quirky config handling from the AJP connector

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" connectionTimeout="-1" secretRequired="false" address="127.0.0.1" secret="zac"/>

Lucee won’t start with this config, if I remove the secret, it works (Apache httpd 2.4 mod_proxy_ajp doesn’t support secrets until the next release)

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" connectionTimeout="-1" secretRequired="false" address="127.0.0.1"/>

It’s been already been filed as a bug 64180 – secretRequred=false is ignored if secret=<anything>

If you’re upgrading Tomcat by just replacing the lib directory, remember to copy across mod_cfml-valve_v1.1.11.jar, if your using an older version of mod_cfml, you can grab the new jar version from mod_cfml/java at master · viviotech/mod_cfml · GitHub

The newest version of mod_cfml supports returning a 307 to the first request (first request always gets a redirect), which is important, as otherwise it returns a 302 to the first request, which if it’s a POST from say a javascript app, will break everything as the POST becomes a GET and drops all the form fields.

1 Like

Does it change behaviour if you specify address="::1" vs address="127.0.0.1" (i.e. ipv6 vs ipv4)?