TOMCAT CVE-2020-1938: Ghostcat (AJP)

Time to check and update your bundled Tomcat version

Apache has released patches for several versions of Tomcat.

Apache Version Affected Release Versions Fixed Version
Apache Tomcat 9 9.0.30 and below 9.0.31
Apache Tomcat 8 8.5.50 and below 8.5.51
Apache Tomcat 7 7.0.99 and below 7.0.100

Commandbox doesn’t use Tomcat, but the standard Lucee distribution does.

You can mitigate this without updating by configuring tomcat to use a secret, as noted in the above links.

Unfortunately, the Apache Httpd connector doesn’t support this yet (bug was filed in 2012 sigh)

https://bz.apache.org/bugzilla/show_bug.cgi?id=53098

and mod_cfml still won’t work with the latest Apache release if it was available

3 Likes

OS windows Server 2012 using IIS & Lucee Installer & boncode
I’ve tried to update tomcat 8 to 8.5.51 just by doing the following steps:

I can successfully access port 8888 an every domain and it seems to be all good, but somehow the boncode connector doesn’t get connected. I’m receiving a:

Error connecting to Apache Tomcat instance.
Please check that a Tomcat server is running at given location and port.
Details:
Es konnte keine Verbindung hergestellt werden, da der Zielcomputer die Verbindung verweigerte 127.0.0.1:8009

You can change this message by changing TomcatConnectErrorURL setting in setting file.

Is there something I am missing? Do I have to update something for the boncode connector?

UPDATE: catalina.log

23-Feb-2020 10:21:56.948 SCHWERWIEGEND [main] org.apache.catalina.core.StandardService.startInternal Failed to start connector [Connector[AJP/1.3-8009]]
	org.apache.catalina.LifecycleException: Der Start des Protokoll-Handlers ist fehlgeschlagen
		at org.apache.catalina.connector.Connector.startInternal(Connector.java:1057)
		at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
		at org.apache.catalina.core.StandardService.startInternal(StandardService.java:440)
		at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
		at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:766)
		at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
		at org.apache.catalina.startup.Catalina.start(Catalina.java:688)
		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:498)
		at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:343)
		at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:474)
	Caused by: java.lang.IllegalArgumentException: The AJP Connector is configured with secretRequired="true" but the secret attribute is either null or "". This combination is not valid.
		at org.apache.coyote.ajp.AbstractAjpProtocol.start(AbstractAjpProtocol.java:274)
		at org.apache.catalina.connector.Connector.startInternal(Connector.java:1055)
		... 12 more

you need to define a secret!

1 Like

We’re on Tomcat 9.x rather than 8.x, but having applied the patch the default settings is now to require a secret. That means you’ll need to set one both in server.xml and in your BonCode settings file:

Server.xml

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" maxThreads="300" packetSize="65536" secret="XXXX" />

BonCodeAJP13.settings

<RequestSecret>XXXX</RequestSecret>

1 Like

Apache 2.5 supports secrets…mod_cfml isn’t available for 2.5 / VC16…

Only Apache 2.4 is widely available for Windows, a backport has been requested
http://apache-http-server.18135.x6.nabble.com/mod-ajp-adding-quot-secret-xxx-quot-parameter-to-config-yields-syntax-error-td5051515.html

The secret is sent when the secret=secret_keyword parameter is used in ProxyPass or BalancerMember directives. The backend needs to support secret and the values must match. request.secret or requiredSecret are documented in the AJP configuration of the Apache Tomcat.

https://httpd.apache.org/docs/trunk/mod/mod_proxy_ajp.html

Incidentally, here are some straightforward instructions for upgrading Tomcat to the latest point release on Windows without having to re-install:

1 Like

Hi Julian and Zac, thanks for your post, advice and links. I did that all, but I am getting now a strange IIS “403 forbidden” page. I did nothing to the webroot, it is absolutely the same because it residest outside the lucee installation folder. To me it looks that boncode is not giving the secret to tomcat. Besides that, I am having this error:

23-Feb-2020 12:02:39.752 WARNING [127.0.0.1-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Thread-566] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.net.SocketInputStream.socketRead0(Native Method)
 java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
 java.net.SocketInputStream.read(SocketInputStream.java:171)
 java.net.SocketInputStream.read(SocketInputStream.java:141)
 org.apache.http.impl.conn.LoggingInputStream.read(LoggingInputStream.java:87)
 org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
 org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
 org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
 org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
 org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
 org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
 org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
 org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
 org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
 org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
 org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
 org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
 org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
 org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
 org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
 org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
 lucee.commons.net.http.httpclient.HTTPEngine4Impl._invoke(HTTPEngine4Impl.java:271)
 lucee.commons.net.http.httpclient.HTTPEngine4Impl.get(HTTPEngine4Impl.java:113)
 lucee.commons.net.http.HTTPEngine.get(HTTPEngine.java:91)
 lucee.runtime.schedule.ExecutionThread.execute(ExecutionThread.java:109)
 lucee.runtime.schedule.ExecutionThread.run(ExecutionThread.java:59)

Which Boncode and Lucee versions are you using?

I’m pretty sure that error is unrelated, it’s due to problems with the scheduler https://luceeserver.atlassian.net/browse/LDEV-2712 and linked tasks

So for now we are back online normally running with:

Version Lucee 5.2.9.31
Betriebssystem Windows Server 2012 R2 (6.3) 64bit
Servlet Container Apache Tomcat/8.5.51
Java 1.8.0_242 (Azul Systems, Inc.) 64bit
Architecture 64bit

but I coulnd’t get AJP and BondCode-Connector to work with secretRequired and requiredSecret=“xxxx” like @Julian_Halliwell advised (thanks by the way for ultra fast reply Julian). What I did finally was to setup secretRequired=“false” and that did it.

Our BoncodeConnector is Version 1.0.25, and setting RequestSecret to support Tomcat requiredSecret setup has been added to boncode in 1.0.24. However

server.xml

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" requiredSecret="thisismysecret" secretRequired="true" />

and

BonCodeAJP13.settings

<Settings>
  <Port>8009</Port>
  <Server>localhost</Server>   
  <MaxConnections>200</MaxConnections>
  <LogLevel>0</LogLevel>
  <LogDir>c:\temp</LogDir>
  <FlushThreshold>0</FlushThreshold>
  <EnableRemoteAdmin>False</EnableRemoteAdmin>
  **<RequestSecret>thisismysecret</RequestSecret>**
</Settings>

Always gave a 403 Forbidden. There is a hint to that link at this link here (see “RequestSecret”):
http://boncode.net/connector/webdocs/Tomcat_Connector.htm#_Toc520189759

But all that didn’t work.

Does the Apache Tomcat/Lucee windows service account have access to the tomcat installation folder? Perhaps permissions were overwritten when you upgraded?

This may be your problem. In the patched versions, this attribute should be secret, not requiredSecret (which is what the pre-patch versions used).

secret="thisismysecret"

In the patched versions you don’t need secretRequired="true" because that’s now the default.

I’ve tried that, but neither secret=“thisismysecret” nor requiredSecret=“thisismysecret” worked. I had to explicitly set secretRequired=“false”, and that made it for now. This wouldn’t cause a big impact in our configuration/setup, because only the admin have access to the server and webroots. This is OK for now.

That shouldn’t be the case. All I am doing now is changing server.xml and boncodeAJP13.settings. Maybe secretRequired=“true” creates/levels up to higher strict file permissions… really can’t say right know. Will try/error more about this soon. For now lots of thanks to you and all here helping out of this.

That’s right because the default has changed in the patched version. This solves the problem but obviously isn’t ideal - but as long as you’ve locked down direct access to Tomcat over AJP should be ok I guess.

Well you’ve also upgraded Tomcat, surely? But I wouldn’t expect file permissions to have changed, it’s just that’s what 403 errors are usually caused by hence the suggestion to check.

Last thing, did you restart Lucee/Tomcat? Sorry if that’s obvious, but it’s required to pick up changes to server.xml.

Yes, I did. No need to worry about questioning what ever comes to your mind. Any idea/hint about what ever could be going on is always very appreciated and warmly welcomed.

I can live with that configuration for now, because it isn’t any different from before. Of course enhancing security on AJP with secret key would be much better. Going to dig into this issue as soon as I can. Still glad you’ve identified the problem pretty fast and shared it here. Thanks!

Have you tried updating your boncode connector to the current version (1.0.41)?

1 Like

Definitely worth trying. I had one site which was using an old Boncode version by mistake and that did produce the 403 Tomcat error. Updating to the latest fixed it.

2 Likes

Good news on the Apache Httpd side of things

mod_proxy_ajp: patch to set worker secret passed to tomcat
https://bz.apache.org/bugzilla/show_bug.cgi?id=53098

Backported today to 2.4.x as r1874456, will be part of 2.4.42

This is totally embarrassing to post, but I got to do this: I was adding the settings in the BonCodeAJP13.settings of the installation files all the time, AND NOT in C:\windows\BonCodeAJP13.settings. Changed it, restarted IIS and Lucee and all worked. Sigh… frustrating… Really, sorry for bothering you with my stupidity. Zillions of apologies ( @martin , @Zac_Spitzer, @Julian_Halliwell and every one else here)

4 Likes

If you run Tomcat on Linux and depending on which version you have been before be aware that the file permissions are different with 8.5. This has impact on file uploads with CF.

You might want to check out this and set UMASK accordingly:

|Version|Lucee 5.2.9.31|

|Betriebssystem|Windows Server 2012 R2 (6.3) 64bit|
|Servlet Container|Apache Tomcat/8.5.51|
|Java|1.8.0_242 (Azul Systems, Inc.) 64bit|
|Architecture|64bit|
IIS 8.5.96.00

We have seen a significant impact in TTFB (Server Response Time viewed in Googled Dev Tool Network) after the patch. A blank cfm page over port 8888 was responding in a couple of milliseconds. None cached images over IIS https about 25ms. But when retrieving the same blank cfm page over IIS (AJP with Boncode server.xml connector attribute secretRequired=“false”) all requests need over 1.03 Seconds (always). Downgraded to Tomcat Apache Tomcat/8.0.28 for testing and TTFB came back to 6ms. This performance impact is very probably related to the patch. Can somebody confirm that?