TOMCAT CVE-2020-1938: Ghostcat (AJP)

I’ve tried that, but neither secret=“thisismysecret” nor requiredSecret=“thisismysecret” worked. I had to explicitly set secretRequired=“false”, and that made it for now. This wouldn’t cause a big impact in our configuration/setup, because only the admin have access to the server and webroots. This is OK for now.

That shouldn’t be the case. All I am doing now is changing server.xml and boncodeAJP13.settings. Maybe secretRequired=“true” creates/levels up to higher strict file permissions… really can’t say right know. Will try/error more about this soon. For now lots of thanks to you and all here helping out of this.

That’s right because the default has changed in the patched version. This solves the problem but obviously isn’t ideal - but as long as you’ve locked down direct access to Tomcat over AJP should be ok I guess.

Well you’ve also upgraded Tomcat, surely? But I wouldn’t expect file permissions to have changed, it’s just that’s what 403 errors are usually caused by hence the suggestion to check.

Last thing, did you restart Lucee/Tomcat? Sorry if that’s obvious, but it’s required to pick up changes to server.xml.

Yes, I did. No need to worry about questioning what ever comes to your mind. Any idea/hint about what ever could be going on is always very appreciated and warmly welcomed.

I can live with that configuration for now, because it isn’t any different from before. Of course enhancing security on AJP with secret key would be much better. Going to dig into this issue as soon as I can. Still glad you’ve identified the problem pretty fast and shared it here. Thanks!

Have you tried updating your boncode connector to the current version (1.0.41)?

1 Like

Definitely worth trying. I had one site which was using an old Boncode version by mistake and that did produce the 403 Tomcat error. Updating to the latest fixed it.

2 Likes

Good news on the Apache Httpd side of things

mod_proxy_ajp: patch to set worker secret passed to tomcat
https://bz.apache.org/bugzilla/show_bug.cgi?id=53098

Backported today to 2.4.x as r1874456, will be part of 2.4.42

This is totally embarrassing to post, but I got to do this: I was adding the settings in the BonCodeAJP13.settings of the installation files all the time, AND NOT in C:\windows\BonCodeAJP13.settings. Changed it, restarted IIS and Lucee and all worked. Sigh… frustrating… Really, sorry for bothering you with my stupidity. Zillions of apologies ( @martin , @Zackster, @Julian_Halliwell and every one else here)

4 Likes

If you run Tomcat on Linux and depending on which version you have been before be aware that the file permissions are different with 8.5. This has impact on file uploads with CF.

You might want to check out this and set UMASK accordingly:

|Version|Lucee 5.2.9.31|

|Betriebssystem|Windows Server 2012 R2 (6.3) 64bit|
|Servlet Container|Apache Tomcat/8.5.51|
|Java|1.8.0_242 (Azul Systems, Inc.) 64bit|
|Architecture|64bit|
IIS 8.5.96.00

We have seen a significant impact in TTFB (Server Response Time viewed in Googled Dev Tool Network) after the patch. A blank cfm page over port 8888 was responding in a couple of milliseconds. None cached images over IIS https about 25ms. But when retrieving the same blank cfm page over IIS (AJP with Boncode server.xml connector attribute secretRequired=“false”) all requests need over 1.03 Seconds (always). Downgraded to Tomcat Apache Tomcat/8.0.28 for testing and TTFB came back to 6ms. This performance impact is very probably related to the patch. Can somebody confirm that?

yeah, I’m seeing that same problem, with 1.03s per request with 8.5.51, via Apache not IIS

Bug filed here 64182 – 1.03s delay on every request via ajp

I also encountered a problem with quirky config handling from the AJP connector

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" connectionTimeout="-1" secretRequired="false" address="127.0.0.1" secret="zac"/>

Lucee won’t start with this config, if I remove the secret, it works (Apache httpd 2.4 mod_proxy_ajp doesn’t support secrets until the next release)

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" connectionTimeout="-1" secretRequired="false" address="127.0.0.1"/>

It’s been already been filed as a bug 64180 – secretRequred=false is ignored if secret=<anything>

If you’re upgrading Tomcat by just replacing the lib directory, remember to copy across mod_cfml-valve_v1.1.11.jar, if your using an older version of mod_cfml, you can grab the new jar version from mod_cfml/java at master · viviotech/mod_cfml · GitHub

The newest version of mod_cfml supports returning a 307 to the first request (first request always gets a redirect), which is important, as otherwise it returns a 302 to the first request, which if it’s a POST from say a javascript app, will break everything as the POST becomes a GET and drops all the form fields.

1 Like

Does it change behaviour if you specify address="::1" vs address="127.0.0.1" (i.e. ipv6 vs ipv4)?

Thanks for confirming bug and filing/posting it Zac. For specific reasons we still make no use of IPV6 for now and we haven’t any address attribute set on 8009 connector tag. Going to set up this configuration on my dev machine as soon as I can and let you know then.

I’ve just added the setting address="::1" and I can confirm the performance changed. Looks like the delay went away with this setting.

2 Likes

The setting file and directory can be queried from the local system on the server by using a handler command like so:

http://localhost/a.cfm?BonCodeConnectorVersion=true

where a.cfm can be any mapped file extension, e.g. a.jsp, a.cfc etc.

2 Likes

I didn’t see a discussion around implications for running in docker containers and for the official image, itself: How to deal with TOMCAT CVE-2020-1938: Ghostcat (AJP) · Issue #61 · lucee/lucee-dockerfiles · GitHub

Hmm, I don’t appear to be using mod_proxy_ajp/mod_jk, anyway (rather, I’m using mod_proxy), so I think quick solution for me will be to remove this from server.xml:

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

Nope, wrong guess, at least with the RC version I’m running.

Commenting it out (below) didn’t change the message in the logs:

RUN sed \
    -i 's+\(<Connector [^\>]*AJP/[^>]*>\)+<!-- TOMCAT CVE-2020-1938: Ghostcat (AJP) \1 -->+' \
    '/usr/local/tomcat/conf/server.xml'

I think I’ve got to find a snapshot with the fix…

…but I have it already: 9.0.31.0 (as part of image 5.3.5.80-SNAPSHOT-tomcat9.0-jdk11-openjdk).

I have just been bitten with this also. I did a fresh install of Lucee on Windows & IIS using the installer to replace an old Lucee 4.5.

We could not understand why the site was so slow after the install. I remembered this thread and added in the address="::1" attribute and we are back up to speed.

Does anybody know if there is a reason that this is not included as part of the installation?

The Tomcat docs state:

For servers with more than one IP address, this attribute specifies which address will be used for listening on the specified port. By default, the connector will listen on the loopback address. Unless the JVM is configured otherwise using system properties, the Java based connectors (NIO, NIO2) will listen on both IPv4 and IPv6 addresses when configured with either 0.0.0.0 or ::. The APR/native connector will only listen on IPv4 addresses if configured with 0.0.0.0 and will listen on IPv6 addresses (and optionally IPv4 addresses depending on the setting of ipv6v6only) if configured with ::.
http://tomcat.apache.org/tomcat-9.0-doc/config/ajp.html

Perhaps having this setting in by default makes too many assumptions about peoples setup? Maybe it is just something that we need to remember to check following and install.

@martin: There has been already a debate about this here at github. The installer was always (and thankfully) supplied by @Jordan_Michaels (formerly at viviotech). Jordan is not at viviotech anymore and I don’t know if he or another person is supplying/will supply the future installers. But I know somebody supplied the last actual ones. I haven’t heard about Jordan for a long time now. I miss him and I hope he is doing well in these difficult times.

1 Like

@andreas - interesting read, thanks for pointing me to that. I did not know that Jordan was no longer at Viviotech - he has been a familiar name for a long time now.

Two additional points on this, first on Jordan’s move then on the possible future of ajp within tomcat.

First fwiw, here’s a post with a bit more when Jordan announced his move:

https://blog.viviotech.net/stepping-down/

In other “news anyone could have missed” I’ll point out that at tomcatcon 2019 the tomcat team discussed their intent to depreciate and ultimately remove ajp entirely. I expressed surprise and lament for the sake of so many I knew who used it (CFers and Lucee folks, though I didn’t say that as it would not likely help my cause) and for the sake of surely many others who do. They didn’t seem moved.

Igal was there and perhaps as a committer he may have new news. I’ve not followed it. Just thought I’d offer the heads up. Bilal, had you hearf or seen anything, relative to boncode perhaps?

Sorry that these are both a bit of a tangent on the thread. It just seemed an opportune time and audience in which to bring it up.

1 Like

Viviotech is still providing the installers for now. The Lucee installer work was the property of Viviotech and when Jordan moved on, he left it behind and entrusted it to another employee there. The installer builds have also been provided to LAS with the hope of someday getting them automated directly in the builds, but that will require someone with the time to do it.