AjpProcessor.service Error processing request java.lang.NullPointerException

dev support
1

This 2012 Server has around 110 websites on it, we have 4 other servers with around 5 sites each, same OS, Java, Tomcat and Lucee versions running IIS. However, this is the only server that has any issue. If you reboot the box, any website on that server will throw a 503 Server Unavailable error.

Notification: ExecuteRequestHandler
Module ManagedPipelineHandler
Handler BonCode-Tomcat-CFM-Handler
Error Code 0x00000000

Sometimes after you refresh the page you’ll get a 500 error

Module ManagedPipelineHandler
Notification ExecuteRequestHandler
Handler BonCode-Tomcat-CFM-Handler
Error Code 0x00000000

On the third refresh, the site will load fine. No further issues until the server is rebooted.

The only thing I see in the tomcat logs are as follows:

10-Aug-2021 16:46:58.707 SEVERE [ajp-nio-127.0.0.1-8009-exec-789] org.apache.coyote.ajp.AjpProcessor.service Error processing request
java.lang.NullPointerException
at org.apache.catalina.connector.Response.sendRedirect(Response.java:1324)
at mod_cfml.core.doRedirect(core.java:551)
at mod_cfml.core.invoke(core.java:536)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:432)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Unknown Source)

Not sure if that error is related, but I do not see this log entree in any of the other servers. This was also an issue with the previous version of Lucee. I’m assuming it’s an error with Boncode though I have no clue on where to start to fix the problem.

Thank you!

OS: Windows Server 2012 R2
Java Version: 11.0.11 (AdoptOpenJDK) 64bit
Tomcat Version: Apache Tomcat/9.0.35
Lucee Version: Lucee 5.3.8.201

2

Dumb question, but are you running IIS on that server that is having the issue?

3

Correct. Sorry I didn’t specify.

4

The issue you have, We have on multiple boxes running Windows X, where X is 2012R2 - current

first check to make sure WinRoot\system32\drivers\etc\hosts file exists and contains an uncommented entry for
127.0.0.1 localhost BOXNAME BOXFQDN

Example
127.0.0.1 localhost server114 server114.example.com

I am going to just guess the system has more than 1 NIC, as that usually is the case with most of these boxes having an issue.

If and when you upgrade to windows 2016 or later you will want to either do this or disable DNS round robin, which you can look at the entry here

5

This box has only 1 nic card, and yes that entry is commented out for 127.0.0.1, however, I’m unclear on what to use for the fqdn of the box as it’s a VM at aws.

6

Awww the plot thickens.

Put 127.0.0.1 localhost

save it, then flushDNs and it will just work.

7

I’m still getting the same issues and errors

11-Aug-2021 14:33:09.379 SEVERE [ajp-nio-127.0.0.1-8009-exec-506] org.apache.coyote.ajp.AjpProcessor.service Error processing request
java.lang.NullPointerException
at org.apache.catalina.connector.Response.sendRedirect(Response.java:1324)
at mod_cfml.core.doRedirect(core.java:551)
at mod_cfml.core.invoke(core.java:536)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:432)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Unknown Source)

8

Which version of mod cfml value.jar ?

9

1.1.11

10

turn on logging

11

The only error I see in the logs is: [mod_cfml][67] ERROR (sent to client): 503: Time Between Contexts has not been fulfilled. Please wait a few moments and try again.

This would be normal behavior as I’ve not changed timeBetweenContexts=0, however, I normally didn’t see this error thrown in IIS. Safe to assume this is a non-issue?

12

Does this happen only after reboot? Looks like mod_cfml is having a long time to verify all the contexts, but I’m not really sure.

After reboot Lucee/Tomcat may be reinspecting the web contexts of your 110 websites. Also, your templates will be recompiled on each first request (by default). If you have a good amount of websites that also have a good amount of well known/popular URLs (visitors and bots), templates may be getting requested again and again during all that web context and template inspection/compilation occurs.

Usually these requests would just time out by Tomcat, but if you have tweaked the default IIS or tomcat settings, it’s possible that IIS 500errors will be generated before that happens.

To rule that out, I’d try rebooting the server with port 80/443 closed/blocked per firewall in such a manner that only you can retrieve the URLs locally. Then monitor your catalina.out and wait until Tomcat finishes every web context scan and Tomcat gets fully available. Then try opening URLs locally over port 80/443 and check if the issue persists.

The steps above are only to identify the issue. If identified you can take further steps to avoid the 500errors.

13

This could be related to mod_cfml context creation time.
From what I understand the mod_cfml service will require some time to create contexts dynamically especially for 110 websites. This will result in a startup delayed response.

To test this you can create one tomcat context manually and see how fast that site will be available after your initial call and whether you experience the same errors.

14

Turns out it is mod_cfml creation time. Thanks, everyone!

15

@River_Bender I’ve run into the same error message. What did you change with mod_cfml and the creation time? Thanks!

16

Not that I am that person, but
timeBetweenContexts=“2000” is the default
set that to
timeBetweenContexts=“4”

Per the doc
The default setting of 2000 means “a maximum of 1 new website per 2 seconds may be added to Tomcat”

17

Sorry for the delay. In server.xml modify this line: <Valve className=“mod_cfml.core” loggingEnabled=“true” maxContexts=“200” timeBetweenContexts=“100” … set the ‘timeBetweenContexts’ to something other than the default 2000 or 2 seconds.

18

Thanks @Terry_Whitney @River_Bender! Unfortunately, that didn’t fix the issue.

I’m trying to run the latest Lucee prod build on W2012, Apache24, Tomcat/Lucee with the below configurations. I’ve followed the various Viviotech and blog articles referenced in this and other chains but don’t seem to be able to get Apache and Tomcat to process CF requests. This post doesn’t use the Ajp connector so I tried that with no success. My current Apache error message is:

[Tue Jan 11 17:57:30.607265 2022] [proxy:error] [pid 7080:tid 1020] (OS 10061)No connection could be made because the target machine actively refused it. : AH00957: http: attempt to connect to 127.0.0.1:8888 (127.0.0.1) failed

HTTPD (standard install with the below conf updates)

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so

<VirtualHost *:80>
  ServerName xxx.org
  DocumentRoot "C:/lucee/tomcat/webapps/mpm"
  <Directory "C:/lucee/tomcat/webapps/mpm">
	  DirectoryIndex index.cfm
	  AllowOverride All
      Require all granted
  </Directory>
</virtualhost>
<VirtualHost localhost>

</virtualhost>
<IfModule mod_proxy.c>
        ProxyPreserveHost On
        ProxyPassMatch ^/(.+\.cf[cm])(/.*)?$ http://127.0.0.1:8888/$1$2
        ProxyPassMatch ^/(.+\.cfchart)(/.*)?$ http://127.0.0.1:8888/$1$2
        ProxyPassMatch ^/(.+\.cfml)(/.*)?$ http://127.0.0.1:8888/$1$2
        ProxyPassReverse / http://127.0.0.1:8888/
        #ProxyPassMatch ^/(.+\.cf[cm])(/.*)?$ ajp://127.0.0.1:8009/$1$2
</IfModule>

LoadModule modcfml_module modules/mod_cfml.so
CFMLHandlers ".cfm .cfc .cfml"
ModCFML_SharedKey "xxx"

server.xml

    <Connector protocol="AJP/1.3"
	    port="8009"
	    secret="xxx"
	    secretRequired="true"
               redirectPort="8443" />

	  <Host name="xxx.org" appBase="webapps"
			unpackWARs="true" autoDeploy="true"
			xmlValidation="false" xmlNamespaceAware="false">
			<Context path="" docBase="C:/lucee/tomcat/webapps/mpm" />
			
	  </Host>

      <Host name="127.0.0.1"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">
			
			<Context path="" docBase="C:/lucee/tomcat/webapps/mpm" />
			<Alias>localhost</Alias>
			
			<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"  
               prefix="localhost_access_log." suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b"/>
			
			<Valve className="mod_cfml.core"
			loggingEnabled="true"
			maxContexts="200"
			timeBetweenContexts="20000"
			scanClassPaths="false"
					responseCode="307"
			sharedKey="xxx"/> 
			

		  </Host>
19

Mod_Proxy and Mod_APJ do nearly the same thing, use one not both in this instance.

Fast Method
Start fresh put everything back to their prospective defaults.
Fire up lucee. You should be able to connect to your lucee instance on local:8888 (verify this step)
Add an entry in /winroot/system32/drivers/etc/hosts
Xxx.Xxx.Xxx.Xxx xxx.org , ie the real IP address of your external or proxied interface for your application you are trying to host, example if 123.45.67.89 was foo.com you would have
123.45.67.89 foo.com www.foo.com anyothernameicanthinkof.foo.com

in your apache.config

<VirtualHost *:80>
  ServerName xxx.org
  DocumentRoot "C:/lucee/tomcat/webapps/mpm"
  <Directory "C:/lucee/tomcat/webapps/mpm">
	  DirectoryIndex index.cfm
	  AllowOverride All
      Require all granted
  </Directory>
<IfModule mod_proxy.c>
    <Proxy *>
    Allow from 127.0.0.1
    </proxy>
    ProxyPreserveHost On
    ProxyPassMatch ^/(.+\.cf[cm])(/.*)?$ http://127.0.0.1:8888/$1$2
    ProxyPassMatch ^/(.+\.cfchart)(/.*)?$ http://127.0.0.1:8888/$1$2
    ProxyPassMatch ^/(.+\.cfml)(/.*)?$ http://127.0.0.1:8888/$1$2
    # optional mappings
    #ProxyPassMatch ^/flex2gateway/(.*)$ http://127.0.0.1:8888/flex2gateway/$1
    #ProxyPassMatch ^/messagebroker/(.*)$ http://127.0.0.1:8888/messagebroker/$1
    #ProxyPassMatch ^/flashservices/gateway(.*)$ http://127.0.0.1:8888/flashservices/gateway$1
    #ProxyPassMatch ^/openamf/gateway/(.*)$ http://127.0.0.1:8888/openamf/gateway/$1
    #ProxyPassMatch ^/rest/(.*)$ http://127.0.0.1:8888/rest/$1
    ProxyPassReverse / http://127.0.0.1:8888/
</ifmodule>

</virtualhost>

In Server.XML
comment out

 secret="xxx"

You do not need this, in this instance.

20

Thanks @Terry_Whitney! I did a clean install and followed your configurations. I’m now receiving a 403 error:

[Wed Jan 12 11:03:15.017716 2022] [authz_core:error] [pid 6368:tid 1000] [client xx.xx.xx.xxx:49713] AH01630: client denied by server configuration: proxy:http://127.0.0.1:8888/lucee/admin/web.cfm

Here are my configs:

HTTPD

<VirtualHost *:80>
  ServerName xxx.org
  DocumentRoot "C:/lucee/tomcat/webapps/mpm"
  <Directory "C:/lucee/tomcat/webapps/mpm">
	  DirectoryIndex index.cfm
	  AllowOverride All
      Require all granted
  </Directory>
<IfModule mod_proxy.c>
    <proxy *>
    Require ip 127.0.0.1
    </proxy>
    ProxyPreserveHost On
    ProxyPassMatch ^/(.+\.cf[cm])(/.*)?$ http://127.0.0.1:8888/$1$2
    ProxyPassMatch ^/(.+\.cfchart)(/.*)?$ http://127.0.0.1:8888/$1$2
    ProxyPassMatch ^/(.+\.cfml)(/.*)?$ http://127.0.0.1:8888/$1$2
    # optional mappings
    #ProxyPassMatch ^/flex2gateway/(.*)$ http://127.0.0.1:8888/flex2gateway/$1
    #ProxyPassMatch ^/messagebroker/(.*)$ http://127.0.0.1:8888/messagebroker/$1
    #ProxyPassMatch ^/flashservices/gateway(.*)$ http://127.0.0.1:8888/flashservices/gateway$1
    #ProxyPassMatch ^/openamf/gateway/(.*)$ http://127.0.0.1:8888/openamf/gateway/$1
    #ProxyPassMatch ^/rest/(.*)$ http://127.0.0.1:8888/rest/$1
    ProxyPassReverse / http://127.0.0.1:8888/
</ifmodule>

</virtualhost>

SERVER

<Connector protocol="AJP/1.3"
	    port="8009"
               redirectPort="8443"
		secretRequired="false" />

      <Host name="127.0.0.1"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">
			
			<Context path="" docBase="C:/lucee/tomcat/webapps/mpm" />
			<Alias>localhost</Alias>
			
			<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"  
               prefix="localhost_access_log." suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b"/>
			
			<Valve className="mod_cfml.core"
			loggingEnabled="true"
			maxContexts="200"
			timeBetweenContexts="2000"
			scanClassPaths="false"
					responseCode="307"
			sharedKey="xxx"/> 

		  </Host>