Logging on Tomcat Servlet via a custom java agent no longer working with Tomcat 9

Hi everyone,
i have issue on Lucee 5.3.9.160
i have created a jar (Agent) for capturing log request on tomcat servlet like SeeFusion.
on lucee 5.2.9.31 Java 1.8 and Tomcat 8, it can be capturing request on tomcat servlet. but if i implement on Lucee 5.3.9.160, Java 10 and Tomcat 9, can’t capturing log request. as well as seefusion that i inserted to Lucee 5.3.9. its not showed up log request on servlet.

*No log error, i don’t find any error on catalina.out.

OS: Linux Centos 7.9
Java Version: 10
Tomcat Version: 9
Lucee Version: > 5.3.

without seeing your code it’s hard to help…

does it still work with 5.3.9 on Tomcat 8? then I’d be checking the changelog for Tomcat 9?

hi zack, thanks for your resp.

i think, the issue is not coming from on my agent jar. because if i curl on lucee server on port 8888 (to servlet), my agent / seefusion capturing request the request

if i hit from outside, the request can’t capturing.

my question :

  1. is there anything i need to check on the connector between tomcat and httpd ?
  2. do i need to open port 8888 too ?
    here my nestat
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1042/sshd
tcp        0      0 127.0.0.1:8088          0.0.0.0:*               LISTEN      1050/influxd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1432/master
tcp6       0      0 :::46422                :::*                    LISTEN      18331/java
tcp6       0      0 :::8086                 :::*                    LISTEN      1050/influxd
tcp6       0      0 :::22                   :::*                    LISTEN      1042/sshd
tcp6       0      0 :::8888                 :::*                    LISTEN      18331/java
tcp6       0      0 ::1:25                  :::*                    LISTEN      1432/master
tcp6       0      0 :::8707                 :::*                    LISTEN      18331/java
tcp6       0      0 127.0.0.1:8005          :::*                    LISTEN      18331/java
tcp6       0      0 :::8999                 :::*                    LISTEN      18331/java
tcp6       0      0 127.0.0.1:8009          :::*                    LISTEN      18331/java
tcp6       0      0 :::80                   :::*                    LISTEN      18557/httpd
  • How/where have you added that jar agent?
  • how is your Lucee connected to Tomcat? Reverse Proxy or AJP? How did you install/update Tomcat and Lucee? Are you using mod_cfml? Are there any changes that have been made in difference to the former Tomcat config files at path-to-installation/tomcat/conf/catalina/ ?

Smells to me like that agent jar is only available at localhost on port 8888.

hi andreas, thanks for reply

i put the agent to /opt/lucee/tomcat/lib/

with AJP, after i check on the server.xml i found a different value.
on our lucee 5.2.9.31 there’s attribute

<Connector port="8888" protocol="HTTP/1.1"
connectionTimeout="20000" secret="{secret}" secretRequired="true" redirectPort="8443" />

but on lucee 5.3.9.160

<Connector port="8888" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443"
               maxHttpHeaderSize="52246" 
               relaxedPathChars="[]|"
               relaxedQueryChars="[]{,}" />

is any mistake ?

i extract it from site to the server

yes, this my mod cfml value on server.xml

<Valve className="mod_cfml.core"
		loggingEnabled="false"
		maxContexts="200"
		timeBetweenContexts="2000"
		scanClassPaths="false"
                responseCode="307"
		sharedKey="SECRET"
		/> 

and here’s where i get shared key from httpd

nope, im not update anything on this path.

**im new with lucee server, the previous server was done by someone else.

According to your httpd this is not an AJP connection, but a http proxy reverse to port 8888. Regarding attributes:

I always thought these are for AJP connectors. Don’t know if these attributes are ignored on http protocol. But as you said these settings were working, doesn’t sound this could be the issue.

Because you said, the agent works when you access your localhost:8888 directly, it could have something to do within the proxy passing connection. Proxy reverse from apache strips down some data from the http stream before directing it to tomcat. You might need to add additional headers for tomcat, e.g. a x-forward header. But you should have seen that settings and valves in your previous server.xml then.

Also, is it possible that that agent library is available on localhost:8888 only, but not to your web contexts (e.g. yourdomain:8888 where the logging is not working?

hi andreas

sry im late to reply this topic
i have explore on httpd / apache as a proxy server, but i dont have any additional clue for my problems.
but sometimes, after i restart the lucee tomcat and i leave this server (e.g overnight). the lucee can captured the log request. its weird and make me depressed :frowning:

this is for what i tried

  1. conf file (custom on conf.d on httpd, extra on apache) :
Header set Access-Control-Allow-Origin "*"
Header set Access-Control-Allow-Methods "POST, GET, OPTIONS, DELETE, PUT"
Header set Access-Control-Max-Age "1000"
Header set Access-Control-Allow-Headers "*"
.
.
.
Alias /s "/media/sf7/main/src/web/"
<Directory "/media/sf7/main/src/web/">
  DirectoryIndex index.cfm
  Require all granted
</Directory>

because you said

so i’ve tried to change this headers.

  1. server.xml tomcat
<!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector protocol="AJP/1.3"
	    port="8009"
	    secret="e983b02ec44d7e271a5ce33fc8a359e0*********"
	    secretRequired="true"
	    redirectPort="8443" Address="::"/>

**im not sure with this server.xml, does it affect or not. because, i compare the port 8009 on previous lucee version.

  1. Change JVM Version from 10.44 to 10.0.2+13

What version of HTTPD? Any third party install such as CODEIT or REMI or just the patched vanilla version?

make sure the proxy modules are installed

httpd -S | grep proxy

make sure they are not commented out in /etc/httpd/cond.modules.d/00-proxy.conf

make sure its not SELINUX screwing with your installation,
setenforce 0

then
systemctl restart httpd
systemctl restart lucee_ctl (or whatever you have lucee running as)

if that fails, look at /var/log/httpd/ error logs and move your proxy settings after modcfml, not before.

1 Like

hi Terry, thanks for your time

this is my version of httpd

Server version: Apache/2.4.6 (CentOS)
Server built:   Mar 24 2022 14:57:57

Nope

Mutex proxy-balancer-shm: using_defaults
Mutex proxy: using_defaults

is this ?

on my httpd loadmodule proxy is active

cat 00-proxy.conf
# This file configures all the proxy modules:
LoadModule proxy_module modules/mod_proxy.so
LoadModule lbmethod_bybusyness_module modules/mod_lbmethod_bybusyness.so
LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so
LoadModule lbmethod_bytraffic_module modules/mod_lbmethod_bytraffic.so
LoadModule lbmethod_heartbeat_module modules/mod_lbmethod_heartbeat.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule proxy_express_module modules/mod_proxy_express.so
LoadModule proxy_fcgi_module modules/mod_proxy_fcgi.so
LoadModule proxy_fdpass_module modules/mod_proxy_fdpass.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_scgi_module modules/mod_proxy_scgi.so
LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so

if using apache, i can’t found this file on the path.
but, i found similiar name

/usr/local/apache2/conf/extra/proxy-html.conf

i have run this step, log still not captured on tomcat.
here’s output of error_log

[Thu Oct 13 17:48:01.671095 2022] [mpm_event:notice] [pid 27964:tid 140631257417536] AH00492: caught SIGWINCH, shutting down gracefully
[Thu Oct 13 17:48:43.826655 2022] [ssl:warn] [pid 32608:tid 139802047801152] AH01873: Init: Session Cache is not configured [hint: SSLSessionCache]
[Thu Oct 13 17:48:43.828071 2022] [http2:info] [pid 32608:tid 139802047801152] AH03090: mod_http2 (v1.15.24, feats=CHPRIO+SHA256+INVHD+DWINS, nghttp2 1.46.0), initializing...
[Thu Oct 13 17:48:43.829605 2022] [mpm_event:notice] [pid 32608:tid 139802047801152] AH00489: Apache/2.4.51 (Unix) OpenSSL/1.1.1l configured -- resuming normal operations
[Thu Oct 13 17:48:43.829642 2022] [core:notice] [pid 32608:tid 139802047801152] AH00094: Command line: '/usr/local/apache2/bin/httpd'

I would start by making sure your custom code works on tomcat/lucee port 8888
if you havent already, open up port 8888 to test
sudo ufw allow 8888

Then in your ssl.conf for your vhost or main config you need the lines
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 300

This will resolve the issue with sslCache not defined in your log
This will help mod_http2 work correctly

As for your jar, you may want to post the code, it only helps the community and there are several ColdFusion experts who visit these forums who may have insight into correcting the java code.

i was tested this port to open public. and it can’t be answer for my issue.

i am testing with SeeFusion jar version 5, not with my own agent jar.

anyway, i’ve added this line on httpd.conf

        ProxyPass / http://127.0.0.1:8888/

and this is error_log, when i open the page on external server.

[Mon Oct 17 16:00:14.745082 2022] [authz_core:debug] [pid 40943] mod_authz_core.c(809): [client 10.128.0.1:62607] AH01626: authorization result of Require all granted: granted, referer: http://192.168.101.115:32313/
[Mon Oct 17 16:00:14.745129 2022] [authz_core:debug] [pid 40943] mod_authz_core.c(809): [client 10.128.0.1:62607] AH01626: authorization result of <RequireAny>: granted, referer: http://192.168.101.115:32313/
[Mon Oct 17 16:00:14.745182 2022] [proxy:debug] [pid 40943] mod_proxy.c(1123): [client 10.128.0.1:62607] AH01143: Running scheme http handler (attempt 0), referer: http://192.168.101.115:32313/
[Mon Oct 17 16:00:14.745188 2022] [proxy_ajp:debug] [pid 40943] mod_proxy_ajp.c(722): [client 10.128.0.1:62607] AH00894: declining URL http://127.0.0.1:8888/sf7/, referer: http://192.168.101.115:32313/
[Mon Oct 17 16:00:14.745195 2022] [proxy_fcgi:debug] [pid 40943] mod_proxy_fcgi.c(972): [client 10.128.0.1:62607] AH01076: url: http://127.0.0.1:8888/sf7/ proxyname: (null) proxyport: 0, referer: http://192.168.101.115:32313/
[Mon Oct 17 16:00:14.745200 2022] [proxy_fcgi:debug] [pid 40943] mod_proxy_fcgi.c(975): [client 10.128.0.1:62607] AH01077: declining URL http://127.0.0.1:8888/sf7/, referer: http://192.168.101.115:32313/
[Mon Oct 17 16:00:14.745207 2022] [proxy:debug] [pid 40943] proxy_util.c(2221): AH00942: HTTP: has acquired connection for (127.0.0.1)
[Mon Oct 17 16:00:14.745213 2022] [proxy:debug] [pid 40943] proxy_util.c(2274): [client 10.128.0.1:62607] AH00944: connecting http://127.0.0.1:8888/sf7/ to 127.0.0.1:8888, referer: http://192.168.101.115:32313/
[Mon Oct 17 16:00:14.745222 2022] [proxy:debug] [pid 40943] proxy_util.c(2454): [client 10.128.0.1:62607] AH00947: connected /sf7/ to 127.0.0.1:8888, referer: http://192.168.101.115:32313/
[Mon Oct 17 16:00:14.745323 2022] [proxy:debug] [pid 40943] proxy_util.c(2690): AH00951: HTTP: backend socket is disconnected.
[Mon Oct 17 16:00:14.745756 2022] [proxy:debug] [pid 40943] proxy_util.c(2830): AH02824: HTTP: connection established with 127.0.0.1:8888 (127.0.0.1)
[Mon Oct 17 16:00:14.745773 2022] [proxy:debug] [pid 40943] proxy_util.c(2997): AH00962: HTTP: connection complete to 127.0.0.1:8888 (127.0.0.1)
[Mon Oct 17 16:00:14.838144 2022] [proxy:debug] [pid 40943] proxy_util.c(2236): AH00943: http: has released connection for (127.0.0.1)

Without the configuration files or knowing what your code does its hard to tell what you have going on but It looks like you have a springboot application.

Does your java jar, run as a spingboot service and need its own port? hence the 32313 port?

How I would configure this
Bind your service to 127.0.0.1 or 127.0.0.2
Add entry to /etc/hosts for your service, such as myjarapp.mydomain.moo
make sure all traffic on loop back is accepted
If your app needs its own port, and its own service, then it needs to run in the same user/group as lucee or have access to the same user/group as lucee (for testing, use or apache.apache)
then use curl or lynx at the command to check the status of your app. if its running and working, then reconfigure your httpd.config to allow for a proxy either as a subdomain or as a directory.

If its just a JAR FILE, then you should post your jar code as literally over half the ColdFusion experts hit this forum on a daily basis and could point how to fix this or offer services on how to fix this if its something you feel you cant share as its some product you want to sell.

okey, this is my code on my own jar.

this my class TomcatValve

public class TomcatValve implements ContainerListener {

	private static final Logger LOG = Logger.getLogger(TomcatValve.class.getName());
	private FilterDef filterDef;
	private FilterMap filterMap;

	public TomcatValve(Container con) {

		LOG.info("Tomcat Valve Started");
		try {
			this.filterDef = new FilterDef();
			
			this.filterDef.setFilterClass("com.monitor.agent.connector.servlet.MonitorFilter");
			this.filterDef.setFilterName("MonitorAgent");

			this.filterMap = new FilterMap();
			this.filterMap.addURLPattern("/*");
			this.filterMap.setFilterName("MonitorAgent");

			LOG.info("Tomcat name " + con.getName());

			addFilterToContainer(con);
			con.addContainerListener(this);
		}catch (Exception e){
			LOG.info("Something err in Tomcat Valve " + e.getStackTrace().toString());
		}
	}

	public void containerEvent(ContainerEvent containerEvent) {
		try {
			if ((!containerEvent.getType().equalsIgnoreCase("removeValve"))
					&& (!containerEvent.getType().equalsIgnoreCase("removeChild"))) {
				addFilterToContainer(containerEvent.getContainer());
			}
		}catch (Exception e){
			LOG.info("Something error on containerEvent" + e.getMessage().toString());
		}
	}

	private void addFilterToContainer(Container con) {
		try {
			StandardContext cx;
			if ((con instanceof StandardContext)) {
				cx = (StandardContext) con;
				
				if (cx.findFilterDef("MonitorAgent") == null) {
					cx.addFilterDef(this.filterDef);
					cx.addFilterMapBefore(this.filterMap);
					cx.filterStop();
					cx.filterStart();
				}
			} else {
				for (Container child : con.findChildren()) {
					if (!hasMeListening(con.findContainerListeners())) {
						con.addContainerListener(this);
					}
					addFilterToContainer(child);
				}
			}
		}catch (Exception e){
			LOG.info("Something error on addFilterToContainer " + e.getMessage().toString());
		}
	}

my class MonitorFilter

public class MonitorFilter implements Filter {

    private static final Logger LOG = Logger.getLogger(MonitorFilter.class.getName());
    private MonitorAgent agent;
    private AppConfig appConfig;


    @Override
    public void init(FilterConfig filterConfig) throws ServletException {
        try {
            this.agent = MonitorAgent.INSTANCE;
        }catch (Exception e){
            LOG.info("Error INIT " +e.getMessage());
        }
    }

    @Override
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
        if (request instanceof HttpServletRequest && response instanceof HttpServletResponse) {
            HttpServletRequest httpRequest = (HttpServletRequest) request;
            HttpServletResponse httpResponse = (HttpServletResponse) response;
            boolean isSecure = httpRequest.isSecure();
            String method = httpRequest.getMethod();
            String queryString = httpRequest.getQueryString();
            String pathInfo = WebUtil.getRequestPathInfo(httpRequest);
            String remoteAddr = WebUtil.getClientIpAddress(httpRequest);
            String headerSeeProfile = httpRequest.getHeader("X-SeeProfile");
            String reqPage = WebUtil.getCookieByName(httpRequest, "REQPAGE");
            int serverPort = httpRequest.getServerPort();
            RequestThread thread = RequestThread.getCurrentRequest();
            if (thread == null) {
                RequestInfo info = new RequestInfo(remoteAddr, httpRequest.getRemoteHost(), httpRequest.getServerName(), httpRequest.getRequestURI(), pathInfo, method, queryString, headerSeeProfile, reqPage, isSecure, serverPort);
                thread = new RequestThread(info);
                this.agent.addRequestToList(thread);
            }
            RequestWrapper requestWrapper = new RequestWrapper(thread, httpRequest);
            ResponseWrapper responseWrapper = new ResponseWrapper(thread, httpResponse);
            try {
                chain.doFilter((ServletRequest)requestWrapper, (ServletResponse)responseWrapper);
                thread.getRequestInfo().setResponseCode(responseWrapper.getStatus());
            } catch (IOException e) {
                LOG.log(Level.SEVERE, "IOException", e);
            } catch (ServletException e) {
                LOG.log(Level.SEVERE, "ServletException", (Throwable)e);
            } finally {
                this.agent.removeRequestFromList(thread);
            }
        } else {
            chain.doFilter(request, response);
        }
    }

    @Override
    public void destroy() {    }
}

this is my class Monitor Valve

public class MonitorValve extends ValveBase implements Lifecycle, ContainerListener {

    private static final Logger LOG = Logger.getLogger(MonitorValve.class.getName());

    private ContainerListener valve;
    private MonitorAgent agent;

    public void initInternal() throws LifecycleException {
        String version = ServerInfo.getServerNumber();
        LOG.info("MonitorValve: Loading Tomcat Valve for Tomcat " + version);
        try {
            this.valve = new TomcatValve(getContainer());
            super.initInternal();
        } catch (Exception e) {
            LOG.info("Something went wrong Loading Tomcat Valve" + e.getMessage());
        }

    }

    public void invoke(Request request, Response response) throws IOException, ServletException {
        LOG.info("Invoke Request " + request.getContext().getName());
            getNext().invoke(request, response);
    }

    public void containerEvent(ContainerEvent containerEvent) {
        LOG.info("To Container Event " + containerEvent.toString());
        this.valve.containerEvent(containerEvent);
    }

}

**for this case, im not using with my own jar. i test it with SeeFusion to capture log request on servlet tomcat.