Logging in JSON format

yamikuronue · April 12, 2018, 2:10pm

Hiya. I have a weird edge case problem I was hoping someone could help me with? I have a number of legacy apps on Lucee that use cflog tags for various reasons. I’m currently piping all logs (application and Lucee’s own logs) to Graylog by watching folders on disk and collecting the output, but the logs are annoying to parse in Graylog and we’re missing information. Ideal case, all the logs across the whole system would be in json format, written to disk in a known location, so that I can easily parse them in Graylog.

It looks like I can install something like log4j and add a log4j.properties file, but it’s not going to affect the cflog outputs. I can see people redirecting standard output to a udp stream, but for various reasons I can’t send logs over udp to graylog, they have to be on disk. It looks like in the lucee-web.xml.cfm there’s some extra options that might help me here, but I don’t understand how they work. I could also install something like Logbox, but I’d need to change all the current logging tags to make it work.

Any ideas?

bdw429s · April 12, 2018, 2:16pm

Thanks @yamikuronue! @isapir or @micstriit can one (or both) of you chime in on this one? I’m not sure there’s any other people who understand enough about how Lucee logs under the covers to answer this.

isapir · April 12, 2018, 4:55pm

I hope that you’re using Lucee 5, which already uses log4j as the underlying logging framework.

Please see the comments in the default lucee-web.xml circa line 180.

github.com

lucee/Lucee/blob/master/core/src/main/java/resource/config/web.xml#L180


      
          <?xml version="1.0" encoding="iso-8859-1"?>
          
          
<cfLuceeConfiguration version="2.0">
          
          
<!-- 
          Path placeholders:
          	{lucee-web}: path to the Lucee web directory typical "{web-root}/WEB-INF/lucee"
          	{lucee-server}: path to the Lucee server directory typical where the Lucee.jar is located
          	{lucee-config}: same as {lucee-server} in server context and same as {lucee-web} in web context}
          	{temp-directory}: path to the temp directory of the current user of the system
          	{home-directory}: path to the home directory of the current user of the system
          	{web-root-directory}: path to the web root
          	{system-directory}: path to thesystem directory
          	{web-context-hash}: hash of the web context
          -->
          
          
    <!--
              arguments:
          		close-connection - 	write connection-close to response header
          		suppress-whitespace	-	supress white space in response
          		show-version - show lucee version uin response header
          	 -->
          	<setting />
          
          
<!--	definition of all database used inside your application. 										-->
          <!--	above you can find some definition of jdbc drivers (all this drivers are included at lucee) 	-->
          <!--	for other database drivers see at: 																-->
          <!--	 - http://servlet.java.sun.com/products/jdbc/drivers 											-->
          <!--	 - http://sourceforge.net 																		-->
          <!--	or ask your database distributor 																-->
          
          
	<data-sources>
          	</data-sources>
          	<mode develop="false"></mode>
          	
          	<resources>
              	<!--
                  arguments:
          		lock-timeout   - 	define how long a request wait for a lock
          	 	-->
              	<resource-provider 
              		scheme="ram"
              		class="lucee.commons.io.res.type.ram.RamResourceProvider"
              		arguments="case-sensitive:true;lock-timeout:1000;"/>
              </resources>
              
              <remote-clients directory="{lucee-web}remote-client/"/>
          	
          	
          	<!--
          		deploy-directory - directory where java classes will be deployed
          		custom-tag-directory - directory where the custom tags are
          		tld-directory / fld-directory - directory where additional Function and Tag Library Deskriptor are.
          		temp-directory - directory for temporary files (upload aso.)
          	 -->
          	<file-system
          		deploy-directory="{lucee-web}/cfclasses/"
          		tld-directory="{lucee-web}/library/tld/"
          		fld-directory="{lucee-web}/library/fld/"
          		temp-directory="{lucee-web}/temp/">
          	</file-system>
          	
          	<!--
          	scope configuration:
          	
          		cascading (expanding of undefined scope)
          			- strict (argument,variables)
          			- small (argument,variables,cgi,url,form)
          			- standard (argument,variables,cgi,url,form,cookie)
          			
          		cascade-to-resultset: yes|no
          			when yes also allow inside "output type query" and "loop type query" call implizid call of resultset
          			
          		merge-url-form:yes|no
          			when yes all form and url scope are synonym for both data
          		
          		client-directory:path to directory where client scope values are stored
          		client-directory-max-size: max size of the client scope directory
          	-->
          	<scope 
          		client-directory="{lucee-web}/client-scope/"
          		client-directory-max-size="100mb"/>
          		
          	<mail>
          	</mail>
          	
          	
          	<!--
          	define path to scedule task directory
          		directory: path
          	-->	
          	<scheduler 
          		directory="{lucee-web}/scheduler/"/>
          	
          	<mappings>
          	<!--
          	directory mapping:
          		
          		trusted: yes|no
          			trusted cache -> recheck every time if there are changes in the called cfml file or not.
          		virtual:
          			virtual path of the application
          			example: /somedir/
          			
          		physical: 
          			physical path to the apllication
          			example: d:/projects/app1/webroot/somedir/
          			
          		archive:
          			path to an archive file:
          			example: d:/projects/app1/rasfiles/somedir.ras
          		primary: archive|physical
          			define where lucee first look for a called cfml file.
          			for example when you define physical you can partiquel overwrite the archive.
          		-->
          		<mapping
          			readonly="yes"
          			trusted="true"
          			virtual="/lucee/"
          			physical="{lucee-web}/context/"
          			archive="{lucee-web}/context/lucee-context.lar"
          			primary="physical"
          			toplevel="yes"/>
          	</mappings>	
          	
          	<custom-tag>
          		<mapping
          			trusted="yes"
          			physical="{lucee-web}/customtags/"/>
          	</custom-tag>
          	
          	<ext-tags>
          		<ext-tag type="java" name="HelloWorld" class="lucee.cfx.example.HelloWorld"/>
          	</ext-tags>
          	
          	<!--
          	component:
          		
          		base: 
          			path to base component for every component that have no base component defined 
          		data-member-default-access: remote|public|package|private
          			access type of component data member (variables in this scope)
          		use-shadow: if true component variable scope has a second scope, not only the this scope
          	-->
          	<component 
                  base="/lucee/Component.cfc"
          		data-member-default-access="public"
          		use-shadow="yes"> 
          	</component>
          	
          	<!--
          	regional configuration:
          		
          		locale: default: system locale
          			define the locale 
          		timezone: default:maschine configuration
          			the ID for a TimeZone, either an abbreviation such as "PST", 
          			a full name such as "America/Los_Angeles", or a custom ID such as "GMT-8:00". 
          		timeserver: [example: swisstime.ethz.ch] default:local time
          			dns of a ntp time server
          	-->
          	<regional/>
          	
          	<!--
          		enable and disable debugging
          	 -->
          	<debugging
          		template="/lucee/templates/debugging/debugging.cfm"/>
          		
          	<application
          		cache-directory="{lucee-web}/cache/"
          		cache-directory-max-size="100mb"/>
          </cfLuceeConfiguration>

This file has been truncated. show original

You can use XML (yuck!@#) which will be easier to setup and give you the information that you need at the expense of the weight of the XML format.

You can also specify a custom Pattern, or even a custom Class. I haven’t tried that myself, but see java - Does log4j support JSON format? - Stack Overflow for some possible options.

isapir · April 12, 2018, 5:02pm

P.S. Since we support XML format out of the box, I think that we should support JSON as well. Feel free to open a ticket for that in JIRA.

Zackster · April 12, 2018, 5:05pm

the docs need updating for Lucee 5 then

Jamie_Jackson · April 22, 2019, 9:31pm

https://luceeserver.atlassian.net/browse/LDEV-2248

Jamie_Jackson · April 22, 2019, 9:53pm

Can you or anybody else give any more detail on the steps needed to create a new appender and hook it up? I figure each sentence from someone who has any idea what they’re talking about will save me 8 hours of floundering.

I was able to implement JSON logs for Lucee’s tomcat by following this ( Enable JSON Logging in Tomcat 8 with Log4j2) but now I’m trying to move on to the proper Lucee logs.

FYI, I’m doing this so that I can set up a filebeat to shoot logs to Elastic Stack, and I figure it will be easier if all the files start out as JSON (no parsing necessary).

Update: I wonder if it’s possible to use the classes that I used for tomcat. Maybe some way to specify org.apache.logging.log4j.core.layout.JsonLayout in lucee-web.xml? I don’t know what that syntax would look like, though.

Pseudo-XML:

		<logger
			appender="console"
			appender-arguments="path:{lucee-config}/logs/exception.log"
			layout="org.apache.logging.log4j.core.layout.JsonLayout"
			level="info"
			name="exception"
		/>

Brian_Harcourt · October 15, 2020, 3:12pm

Jamie,
Did you reach any resolution on this? I’ve been trying to get JSON logs for a few days now using the logstash log4j-1.2 class and not having any luck. What did you do?

Jamie_Jackson · October 13, 2022, 6:21pm

I’m late to reply, but no, no resolution for JSON logs. I’m basically treating Lucee logs as CSV when parsing them.

Zackster · October 13, 2022, 6:49pm

you may find the datadog layout a good starting point?

github.com

lucee/Lucee/blob/5.3/core/src/main/java/lucee/commons/io/log/log4j2/layout/DataDogLayout.java

package lucee.commons.io.log.log4j2.layout;

import java.io.ByteArrayOutputStream;
import java.io.PrintWriter;
import java.io.StringWriter;
import java.lang.reflect.Method;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;

import javax.servlet.http.Cookie;

import org.apache.logging.log4j.core.LogEvent;
import org.apache.logging.log4j.core.layout.AbstractStringLayout;

import lucee.commons.io.CharsetUtil;
import lucee.commons.io.log.LogUtil;
import lucee.loader.engine.CFMLEngine;
import lucee.loader.engine.CFMLEngineFactory;
import lucee.loader.util.Util;

This file has been truncated. show original

or the sentry extension

dawesi · October 14, 2022, 5:13am

pitty there isn’t a logparser on linux, like on windows… that’s a killer log reader/importer/exporter…

Download Log Parser 2.2 from Official Microsoft Download Center

Jamie_Jackson · October 15, 2022, 2:41pm

Thanks, @Zackster. I probably won’t revisit this myself since my solution has been working adequately, but maybe someone else will pick up the ball.

@dawesi, it’s been a while, but I’ve used lnav for Linux.