Reading a large file from the end backwards

Zackster · March 24, 2018, 9:53am

is there an efficient way to read a file line by line, but starting from the end?

my use case is grabbing the just most recent entries from a lucee log files to show
a current server status page in the Log analyzer plugin, with a refresh option which shows
what has happened since the status page was last loaded

is cflooping over the file is most memory efficient approach, or does log4j etc have
something to offer here?

Geo_Geo · March 24, 2018, 3:17pm

Doing this without getting your hands “dirty”, i don’t believe is possible.

Now, a way to approach the problem. Depending on the filesize, you may place the contents on ram:// and do your work there. This is going to be very fast. If however the file contents are a lot, your code may consume all your RAM and you don’t want to do this. In this case, you can dump the file content into a temp SQL table and just reverse the order with a simple query.

Hope this helps, or at list gives a first direction.
George

bdw429s · March 24, 2018, 11:20pm

Check out the tail command in CommandBox. I did this with a random access file object which is pretty easy to use. It doesn’t load the entire file into memory.

github.com

Ortus-Solutions/commandbox/blob/development/src/cfml/system/modules_app/system-commands/commands/tail.cfc

/**
 * Tail the end of a file from the filesystem. Path may be absolute or relative to the current working directory.
 * .
 * {code:bash}
 * tail file.txt
 * {code}
 * Displays the contents of a file to standard CommandBox output according to the number of lines argument.
 * .
 * Use the "lines" param to specify the number of lines to display, or it defaults to 15 lines.
 * .
 * {code:bash}
 * tail file.txt 100
 * {code}
 **/
 component {
	property name="printUtil"		inject="print";
	property name='ansiFormatter'	inject='AnsiFormatter';

	/**
	 * @path file or directory to tail or raw input to process

This file has been truncated. show original

isapir · March 29, 2018, 5:29pm

See

http://blog.rasia.io/blog/harnessing-the-power-of-java-in-cfml.html

Zackster · March 30, 2018, 4:52pm

this is great, i have some nice stuff happening with the log analyzer plugin

isapir · March 30, 2018, 4:53pm

I figured that you might want to update the plugin if you have time for that

Zackster · March 30, 2018, 5:12pm

so i’d like to avoid using listToArray what’s the best way to parse this

"INFO","XNIO-2 task-1","03/31/2018","01:03:13","Instrumentation","found [lucee.runtime.instrumentation.ExternalAgent] in ClassLoader [sun.misc.Launcher$AppClassLoader@5c647e05]" ?

isapir · March 30, 2018, 9:52pm

Well, you can’t really use list functions even if you want to, because the “message” might contain the delimiter "," so you have to parse it as CSV.

I actually wrote an implementation once but I’ll have to look it up.

Zackster · March 30, 2018, 10:09pm

basically for each row you can trust the first 5 to be well formed, then basically you’re looking for a single double quote at the start of a new line?

joe.gooch · March 30, 2018, 11:43pm

I’d just use OpenCSV
http://opencsv.sourceforge.net/

You can use the CSVReader interface if it’s coming from a file, or use CSVParser directly on strings.

Or you can write yet another parser

isapir · March 30, 2018, 11:44pm

That “trust” relies on the assertion that the user kept the default layout in the log settings, and did not use a custom pattern or layout class.

If that’s the case (which I would argue that it is not, based on the above), then you can use the built in functions listFirst(logline, ",", true, 5) and listRest(logline, ",", true, 6) to separate the “metadata” from the “message”.

Zackster · March 30, 2018, 11:54pm

I’m just going to grab the first chunk up to (i.e the start of the stack) and then do pretty much that

has anyone complained about this old code breaking?

github.com

zspitzer/lucee-logviewer/blob/master/source/cfml/plugins/Action.cfc#L238


      
          		<cfparam name="form.logfile" default="#url.logfile#" />
          		<cfset form.logfile = getLogPath(file=form.logfile) />
          
          
		<cfparam name="url.sort" default="date" />
          		<cfparam name="url.dir" default="desc" />
          
          
		<cfloop file="#form.logfile#" index="sLine">
          			<!--- If line starts with a quote, then it is either an error line, or the end of a dump--->
          			<cfif left(sLine, 1) eq '"'>
          				<!--- if not a new error --->
          				<cfif not refind('^"[A-Z-]+","', sLine)>
          					<cfif isDefined("aDump") and ArrayLen(aDump) gt 1>
          						<cfif isStruct(aDump[6])>
          				 			<cfset aDump[6].detail &= Chr(13) & Chr(10) & sLine>
          						<cfelse>
          							<cfset sTmp = aDump[6]>
          							<cfset aDump[6]          = structNew()>
          							<cfset aDump[6].error    = sTmp>
          				 			<cfset aDump[6].detail   = sLine>
          							<cfset aDump[6].fileName = "" />
          							<cfset aDump[6].lineNo   = "" />

I’m making pretty good progress, even extracting out the cfml stack trace

with click to expand

left on my todo list is some graphic design love and then using the tail function,
showing a summary page which tails all the current logs and shows what’s happened
recently, with filters by severity / log type, with a polling update which just grabs any
changes since the page was displayed / last poll

Zackster · March 31, 2018, 10:25am

Zackster · March 31, 2018, 6:26pm

first alpha of the log analyzer extension version 3.0.0 is up, if you want to try it out,
you can grab the .lex file from here, it’s using this new approach of reading files in reverse

when you click refresh, it just reads the log file in reverse returning any entries since you
last displayed the log file and dynamically inserts them into the page.

I’m keen to know if the log parsing falls over on different log files, please test!

Zackster · April 2, 2018, 1:38pm

latest improvements

the screenshot is the opening page, instant latest logs gratification
extracts cfc|cfc|lucee paths, click to search for related logs
simply select any text to auto search for other logs with that text
auto refresh options

Zackster · April 3, 2018, 12:42pm

@isapir is there a way to flush the logs if stream timeout isn’t 0?