Appending to file via file handle unreliable so need a way to force-flush

dev
1

I touched on this on Slack yesterday, but I’ve distilled a repro case that demonstrates a “shortfall” (cough) in Lucee’s file-appending handling.

Here’s the repro:

fh = fileOpen("/var/tmp/appendToMe.txt", "append")

testLine = repeatString("X", 1024) & chr(10)

for (i=1; i <= 20; i++) {
	fileWrite(fh, "#i#: #testLine#")
	sleep(2000)
}

A reasonable expectation here is that ever 2sec I see another kB of Xs appending to my file. The time delay is cosmetic… it’s just to make it easier to see what happens when.

On ColdFusion 2021, this is exactly what I see.

On Lucee 5.4.1.8 (in a Docker container built FROM lucee/lucee:5.4.1.8-nginx) what I see is that after 14sec I see the first 7 lines; after 28sec I see the next 7 lines, and I never see the last 6 lines.

Lucee seems to be be waiting until there’s ~7kB of data in some buffer before flushing it to the file. Although it also sometimes never flushes this buffer to the file (even when the file is closed @ the end of the request).

Zac initially said this to me on Slack:

that’s the operating system

I could see how there might be some buffering between Lucee/JVM/OS/filesystem, so this sounded reasonable until I tested the same code on CF. My CF2021 container is running on the same OS as the Lucee one (not my host OS, I mean coincidentally both CF2021 and Lucee use the same version of Ubuntu as their base OS image). Even if the OS or underlying file system has a hand in this, clearly CF manages to do what it’s actually been asked to do, and Lucee does not.

For the operation I am implementing, I need Lucee to do what it’s been told to do… append the data to the file… when I ask it to. Not “later, and actually maybe never”. This is no good.

I understand that sorting this out properly will take longer than the amount of time I have to wait, if it’s taken-up at all. Plus I will not be upgraded Lucee anyhow, so I’m not after a bug fix for this.

What I am after, hopefully, is a way to tell Lucee “flush yer buffer”. I don’t mind hooking into the file handle object and calling undocumented methods on the underlying Lucee implementation. This is less than ideal, but it is what it is.

If push comes to shove I can revert to using fileAppend on the file path, but I was trying to remove the overhead of the intrinsic file-open and file-close that that process entails, as this file will be getting appended to a lot: this whole undertaking is to work around instability in Lucee’s logging which is causing us grief.

Anyone know how to flush this buffer Lucee is holding on to?

Cheers.

2

as i suggested on slack, fh.getResource().getOutputStream().flush() might work?

thread is here Slack

3

OK, but we’ve already discussed that on the slack thread - before you also posted here, above - that that didn’t do anything. And you replied to that message so it’s not as if you didn’t see it.

Not sure the merits of repeating it here now?

4

The fileOpen has append mode, yet it is followed by fileWrite. Might it be that the issue arises because of the mixed message?

I expect either of the following 2 variations to work as expected. That is, instantly.

fh = fileOpen("/var/tmp/appendToMe.txt", "append");
fileAppend(fh.path & "/" & fh.name, "some appended text");

fh = fileOpen("/var/tmp/appendToMe.txt", "write");
fileWrite(fh.path & "/" & fh.name, "some written text");
1 Like