Cross container session sharing

dev support
1

OS: Ubuntu 20.04
Java Version: jdk11 (corretto, openjdk)
Tomcat Version: 9
Lucee Version: 5.3.9

I’m aiming at a single Lucee application load balanced across two nodes running, say, 3 containers each. The authenticated user info will be stored in session scope. So, I’d like to share sessions across all 6 instances across both nodes. After playing around with it a bit, I seem to have found the following knowledge:

1 - In Docker

Use sessionType="jee".
Docker says:

Session management

The default session type in Lucee is “cfml”. This often causes issues for Lucee servers running in Docker containers. If you use CFML sessions you should set the session type to “j2ee” in a Lucee configuration file or Application.cfc.

2 - Clustering (sharing sessions)

must be in sessionType="cfml"

3 - Ergo:

Clustering dockered lucee instances means the clustering must be done in tomcat. Then you deploy Lucee in those clustered tomcat instances.

Can anyone verify my findings? Or is it possible that the Docker hub recommendation to use sessionType="jee" a hang-over from the Lucee 4.5 Docker hub readme and someone now knows that sessionType="cfml" in dockered Lucee 5 is stable?

B.

2

We use Lucee in Docker pods and have them set to cfml sessions; one pointing to a DB to hold the sessions and one pointing to a redis container for the sessions

	// Session Information
	this.sessionType = "cfml";
	this.sessionManagement = true;
	this.sessionStorage = "SessionManager";
	this.sessionCluster = true;
	this.sessionTimeout = Request.Cache.Thirty;
	this.setClientCookies = false;
	this.setDomainCookies = false;

And that works fine for us.

3

just to be clear, that’s referring to

  • “datasource-name”|“cache-name”: when you select a name of an available datasource or cache, the session scope will be stored in there
4

And also (just for clarification)

	this.setClientCookies = false;
	this.setDomainCookies = false;

means we’re setting the cookies onSessionStart manually, as copy/pasting this code won’t have sessions working correctly

5

Thank you @jedihomer && @Zackster. Yup, @jedhomer’s SessionManager is the name of his datasource-name or cache-name.

@jedihomer, Are you using sticky sessions? I’d like to avoid them but, while I see a lot of people reporting round-robin, it seems like most docs on session stores indicate sticky should be used.

Having said that, I have some Lucee instances that are not in docker containers in round-robin load balancing with ms sqlserver as the session store. But what’s interesting is that round-robin unreliable when I move these instances to PostgreSQL.

6

Nope no sticky sessions, we’re using the GKE ingress to a service which is basically round-robin the calls.

7

You manually set CFID and CFTOKEN in onSessionStart? Interesting. Are you interested in explaining how you found this to work? Or - can you point to resources that would explain why this works?

8

The apps we use this are old… My hazy recollection was to set the cookies as secure, which I don’t think was initially possible when we were writing it.

	// Sort out secure cookies
	header name="Set-Cookie" value="cftoken=0; Path=/; HTTPOnly; Secure;";
	header name="Set-Cookie" value="cfid=" & Session.cfId & "; Path=/; HTTPOnly; Secure;";

Is what we have in onSessionStart to set the cookies.

9

Wow - great info. Thank you!

10

Another long-standing reason to set the cfml session id cookies manually is to ensure they are “session cookies” in the sense of expiring on browser close. If you leave it to setClientCookies=true then they will be stored for 20 days. Good privacy practice is to avoid unnecessary persistent cookies.

1 Like
11

I have spent most of today trying to get this to work in vm installs and k8s with and without a load balancer, using a mysql db (for this POC) as the clustered session store.

Component {
  this.name='impossible-session';
  this.sessionManagement = true;
  this.sessionCluster = true;
  this.setClientCookies = false;
  this.setDomainCookies = false;
  this.sessionType = "cfml";
    var json = deserializeJSON(fileRead('./conf.json')); // define "sessions" datasource
    this.datasources = json.datasources;
    this.sessionStorage = "sessions";

  public void function onApplicationStart(){
    // tried the above in here to, no joy
  }

  public void function OnSessionStart(){
    header name="Set-Cookie" value="cftoken=0; Path=/; HTTPOnly;";
    header name="Set-Cookie" value="cfid=" & Session.cfId & "; Path=/; HTTPOnly;";
    this.logit('OnSessionStart', session.cfid);
  }

  // ...
}
  • the sessions appear in the DB :+1:
  • the data persists accross requests to the same pod :+1:
  • when I am switched to a new pod, the pod does not record a new session in the DB :+1:
  • however the onSessionStart is triggered though (logged to a file)
  • and the session is empty :frowning:
  • when I am switched back to the old pod the data is back

In all the environments it is the same. As I say something is written to cf_session_data but none of the nodes seem to be reading it. I am clearly missing something. Can you please explain

one pointing to a DB to hold the sessions and one pointing to a redis container for the sessions

Session clustering in Lucee has been want-to-have for us for years but nothing seems to work and the docs are very thin.

Many thanks in advance for any info.

12

Don’t you need a load balancer? If you just point your browser to a new IP address wont this be considered a new session? The remote_ip address change (or something) would clue Lucee into the idea that it’s a new session. Then, I think, because you are passing it a cfid, Lucee would start a new session with that cfid. So you get a new empty session.

If you point your browser back to the old pod, that session is still alive, so your session data is back.

Just spit-ballin’

EDIT: Actually I guess your remote_ip wont change. So not sure what it would be but I’m still thinking this theory may hold.

13

@jvc, it’s certainly interesting that the request to the new pod “does not record a new session in the DB”, while “the onSessionStart is triggered” but “the session is empty”.

I have some different thoughts than have been shared so far here by the few others involved, and different diagnostics which may help anyone facing such session challenges. Perhaps any of you already knew some or all of these things, but I don’t see them discussed here.

(Brian, I’d not think it had to do with the ip address to which a request is made, but it can DEFINITELY have to do with the domain to which a request is made. And same if somehow different ips are being used for the pods/instances being requested.)

Consider that setdomain cookies

Indeed, first, as for jvc’s last code which shows setdomaincookies=false (and others concurred they use that), I’ll just note for the sake of completeness that for setdomaincookies the lucee docs say “Applications that are running on clusters must set this value to Yes.”. I’m not saying it will fix things, but have you at least tried that, jvc?

The browser will send cookies to a server only if the cookies it has have a matching domain value to the request being made. (A subdomain would not be the same. I’ll leave it at this for now, though there’s more to be said.)

Consider this.sessioncookie vs manual cookie tweaking

Second, while you show and others here have discussed setting the cookies manually (via those header statements, in our case), do beware that whether it sets a domain on that cookie will depend on other factors…and that may play into what the next request (perhaps to the new pod) may or may not get. And I’ll be focusing on that in a moment. The same goes for other of the cookie attributes you are setting in that code.

I understand why some have historically tweaked those cookies and rewritten them that way (like to change the cookie timeout to have it “expiring on browser close”), but do beware that the NEED for manual cookie tweaking has been replaced with the addition of the this.sessioncookie struct in application.cfc (or the corresponding attribute in cfapplication). This was added in CF10 (2012) and later in Lucee. It’s mentioned briefly in the Lucee docs here.

Again, I understand folks may not want to touch existing code to implement that change, but again I offer it especially for those here who are having session trouble, at least as something to try.

Diagnosing session problems

Finally, I’ve found in diagnosing any session oddities (whether stored in a db or not, whether clustered or not, whether using containers or not, and whether using Lucee or not) that the first key was to see what cookies the CFML request was receiving. Often, you will find that what it’s receiving is NOT what it SHOULD be, for the behavior you want.

So first, lets have you check/log that incoming cookie info on each request (whether you do that on some cfm/cfc page you call or in the application.cfc/cfm that controls it). If you’re using jee sessions, it would be the jsessionid that matters most. As **you show using what lucee calls “cfml” sessions, then I am pretty sure that’s like CF’s default, in which case it’s cfid and cftoken cookies that matter most.

(BTW, have you tried both ways, jvc?)

Be careful how you look for the cookies

When it comes to checking on the cookies coming in, do NOT look at/dump/log the CFML “cookie” scope. That could hold cookie values SET in the request itself before that point (including due to things like such onsessionstart code as you both show, or the engine itself).

Instead, look at the cgi.http_cookie variable, which holds any and all cookies received by the CFML engine from the request. (Granted, that might be manipulated by a web server or other appliance between the user and the CFML engine, but let’s presume for now that is NOT the issue.)

Sepending on that this.sessiontype that Lucee offers, look for the jsessionid value, or the cfid/cftoken pair–within that semi-colon separated list of cookies in that var. You can process/search/extract from it with CFML list functions, of course.

And do look for whether there may be more than one session cookie. I’ve seen that happen with cookies, usually unexpectedly, whether more than one jsessionid or more than one cfid/cftoken pair.)

Does it differ on the instances/pods?

So the key question is: when the request goes to the instance/pod where things “work” (and you say a session is “logged”), what is the session cookie value going into the request there, as compared to going into the pod where you say the 3 things I quoted above?

And if the session cookie value DOES differ, that’s its own question, with things to consider. (You’d want to then look at the cookies as RECEIVED/stored on the browser, to see things like their domain and secure values, as well as the path and httponly values you guys show manipulating. That’s another whole kettle of fish. But let me leave it at this for now.)

Hope that’s of some help, so someone seeing this thread.

14

A load balancer with sticky sessions is essential for clustered sessions to work with Lucee

Sessions are not constantly saved and loaded to the database per every request (that would suck for performance), they are kept on the current node and then written out after a period of time (something like 2 minutes) but only after the session has been used for multiple requests (from memory) this is also important as you don’t want sessions created for every single bot hit hitting your website

from @micstriit [LDEV-2422] - Lucee

as a sidenote avoid empty “onSessionStart“ whenever you can. Lucee only creates a session for the client in 2 cases:

a) when data are stored to a session
b) when a onSessionStart exists

if a and b are not true no session is created

Like most things in Lucee, you can bump up the log level in the server admin for the Scope logging to DEBUG and then you can see logs about all the activity going on with session scopes

15

Lots of prompt feedback, everyone! Awesome :smiley: I will go through in detail to make sure I have ticked all of your suggestions off but for now I can summarise:

I have tried lots of combinations of the above settings, setDomainCookies on|off, manual headers, no manual headers etc, etc…and etc. The code I shared was just where I got to at the end of the day.

To debug, I am outputting the contents of the sessions in index.cfm (not shown). The session ids etc are all correct as per in the input cookie but. The separate centralised logging I mentioned is just to record when onSessionStart is fired and on which node.

When I say the sessions are “empty” I mean the data I have added to the session on host 1 is not there when the session is subsequently dumped on host 2 and I suspect it is because I am missing an extra step, not detailed so far in this thread, hence:

Keeping the sessions centrally off the nodes to obviate persistence in the load balancer and provide session resilience when a node fails is the goal here. I have tried redis intergration before this but I wanted to try the cfml sessions to see how viable Lucee’s built-in clustering was. If the sessions are only updated to the shared store after a delay this feature will not be suitable on it own. I am hoping the missing piece is what @jedihomer said, which I am not clear on:

I can only think that this means session Ids are stored in the “DB” (mysql in the POC I am doing here) but the dynamic session content is being synced to redis either manually (perhaps writing out onRequestEnd and reading-in onRequest) or using a third party session manager. Is that (roughly) the model?

Again, thanks for all the info.

16

Wow, Zack, those are indeed amazing additional insights/aspects of how Lucee clustered and persisted sessions work…which (as far as I know) is QUITE different from both CF and Tomcat session mgt with clusters. Thanks.

That said, are those points documented anywhere? I had searched the docs for more on a couple of things discussed in this thread before I wrote my last long note. I don’t recall ever seeing those points before. They’re quite valuable.

And do they apply whether using jee sessions or not? And if with jee, is it tied to Tomcat or the same on any servlet engine?

(By comparison, ACF Enterprise’s cluster feature offers sticky sessions and/or (if using j2ee sessions) replication–which is technically Tomcat’s session replication. But the CF clustering feature offers NO option for persistence to a db–though Tomcat offers that. That’s why I wonder if the behavior Zack discusses could be Tomcat behavior, which CFers just wouldn’t experience out of the box. FWIW, cf2016 then added the option of storing sessions in Redis–in cf std or ent, though that works only if NOT using J2ee sessions.)

And yep, jvc, it sure does seem that your observed behavior could be influenced by one or more of the things Zack offers. Let’s see what you may find.

17

Jee sessions are controlled/come from tomcat/undertow/etc, Lucee on the other hand has full control of the behaviour of cfml sessions

@micstriit knows more of the intricacies, maybe @Brad_Wood ?

18

Sessions are not constantly saved and loaded to the database per every request (that would suck for performance), they are kept on the current node and then written out after a period of time (something like 2 minutes)

Hold up, we’ve gotten off track. This is only half true and makes a huge difference. In Lucee, there are two modes for session storage.

this.sessionCluster=false;
and
this.sessionCluster=true;

if set to true, lucee uses the storage backend for the session scope as master and Lucee checks for changes in the storage backend with every request.

If set to false (default), the storage is only used as slave, lucee only initially gets the data from the storage. Ignored for storage type “memory”.

What Zac described above where the in-memory cache of the sessions is given priority and the external storage is async is when cluster is turned off. This could be for a single server or with sticky sessions and you simply want to retain session contents across a restart.

When session cluster is enabled, which in my opinion is really the only reasonable way to use session storage, then session data is in fact read and written to the external storage every request. The overhead actually isn’t that bad and Micha attempts to optimize it by only writing keys that have been modified, even though this is the source of some buggy behavior :confused:

1 Like
19

I don’t know much about redis but isn’t it designed/optimized for this kind of situation (being an in-memory data store)?

20

No, you’re confusing a couple things. Redis may store things in its own memory, but it’s still an external process and there is an over-the-wire TCP based communication that happens to get things in and out of Redis. Same with Couchbase, or Memcached, or heck – even SQL Server. Common data is going to be cached in memory on that node. But it’s not “in process”, meaning it’s not in the same memory space as the JVM is using like you would get with RAM cache or the built in EHCache (which are not distributed)