Installing specific versions of extensions in Lucee 5.2.x

hacking
1

We often use the Memcached extension in our Lucee applications, and since it’s not bundled with Lucee core then we need a simple way of installing it as part of a Docker build.

Lucee 5.x supports installing extensions via system properties (and by convention these also work via environment variables, by using an _ in the environment variable name in place of the . in the system variable name) using a comma separated list of extension IDs. This means that we can add a line like this to our Dockerfile to install the latest Memcached and H2 extensions, respectively;

ENV LUCEE_EXTENSIONS "16FF9B13-C595-4FA7-B87DED467B7E61A0,465E1E35-2425-4F4E-8B3FAB638BD7280A"

These extension IDs are found on the Lucee extensions download page:
http://stable.lucee.org/download/?type=extensions

image.png920×357 24.7 KB

However, let’s say that we need to install version 3.0.2.28 of the Memcached extension because we’ve found a bug in 3.0.2.29 that is causing a problem for us in production. There is another way to install Lucee extensions which is to drop them into the Lucee server deploy directory. In our Dockerfile, instead of using the line with the environment variable, we can use wget to download the specific version of the Memcached extension and place it in the appropriate directory;

RUN wget -nv http://extension.lucee.org/rest/extension/provider/full/16FF9B13-C595-4FA7-B87DED467B7E61A0?version=3.0.2.28 -O /opt/lucee/server/lucee-server/deploy/extension-memcached-3.0.2.28.lex

I figured this might be a good workaround, however during the next startup of Lucee after the build (the build starts and stops Tomcat to “prewarm” the installation and ensure that Lucee starts up quickly), I see the following output in the console which shows that Lucee is downloading and installing version 3.0.2.29 of the extension rather than the 3.0.2.28 that I had previously downloaded;

fandango_1   | 12-May-2017 15:35:13.538 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 2021 ms
fandango_1   | start set instrumentation
fandango_1   | sun.misc.Launcher$AppClassLoader
fandango_1   | sun.misc.Launcher$AppClassLoader
fandango_1   | sun.misc.Launcher$AppClassLoader
fandango_1   | found memcached.extension-3.0.2.29.jar:false
fandango_1   | download memcached.extension:3.0.2.29 from http://release.lucee.org/rest/update/provider/download/memcached.extension/3.0.2.29/?webId=965a7cd80d6c48e6c4d3bc4d4732ab20&webSecurityKey=043c7cc5-09ef-4718-ba07-281536ca0fc5&serverId=4bb321329fb282ecb82a59805be1d76d&serverSecurityKey=979b0968-fc36-42fa-847f-9529f2ff666d&allowRedirect=true and copy to /opt/lucee/server/lucee-server/bundles/memcached-extension-3-0-2-29.jar

This raises a couple of questions, and possible feature enhancements;

1) Can we add support for installing specific versions of Lucee extensions via the LUCEE_EXTENSIONS environment variable (and system property), so that we can more easily produce repeatable builds if we need to? for e.g.

ENV LUCEE_EXTENSIONS "16FF9B13-C595-4FA7-B87DED467B7E61A0?version=3.0.2.28"

2) Does the above “automatic upgrade” of the extension to a newer version after Lucee startup happen for a reason, for example is it because 3.0.2.29 is incompatible with Lucee 5.2.x. and so it needed to download 3.0.2.29 to function correctly? Is there any hidden dependency rules that we might need to know about? Or was Lucee just downloading a new version of the extension because it was found?

3) Is there some kind of flag that we can toggle so that Lucee never tries to automatically download newer versions of extensions? Automatic updates would be particularly problematic if you happened to restart a Lucee container in production and it downloaded a newer extension that broke your application. I would want any Lucee server to default to not automatically downloading newer versions of extensions after a restart – once the extension is installed (or a specific version is intended to be installed by placing it in the deploy directory) the version should stay the same until you choose to upgrade it.

For now we’re fine using the latest version of the Memcached plugin, but a way to manage these versions and produce fast, repeatable Docker builds, while avoiding unexpected extension updates, is very important to us :slight_smile: Perhaps we could target some of these things for Lucee 5.3.x?

1 Like
2

very simple :wink:

3

Are you saying that in answer to the first question above, we can have a comma separated list of extensions in that form? i.e.

ENV LUCEE_EXTENSIONS "7E673D15-D87C-41A6-8B5F1956528C605F;name=MySQL;label=MySQL;version=5.1.38"

How about the other two? :smiley:

4

you do (as in the example):
ENV LUCEE_EXTENSIONS "7E673D15-...;name=MySQL;label=MySQL;version=5.1.38,99A4EF8D-...;name=MSSQL;label=MS SQL Server;version=4.0.2206.100"

the syntax is exactly the same.
multiple entries are separated by “,”.

BTW the reason we use this syntax because this is the Syntax used by OSGi, and my example above is in the Manifest read by OSGi. you will see in the example above, this is the syntax used in all over the file.

1 Like
5

Awesome :slight_smile:

What about extension updates getting automatically installed, is there a way to turn that off without specifying every extension?

6

After doing some testing with older Lucee builds, it looks like Core bundles/extensions don’t get automatically updated on startup if a newer version exists (which is good), and if you specify a manifest style value in LUCEE_EXTENSIONS that includes the version number then there is also no risk of auto-updates happening to the extension.

It also seems like label and name aren’t both required, however including one of them does help identify which extension the configuration is for, e.g.;

16FF9B13-C595-4FA7-B87DED467B7E61A0;name=Memcached;version=3.0.2.28

On a related note, is there any configuration available for disabling specific extensions? For example there might be a number of extensions that ship in Lucee core that a particular application doesn’t need (Hibernate, certain database drviers, the Lucee Administrator, etc). Can we disable specific extensions, or do we need to remove the files from disk?

1 Like
7

@justincarter recently posted a nice little tutorial on spinning up memcached extensions in Docker :whale:

8

Is there a way to uninstall specific extensions using the same mechanism?

9

Are there any docs about what each of the environment variables do? I am still trying to install the extension as defined by the the LUCEE_EXTENSIONS variables but it doesn’t seem to be working on Lucee 5.2.5.20

Some docs and clarity outside the LAS team members would be nice.

docker compose example:

version: "3"
services:
  web:
    image: ortussolutions/commandbox
    volumes:
      - ".:/app"
    ports:
      - 8080
    environment:
      - LUCEE_EXTENSIONS="16FF9B13-C595-4FA7-B87DED467B7E61A0;name=Memcached;version=3.0.2.28"
1 Like
10

With Docker it’s important that extensions are installed during the image build for a number of reasons (container startup time, removes the dependency on the extension server being contactable, consistent/repeatable builds, etc), so I’d avoid adding the LUCEE_EXTENSIONS environment variable to the Docker Compose file if possible since they are kind of run-time settings.

I think this is the most basic, working example that I can give as a starting point for installing Memcached, and then configuring it as a session store;

Dockerfile

FROM lucee/lucee52:latest

# install extensions by prewarming lucee
ENV LUCEE_EXTENSIONS "16FF9B13-C595-4FA7-B87DED467B7E61A0;name=Memcached;version=3.0.2.29"
RUN /usr/local/tomcat/bin/prewarm.sh

COPY www /var/www

docker-compose.yml

lucee:
  build: .
  volumes:
    - "/workbench/memcached/www:/var/www"
  ports:
    - "8888:8888"
  links:
    - "memcached:memcached"
memcached:
  image: memcached:alpine
  expose:
    - "11211"

(Note: Your volume paths may vary, the above is just what works for me in my environment).

www/Application.cfc

component {
	this.name = "memcached";

	this.cache.connections["sessions"] = {
		class: "org.lucee.extension.io.cache.memcache.MemCacheRaw",
		bundleName: "memcached.extension",
		bundleVersion: "3.0.2.29",
		storage: true,
		custom: {
			"socket_timeout":"30",
			"initial_connections":"1",
			"alive_check":"true",
			"buffer_size":"1",
			"max_spare_connections":"32",
			"storage_format":"Binary",
			"socket_connect_to":"3",
			"min_spare_connections":"1",
			"maint_thread_sleep":"5",
			"failback":"true",
			"max_idle_time":"600",
			"max_busy_time":"30",
			"nagle_alg":"true",
			"failover":"true",
			"servers": "memcached:11211"
		},
		default: ""
	};
	this.sessioncluster = true;
	this.sessionstorage  = "sessions";

}

www/index.cfm

<cfparam name="session.timestamp" default="#now()#">
<cfdump var="#session.timestamp#">

To test;

  1. Start the containers in the background with docker-compose up -d.
  2. Browse to the index of the app and a timestamp will be written to the session which will be stored in memcached.
  3. Restart just the Lucee container with docker-compose restart lucee
  4. Browse to the index of the app again and you should see that the timestamp remained the same, which means that the session was successfully persisted in memcached across Lucee restarts.

If you’re using CommandBox Docker images I don’t know the most efficient way of installing/prewarming Lucee there, there’s perhaps a command that will start and stop the server for you. I would start with barebones Lucee Docker images to prove it’s working, and then move back up the chain up from there.

Hope that helps :smiley:

2 Likes
11

This post makes reference to a spreadsheet tracking the state of all config options:
https://lucee.daemonite.io/t/lucee-configuration-options/321

12

What do you have in your pre-warm.sh script? is it

 RUN rm -rf /opt/lucee/web/logs/ && \
   /usr/local/tomcat/bin/catalina.sh start && \
   while [ ! -f "/opt/lucee/web/logs/application.log" ] ; do sleep 2; done && \
   /usr/local/tomcat/bin/catalina.sh stop
13

Yep, you can see the source here:
https://github.com/lucee/lucee-dockerfiles/blob/master/5.2/prewarm.sh

1 Like
14

Hi @markdrew I just noticed your comment. I don’t know if you figured it out, but the env var LUCEE_EXTENSIONS is not what Lucee proper looks for. I believe that name is specifically something looked for in the official lucee image that gets passed through to an env var or JVM prop of the name Lucee is actually looking for which is either lucee-extensions (deprecated) or lucee.extensions

The CommandBox image doesn’t provide a passthrough env var called LUCEE_EXTENSIONS. We could, but I’m not sure what the point would be since it’s almost exactly named as the built in env var, just non-standard. With the CommandBox image, just provide the environment variable in the format Lucee itself is looking for and everything should work.

Also, FWIW, since these links are so hard to find, I’ll leave them here as well. Here’s the docs for that env var (which seriously needs some love since it doesn’t even cover specific versions!):
http://docs.lucee.org/guides/running-lucee/configuring-lucee/system-properties.html

And here is the “living” Google Doc that lists all possible env vars and system props. The one in question is on line 5.

15

The Lucee settings that come from Java system properties are also supported as environment variables. From my original post above;

This means that the Java system property lucee.extensions can instead be specified as the environment variable LUCEE_EXTENSIONS, and the same should apply to all the other properties that are listed in the above spreadsheet :slight_smile:

Edit: just to clarify, the Lucee docker images don’t do anything special to find and pass through these environment variables as system properties, they are supported directly by Lucee and should work anywhere AFAIK.

2 Likes
16

We use

ENV LUCEE_EXTENSIONS "66e312dd-d083-27c0-64189d16753fd6f0;name=PDF;version=1.0.0.68,99a4ef8d-f2fd-40c8-8fb8c2e67a4eeeb6;name=MSSQL;version=6.2.2.jre8"

In our Dockerfile and that seems to work, as once we’ve prewarmed saved the image then spun the image up, going to the extensions area shows these as up-to-date.

17

It would be really good if Lucee would either use ExtensionStore ID’s as you are doing or forgebox stubs, I think this would look better no?

ENV LUCEE_EXTENSIONS pdf@1.0.0.68, memcached@3.0.2.25

Since using the UUID is such a pain for admins etc that would look at a random docker file (like I have) :

FROM ortussolutions/commandbox
COPY ./ $APP_DIR/
ENV HEADLESS true
ENV BOX_INSTALL true
ENV LUCEE_EXTENSIONS 16FF9B13-C595-4FA7-B87DED467B7E61A0

Prizes for anyone to know what extension that is :slight_smile:

At the end of the day, the point of this is to make configurations easily readable by humans.

In summary, add stub names for the existing extensions (store, since that is where they come from), if not found, look for them in forgebox.io

steps off soapbox

1 Like
18

Mine looks like

ENV LUCEE_EXTENSIONS "66e312dd-d083-27c0-64189d16753fd6f0;name=PDF;version=1.0.0.68,99a4ef8d-f2fd-40c8-8fb8c2e67a4eeeb6;name=MSSQL;version=6.2.2.jre8"

And that appears to install, whilst I can at least see what it’s going to do…

19

I get you, I see what it’s going to do, but it’s an abusive way to do it for the poor sys admin that I am going to hand off the docker file to.

And then I tell him about the box.json file that also downloads dependencies etc. I know you can read it but there is usually a chain of people that need to know this.

(I wasn’t having a go at you sir, just the implementation)

20

Micha explained above the reason for the current implementation is that it matches the OSGi syntax that is used in the manifest files for installing extensions;

e.g.

It’s not the nicest thing to read but the name and/or label does make it human readable, and I’m ok with it at least being consistent with what is already used in the manifests.

That said, I’d be happy to see a new identifier that wasn’t a UUID if Micha thinks that’s feasible to support :slight_smile:

2 Likes