Setup server-wide datasource from command line

[thefalken]

Is there an easy way to insert a datasource to lucee-server.xml from the command line (say, when Docker is building a docker image ) ?

I want to insert a global datasource shared by all the contexts on the server.

I’ve tried using sed to replace in the XML blob I see being added when I use the server web interface to do this, but Lucee always seems to say

2017-04-28 15:44:37.254 config file /opt/lucee/server/lucee-server/context/lucee-server.xml was not valid and has been replaced
2017-04-28 15:44:37.256 write file:/opt/lucee/server/lucee-server/context/lucee-server.xml

at start up

[bdw429s]

Use CFConfig. This is actually already baked into the Ortus CommandBox Docker images.

Introducing CFConfig : A new way to manage your CF server's configuration from...

CommandBox Docker Image 3.6.0 Released

[justincarter]

We usually set datasources in the application.cfc via the this.datasources[] array, and populate the settings with environment variables. This way your build isn’t tied to a particular environment, you can easily pass in different settings no matter where you want to run your container.

[modius]

We usually set datasources in the application.cfc via the this.datasources array, and populate the settings with environment variables. This way your build isn’t tied to a particular environment, you can easily pass in different settings no matter where you want to run your container.

An example:

system = createObject("java", "java.lang.System");
MY_DSN = "" & system.getEnv("MY_DSN");
if ( len(MY_DSN) ) {
	THIS.datasources[MY_DSN] = {
        "class" = system.getEnv("MY_DSN_CLASS"),
        "connectionString" = system.getEnv("MY_DSN_CONNECTIONSTRING"),
        "database" = system.getEnv("MY_DSN_DATABASE"),
        "driver" = system.getEnv("MY_DSN_DRIVER"),
        "host" = system.getEnv("MY_DSN_HOST"),
        "port" = system.getEnv("MY_DSN_PORT"),
        "type" = system.getEnv("MY_DSN_TYPE"),
        "url" = system.getEnv("MY_DSN_URL"),
        "username" = system.getEnv("MY_DSN_USERNAME"),
        "password" = system.getEnv("MY_DSN_PASSWORD")
    };
}

You could probably get away with hard coding a lot of these values if you are working within a single database type and you’re just changing the location and/or credentials for the database.

[modius]

I’ve tried using sed to replace in the XML blob I see being added when I use the server web interface to do this, but Lucee always seems to say

Sometimes it can be very useful to use the web GUI to update all the permanent settings you want for your Lucee installation. This automatically updates the underlying XML config files and all you need is to capture those changes and push them in at build time.

We store our project configuration in a subfolder structure beneath the location of the Dockerfile:

./config
├── lucee
    ├── lucee-server.xml
    └── lucee-web.xml.cfm

These can be committed to version control and copied via the Dockerfile at build time:

./Dockerfile

# Lucee server configs
COPY config/lucee/lucee-web.xml.cfm /opt/lucee/web/lucee-web.xml.cfm
COPY config/lucee/lucee-server.xml /opt/lucee/server/lucee-server/context/lucee-server.xml

You can map these configuration files into your container using VOLUMES during development so you can easily capture any changes you might make to the admin; for example, a docker-compose.yml file might include:

./docker-compose.yml

  volumes:
    - /workbench/my-app/config/lucee/lucee-server.xml:/opt/lucee/server/lucee-server/context/lucee-server.xml
    - /workbench/my-app/config/lucee/lucee-web.xml.cfm:/opt/lucee/web/lucee-web.xml.cfm

Note, it’s not ideal if you need these values to change at container runtime You need to use ENV variables for this. Fortunately, Lucee allows you to set many configuration flags directly or programmatically via the Application.cfc.

Lucee Configuration Options

There are various points where one can influence the configuration of the underlying Lucee engine: Lucee Admin GUI (changes saved in the lucee-server.xml and lucee-web.xml.cfm config files) Environment variables and System properties Application.cfc (Runtime only) Tag/Function/Other This Google Sheet is a fairly comprehensive overview of current configuration options and priorities (open to view/comment): https://docs.google.com/a/lucee.org/spreadsheets/d/10s-nn_FsoSD_RiLwjYZICacCoC386SjkEG…

[bdw429s]

Sorry, I was on my phone the other day so I couldn’t get an example of the CFConfig call. This is all it takes.

box cfconfig datasource save \
  name=myDSN \
  dbdriver=mysql \
  host=localhost \
  port=3306 \
  database=myDB \
  username=brad \
  password=foobar \
  to=/path/to/server/home

1. It won’t overwrite other config already in the XML file
2. It works on any CF engine/version
3. Can be made dynamic with env vars

Or you could bundle it up with any other configuration you like (CF mappings, timeouts, scope settings, etc) in a JSON file …

.cfconfig.json

{
    "datasources":{
        "myDSN":{
            "host":"localhost",
            "dbdriver":"mysql",
            "password":"foobar",
            "database":"myDB",
            "port":"3306",
            "username":"brad"
        }
    }
}

… and import them all at once like so:

box cfconfig import from=.cfconfig.json to=/path/to/server/home

[Julian_Halliwell]

[Cough!]… http://cfscript.me/

<cfset system = createObject(“java”, “java.lang.System”)>
<cfset MY_DSN = “” & system.getEnv(“MY_DSN”)>
<cfif len(MY_DSN)>
<cfset THIS.datasources[MY_DSN] = {
“class” = system.getEnv(“MY_DSN_CLASS”),
“connectionString” = system.getEnv(“MY_DSN_CONNECTIONSTRING”),
“database” = system.getEnv(“MY_DSN_DATABASE”),
“driver” = system.getEnv(“MY_DSN_DRIVER”),
“host” = system.getEnv(“MY_DSN_HOST”),
“port” = system.getEnv(“MY_DSN_PORT”),
“type” = system.getEnv(“MY_DSN_TYPE”),
“url” = system.getEnv(“MY_DSN_URL”),
“username” = system.getEnv(“MY_DSN_USERNAME”),
“password” = system.getEnv(“MY_DSN_PASSWORD”)
}>
</cfif>

[modius]

[Cough!]… http://cfscript.me/

Niice. Did not know about that little utility Code updated!

[thefalken]

We usually set datasources in the application.cfc via the this.datasources[] array, and populate the settings with environment variables.

But, con-wise, if that shared data source ever changes, you need up update all the projects. We also have per-project datasources, and setting them up via Application.cfc is definitely where I want to get to.

[Julian_Halliwell]

But, con-wise, if that shared data source ever changes, you need up update all the projects. We also have per-project datasources, and setting them up via Application.cfc is definitely where I want to get to.

If the data source parameters are all dynamically populated from EVs as in Geoff’s example, (or even just those that are liable to change) then surely you only have to change the environment variables, not the project code.

[justincarter]

If the data source parameters are all dynamically populated from EVs as in Geoff’s example, (or even just those that are liable to change) then surely you only have to change the environment variables, not the project code.

That’s right, you just update the environment variables in your orchestration tool/platform of choice, no need to change any code/files and no need to do a new Docker image build of your app

To take it a step further, sensitive data is better managed using a secrets store of some kind (Swarm has built-in support for Secrets, or you can use other solutions like Vault, etc). There’s a little bit of extra complexity to deal with here, and the landscape is evolving pretty quickly, but that’s the direction things are heading.

[thefalken]

But, con-wise, if that shared data source ever changes, you need up update all the projects. We also have per-project datasources, and setting them up via Application.cfc is definitely where I want to get to.

If the data source parameters are all dynamically populated from EVs as in Geoff’s example, (or even just those that are liable to change) then surely you only have to change the environment variables, not the project code.

And tell all the team the new environment variables etc. And if we added a new param that had to passed to the (shared) DSN, update all the projects (again).

For now it’s easiest to bake the DSN (and the secrets) into the image I think. Trying to be as simple as possible to start with and not change the existing project code to match our current deployment tool!.

[Julian_Halliwell]

And tell all the team the new environment variables etc.

If team members are each using a locally installed dev server then you’d have to tell them about changes anyway so they can update their local admin settings. The only difference is where those details are stored: in the Lucee server settings versus in the host server’s EVs.

And if we added a new param that had to passed to the (shared) DSN, update all the projects (again).

In theory there’s a custom attribute which allows you to put additional the params together as a struct. You could store that value as an EV so it could be changed without altering app code. In practice it seems to be a bit buggy.

One potential “con” though is that EV changes do require a Lucee/JVM restart to be picked up.

However I’m definitely leaning towards EVs as the best and simplest place for these types of variables.

[justincarter]

For now it’s easiest to bake the DSN (and the secrets) into the image I think. Trying to be as simple as possible to start with and not change the existing project code to match our current deployment tool!.

If you’re only approaching it from a local dev perspective then you could bake them in, but once you consider deploying to other environments like stage and production you’ll quickly be wanting to use settings that come from the environment / orchestration tool.

A single build of a Docker image can be deployed to stage and then to production without any changes, because all your configuration comes from the environment rather than code/files inside your image

For example, for local dev you can store your particular environment variables in a docker-compose.yml file, or in env files that are picked up by your compose file. For stage and production, you might be using Mesos/Marathon in which case the environment variables come from the application definition (think of it as a .json file), or if you were using Docker Cloud it might come from a stack file (basically a compose file with a few extra options). The point is that your code / build / Docker image doesn’t need to change, you can deploy it anywhere and let the settings come from the environment.

[thefalken]

For example, for local dev you can store your particular environment variables in a docker-compose.yml file,

My current (getting started) plan is just to move to Docker for day-to-day development. So I’m building a simple reusable Docker image that non-Docker-knowledgeable people can pull, run and be up and running as easily as with a remote Lucee and a network share (+ a one time change to their hosts file per-project).

Our (eventual) production (and dev, stage etc) containers won’t be based on this image, so it makes more sense to use the environment variables there I think.

[modius]

My current (getting started) plan is just to move to Docker for day-to-day development. So I’m building a simple reusable Docker image that non-Docker-knowledgeable people can pull, run and be up and running as easily as with a remote Lucee and a network share (+ a one time change to their hosts file per-project).

It’s certainly worthwhile using docker-compose up instead of a docker run incantation.

If your project Dockerfile and docker-compose.yml are all part of your git repo then having the relevant ENV variables hardcoded into the docker-compose.yml solves all your problems.

An example, day to day docker-compose.yml i work on:

daemon:
  build: .
  environment:
    - "VIRTUAL_HOST=dae-env-daemon.*"
    - "FARCRY_DSN=daemon"
    - "FARCRY_DSN_CLASS=org.gjt.mm.mysql.Driver"
    - "FARCRY_DSN_CONNECTIONSTRING=jdbc:mysql://0.0.0.0:3306/daemon_daemon?useUnicode=true&characterEncoding=UTF-8&allowMultiQueries=true&useLegacyDatetimeCode=true"
    - "FARCRY_DSN_USERNAME=fubar"
    - "FARCRY_DSN_PASSWORD=encrypted:gibberish"
    - "FARCRY_CONFIG_MEMCACHED_SERVERS=memcached:11211"
    - "SMTP_SERVER=mailtrap.io"
    - "SMTP_PORT=2525"
    - "SMTP_USERNAME=gobbledigook"
    - "SMTP_PASSWORD=gobbledigook"
  volumes:
    - /workbench/dae-env-daemon/project:/var/www/farcry/projects/daemon
  links:
    - memcached:memcached
  ports:
    - "8088"
memcached:
  image: memcached:alpine
  expose:
    - "11211"

[thefalken]

I see what you are saying. But we have a mix of O/S the developers use, so can’t put the volumes in a compose file because the local paths will differ ?

[modius]

Indeed! See…

Daemon Docker Workbench

Short story…

• project specific docker-compose configs that can be versioned and used by all developers on the team, regardless of OS or directory set up
• elegant management of hosts/DNS and ports for applications
• allow mobile, tablet and other network devices to easily access the containerised applications

[ShrinivasKR]

Do the passwords have to be stored in environment variables in the encrypted format they show up in the lucee-server? If so, how would we generate that encrypted password from a known password?

[bdw429s]

The password can be encrypted or not, it’s up to you. if it’s encrypted, it always starts with the text “encrypted:” I have a utility on ForgeBox called the Lucee password util that will do this, or just create the datasource in your admin, go back and edit, and scroll to the bottom of the page where the encrypted version will be output on the page for you to copy.