How do we make configuring Lucee rock?

hacking
1

This is an open discussion about the ways in which a Lucee server, or Lucee application should be configured. We’re introducing some cool new things like system property config (http://lang.lucee.org/t/configure-lucee-via-system-properties/91) and I’d like to be able visualize all the ways config can be stored and used at run time to make sure our approach is holistic and well thought out.

Currently, I can think of three “logical” places that configuration exists

  • Server level
  • Web level
  • Application level

Here are the possible ways I can think of that a developer can “interface” with configuration

  • Browser-based server admin (persisted in lucee-server.xml)
  • Browser-based web admin (persisted in lucee-web.xml)
  • Application.cfc settings
  • <cfadmin> tag (undocumented, must be run in context of web/server context you want to edit)
  • (Java) System Properties - Right now appears to just be a bootstrap to point to the XML config files.
  • OS Environment variables - Not actually implemented, but suggested in the system property post.
  • Edit XML files directly and restart service
  • CLI tooling (doesn’t exist, but I’ll discuss in another thread)

How do we feel about these options? The first line of config levels I think is pretty straight forward. The’s the second list that has some newcomers that could use some fleshing out. Like I mentioned, system props appear to only be useful currently as a shortcut for the settings found in web.xml that point to the luce-server.xml and lucee-web.xml files.

@micstriit hinted that it could be expanded to include most settings. What would that look like for complex, enumerated settings like cf mappings or data sources? Also, how would you specify properties that targeted settings for a specific web context, or even an application name within a given web context? Should all that be something that should be able to be specified in system properties or environment variables?

My big thing is I want consistency between all the ways that settings can be interacted with so everything that can be configured is equally available via each method.

Also, there’s been discussion on the format of Lucee’s persisted settings: XML. Should it stay XML or transition to something different? Should editing of these files be encouraged, or should they be treated as an undocumented internal concern that can change at any time in any release? If the latter, should there be some supported method (either static like JSON, or programmatic like Application.cfc) that all the settings for a server or web context can be stored in a portable fashion and even shipped as part of the app’s source code? (kind of like php.ini)

2

2 posts were merged into an existing topic: What’s the actual remit of this “language” forum?

4

Many other languages use YAML files or JSON files for all their configuration needs. I would personally prefer to see some kind of Lucee Struct in the form of a settings.lucee file which looks like the contents of the this scope in Applicaiton.cfc, something like:

settings.lucee

{
    datasource : { ..  }, 
    ormenabled : true, 
    mappings :  {
       '/fw1' : expandPath("/frameworks/fw1")

    }
}

Essentially what comes out of this file is the whole config and it can also be run at runtime*

*or it is basically a static file like XML but which uses the lucee shortcuts as defined in the lucee-server.xml file {{web-root}} etc.

5

Lots of requirements suggested here and elsewhere. I just want to throw my 2 cents in. I love the 12factor App. Particularly #3, where it talks about configuration being in environment. That suggests a workflow that looks something like this:

  1. set environment variables
  2. run executable

I love this workflow because it makes installation go away. You can spin up multiple processes on a server using the same code base. Creating multiple similar processes running on different ports, making it easy to scale out. Lucee doesn’t seem do this very well. Most app servers don’t make it easy.

4 Likes
6

No solution, but more of an observation at this point.

We avoid multiple applications running within a single JVM wherever possible; our ideal setup is a single application per instance of Lucee Server. Moving to a Docker installation or PaaS like installation, such as Heroku, in practice means single applications per instance. This presents quite different configuration problems to a team that wants to run 50 sites in a shared hosting environment on a single Lucee instance.

In a Docker world I don’t care at all about the distinction between these configurations as there is only a single context and a single application. What I really want is complete configuration via ENV variables – that is, variables delivered via the operational environment, not the Java servlet container, and not the code base. App specific ENV variable configuration might become very complicated if you are hoping to configure more than one web context and/or application.

The problem doesn’t really end with Lucee. I also have to configure Tomcat (or my servlet container of choice) and NGINX (or my web server of choice). In practice you have to tinker with your set up regardless; certainly Lucee could make life easier.

Lucee already helps a great deal by allowing you to configure application level settings in the Application.cfc. It would help if everything could be configured in this way, but i suspect there are a number of things that need to be configurable prior to Application.cfc getting invoked.

Perhaps we could start by working out what can be configured via the Application at runtime, and what has to be configured prior to that at compile time.

1 Like
7

Thought an example might be worthwhile… in FarCry Core you can configure the platform datasource using environment (ENV) variables like so:

Environment Configuration; infrastructure specific, and separate to the app

daemon-prime:
  image: 'quay.io/daemonite/daemon-prime:latest'
  environment:
    - 'COOKIE=SRV insert indirect nocache'
    - EXCLUDE_PORTS=8080
    - VIRTUAL_HOST=daemon-prime.modius.io
    - FARCRY_DSN=daemon
    - FARCRY_DBTYPE=mysql
    - FARCRY_DSN_CLASS=org.gjt.mm.mysql.Driver
    - 'FARCRY_DSN_CONNECTIONSTRING=jdbc:mysql://deepest-darkest.rds.amazonaws.com:3306/daemon_daemon?useUnicode=true&characterEncoding=UTF-8&allowMultiQueries=true&useLegacyDatetimeCode=true'
    - FARCRY_DSN_USERNAME=daemon
    - FARCRY_DSN_PASSWORD=encrypted:topsecretunicornblood
  target_num_containers: 2

Note that’s an actual YAML configuration file for Tutum; very similar to a Docker compose file.

Application.cfc Configuration; simplified for this example

<!--- set up the datasource settings --->
<cfset this.datasources[FARCRY_DSN] = {
	"class" = system.getEnv("FARCRY_DSN_CLASS"),
	"connectionString" = system.getEnv("FARCRY_DSN_CONNECTIONSTRING"),
	"database" = system.getEnv("FARCRY_DSN_DATABASE"),
	"driver" = system.getEnv("FARCRY_DSN_DRIVER"),
	"host" = system.getEnv("FARCRY_DSN_HOST"),
	"port" = system.getEnv("FARCRY_DSN_PORT"),
	"type" = system.getEnv("FARCRY_DSN_TYPE"),
	"url" = system.getEnv("FARCRY_DSN_URL"),
	"username" = system.getEnv("FARCRY_DSN_USERNAME"),
	"password" = system.getEnv("FARCRY_DSN_PASSWORD")
}>

This approach allows us to have a universal Docker image for the application, but configure the application dynamically via the environment for development, staging, production and so on.

Allowing for multiple datasources in a less specific way becomes significantly more complicated.

Long story short – its awesome to configure via the ENV but peoples requirements are going to vary wildly. I’ll have a chat internally about what our team thinks might add real value to our scenarios for configuration.

PS. Does Adobe ColdFusion allow datasource configuration in this way?

8

Ok where should i start?
I try first to comment @bdw429s first post on this topic.

cfadmin/Administartor.cfc

cfadmin always was only handled as internal tag that can change without warning from one release to an other. because of that it is also undocumented and suppressed by the documentation.

When we saw that there is a demand for a functionality like this we started the component org.lucee.cfml.Administartor that is auto imported to every template, like all components of this package. Sadly this component does not cover a lot of functionality, but at least everything is documented and testcases exist for it.

System Properties/Environment Variables

System properties supported by Lucee:

  • lucee.base.dir (base directory for the engine)
  • lucee.server.dir (server context, same as init param lucee-server-directory)
  • lucee.web.dir (web context, same as init param lucee-web-directory)
  • lucee.controller.interval (number of milliseconds between controller calls, 0 to disable controller [useful for benchmark testing etc])

It is planned to extend the supported system properties and provide the same for environment variables.

BUT we added something cool in addition in the last Lucee 5 version, in the lucee server and web xml you can now do this:

<regional locale="{env:locale}" timezone="{system:timezone}"/>

This can be done for all attributes in the server and web xml (not only file path). But that is still not all, Lucee also support this for all file path in the enviroment.

So you can do the following in your code:

dump(expandPath("{system:java.io.tmpdir}"));

or

dump(expandPath("{env:TMPDIR}"));
2 Likes
9

@micstriit The system and env variables are cool, but the only allow for simple variable substitutions, right? How would I declare 3 data sources via environment variables for instance?

org.lucee.cfml.Administrator that is auto imported to every template … at least everything is documented

@micstriit Where is this documented at? Searching for that exact CFC path on Google (in quotes) only returns links to the Lucee reports or forks of it. I found this link, but it seems more like a stub:

I also thought that Application.cfc supported pretty much every setting available in the admin, but in talking to someone trying to pre-configure a CFML application for deployment in CommandBox it seems that one can’t declare custom caches in Application.cfc. Is that the case? Is there a full list somewhere of what can and can’t be configured via Application.cfc?

10

We are current using puppet to deploy our servers and contexts and have ended up tweaking the config xml using augeas. I don’t mind what the format of that file is but it would be good to have it stable and documented so we can be confident in changing it. As the servers are generated automatically we don’t use the web/server admin browser interfaces and having passwords in bits of cfm to run seems like a bit of a bad idea.
I think ideally having a CLI tool for loading setting would be ideal if it could load a config file into the server or context from the command line and also report on the current setup. Even if we could just change the current xml file and get the server to reload it from the CLI that would be a great help.

2 Likes
11

As long as we keep the GUI for all settings, I think a settings file is also fine for settings file die hards.

having said that I use a json config for my apps (and my admin app configures the stack (Azure : iis/tomcat or ubuntu/tomcat)

12

This is an old thread, so maybe this isn’t the place to write this, but…

It would also be nice to include support for secrets, each of which comes from a file (a la docker).

I helped implement secrets support in the CommmandBox docker image (Docker ), which enables CFConfig to use docker secrets.

For Lucee’s docker image I’m also using Jinja2 templating with custom filters to achieve the same thing.

It would be nice to be able to be able to configure this stuff with secrets out of the box. For non-dockerized environments (where secret files wouldn’t always be found at /run/secrets/my_secret_name) there might need to be a controller variable set for specifying the secrets directory (e.g., SECRETS_LOCATION=/path/to/secrets); however, it could default to /run/secrets)

It also might be a good idea to have secrets available in, say, server.system.secrets, so app code can access it in a standard way.

I haven’t given this more than ten minutes of thought, so some of the above might be half-baked.

2 Likes