How do we make automating builds and deployments of Lucee applications rock?

hacking
1

This topic is an open discussion that is sort of a corollary to my previous post, http://lang.lucee.org/t/how-do-we-make-configuring-lucee-rock/193

In that post I’m talking about the different levels of configuration for a server/application and the formats/methodologies of storing that config (static XML, programmatic CFML, etc.) I also touched on how developers can interface or manage said config in that post and that’s what I’d like to unpack here.

Note, the driving force behind this question is automated deployment and provisioning of servers for cloud services, or local tools like Vagrant.

The server and web’s browser-based administrator sites are the two most well known methods of configuring Lucee, but both of these preclude any sort of automation.

The <cfadmin> tag is quite promising but it’s biggest issues is:

  • It’s not documented
  • It requires the server to be running
  • It must be run in the context of the server you want to administer

This doesn’t work for me when I want to write a shell provisioner for Lucee that configures settings on a new installation of Lucee that might not even be started yet for the first time.

Right now what I do is edit the lucee-server.xml and lucee-web.xml files directly which has these drawbacks:

  • They’re not documented
  • They’re not a public API and their format could change at any time
  • Copying the entire XML file will break later builds of Lucee that change the format of the XML
  • Parsing the XML and manipulating it is a bit better, but still tied to the undocumented format

We have rudimentary support for Java system properties in Lucee 5, but they appear to only affect the Lucee bootstrap and don’t allow actual settings to be configured. Roll those together with some kind of php.ini kind of concept and I think we’re getting close if

  • The format is documented
  • The config can be dropped in either the code repo or a server provisioner
  • It’s picked up at runtime-- need to decide hierarchy order of what settings overwrite what.

What I’d really like, and what spawned this entire discussion for me is a supported CLI for configuring Lucee. Let’s look at Apache webserver for inspiration. When installed, it comes with some handy binaries like

  • a2ensite (enable site)
  • a2dissite (disable site)
  • a2enmod (enable mod)
  • a2dismod (disable mod)

With this approach I can either type these commands one-off from my shell prompt to configure my installation or I can make them part of a provisioner script that adds the mods, and sites I need without having to touch config files at all!

$> a2enmod proxy proxy_ajp rewrite
$> a2dissite 000-default.conf

Those configuration bits are agnostic concerning the format of the underlying config files and are a documented public API so I’m safe to call it now and in the future.

This is where I see a huge opportunity. I’d like to see Lucee either modify the cfadmin tag to be able to be pointed to a context outside of where it is running, or possibly open up the Java classes that abstract the manipulation of Lucee’s XML configuration files. I’m thinking of a suite of CLI commands (probably CommandBox commands) that allow for the scripted-configuration of a Lucee server either for the convenience of a CLI-minded admin, or the automation of a Vagrant build.

Change all settings

CommandBox> lucee server inspect-template never
CommandBox> lucee web missing-template-error "error-public.cfm"
CommandBox> lucee server add-mapping virtual="/coldbox" physical="/var/www/libs/coldbox"

Perform administrative tasks against a running server

CommandBox> lucee server update latest stable
CommandBox> lucee server restart

Manage extensions

CommandBox> lucee web install-extension ortus-couchbase-cache
CommandBox> lucee web remove-extension cfbeer

And what about these? (not currently possible)

CommandBox> lucee web outdated-extensions
CommandBox> lucee web update-extensions

Furthermore, if CFML site can now list its dependencies in a box.json and ask CommandBox to “install” them npm-style, why can’t your Lucee server and web extensions be documented in the same manner. Then building out your Lucee server that requires the Ortus Couchbase cache extension, cfspreadsheet, DB2 JDBC driver and CF ORM (these last two will become extensions in Lucee 5!) could be as simple as specifying those as dependencies for your server and then asking the CLI to “install” the server for you.

CommandBox> lucee server install

I can see a whole new world of portability and management coming about for Lucee that will be a must in the world of automated builds and cloud deployment. Thoughts?

1 Like
2

Use Docker images… :whale: :smile:

3

I agree about the <cfadmin> comments. It also has to have the admin password in plain text so it makes it a little less than secure in my book.
If the config file format could be treated as an API and locked down and made public then that would make it a lot simpler to write a tool above.
We have been using Augeas via Puppet to alter that xml file and so far it’s working! Maybe providing an augeas lens for that file would be a big help.

4

Well, actually that’s one of the things that I think makes Lucee more secure-- the fact that each request must be authenticated. It’s really not much different from Adobe CF-- you’d still need the password in your source. If you’re automating the setup of a box from scratch, I’m sure you’d have the default password somewhere in the config so it could be set anyway. I don’t necessarily think auth is that important though for a CLI tool that’s run locally on the same server. If you have access to that server’s file system, you pretty much have full control anyway.

Maybe providing an augeas lens for that file would be a big help

I don’t know anything about Augeas (other than what I just Googled) but I like the sound of that. I had actually talked before internally about whether Lucee (or its community) should work on provisioning tools specifically to integrate with the tools out there like Chef, puppet, etc.

5

I see what you mean about having to authenticate each cfadmin request. That is a good thing. I would just rather not have that password in a bunch scripts in the lucee server webroot.

The external tool should also require authentication or root privileges to run so that anybody who gets a script to run on the box can’t just start changing settings.

I think the use case that the web admin was created for is different to how some people are now deploying applications. We are building boxes automatically and setting all the configuration at build time. My goal would be to remove the admin pages all together. Maybe shipping the whole lucee application as a WAR or even self contained JAR that is already configured. If a setting needs to be changed that can be done in a new build and pushed live.

To do this the simplest thing would be for a stable and documented config file to exist in any format. At the moment it does feel a bit like we are tweaking things behind the servers back and can only guess if it’s doing what the admin interface would do.

1 Like
6

I think these would be the way to go. I am installing commandbox into more and more servers to run tests so it being part of my build process would be fine.

As @modius says though the process with docker now is :

  1. Build an image FROM lucee/lucee4(or 5), COPY your source code
  2. Start up the server, make modifications to the config
  3. Build another image from this image
  4. Push the new image.
    This is sub optimal and it would be better for the process to be ;
  5. Build an image FROM lucee/lucee4(or 5), COPY your source code
  6. CMD apt-get commandbox
  7. EXEC commandbox install commands
  8. push image to repo.

So I am all for CLI system changes WITHOUT starting the server.

7

Point of order… Brad’s example of a2enmod/a2ensite are actually not Apache constructs, but scripts that manipulate files.

The only part of that Apache actually did was implement the ability to import files into the config structure, specifically, a directory of files sorted by name.

So if sites-available/default.conf exists, a2ensite default makes a symbolic link to that file in sites-enabled/default.conf. a2dissite removes the link. All Apache implemented was IncludeOptional sites-enabled/*.conf

(this is very common in the Linux world, i.e. init.d, various directories in /etc, Redhat and Debian do this sort of thing everywhere - because individual config files could come from any of a variety of maintained packages)

One could also argue you’re just as unprotected from config syntax changes in these files in the Apache world… It’s just they have documented the format. (vs lucee)

Were we to suggest this model for configuring Lucee, it would be equivalent to including additional xml files in the config at boot. (i.e. instead of just lucee-web.xml.cfm, it would read lucee-web.xml.cfm, and any file from lucee-web.xml.d/*.xml, in order sorted by file name) Much like Tomcat reads things from conf/Catalina/localhost/ at boot.

In which case, configuring the server could be done through application specific files, or completely exploded out. (i.e. one xml file per datasource, one per mapping, one per whatever you want) So “lucee add dsn” might add a file called dsn-SomeName.xml.

The docker images could be structured so that instead of a monolithic lucee-web.xml.cfm file, all the config changes are represented in individual files.

I’ve adopted this technique to manage my web.xml with ACF. On my dev machine for instance:

web.xml.d/01header.xml
web.xml.d/10context-param.d/05coldfusion.xml
web.xml.d/10context-param.d/90sapphire_msb.xml
web.xml.d/20filter.d/02SearchEngineFriendlyURLFilter.xml
web.xml.d/20filter.d/03Cookies.disabled
web.xml.d/20filter.d/04javamelody.xml
web.xml.d/20filter.d/05seefusion.xml
web.xml.d/20filter.d/10CFMonitoringFilter.xml
web.xml.d/20filter.d/20CFCacheFilter.disabled
web.xml.d/20filter.d/30FlashRemotingControlFilter.disabled
web.xml.d/20filter.d/40WSRPFilter.disabled
web.xml.d/20filter.d/50CFClickJackFilterDeny.xml
web.xml.d/20filter.d/60CFClickJackFilterSameOrigin.xml
web.xml.d/30filter-mapping.d/02SESSearchEngineFriendlyURLFilter.xml
web.xml.d/30filter-mapping.d/03Cookies.disabled
web.xml.d/30filter-mapping.d/04javamelody.xml
web.xml.d/30filter-mapping.d/05seefusion.xml
web.xml.d/30filter-mapping.d/20CFClickJackFilterSameOrigin.xml
web.xml.d/30filter-mapping.d/30CFMonitoringFilter.disabled
web.xml.d/30filter-mapping.d/40FlashRemotingControlFilter.disabled
web.xml.d/30filter-mapping.d/50CFCacheFilter.disabled
web.xml.d/30filter-mapping.d/60WSRPFilter.disabled
web.xml.d/40listener.d/04javamelody.xml
web.xml.d/40listener.d/40HttpFlexSessionBootstrap.disabled
web.xml.d/40listener.d/85sapphire_msb.xml
web.xml.d/50servlet.d/270_MessageBrokerServlet.disabled
web.xml.d/50servlet.d/275_flex2.disabled
web.xml.d/50servlet.d/280_CFInternalServlet.disabled
web.xml.d/50servlet.d/290_CFSwfServlet.disabled
web.xml.d/50servlet.d/300_CFMxmlServlet.disabled
web.xml.d/50servlet.d/310_CFForbiddenServlet.xml
web.xml.d/50servlet.d/320_ColdFusionStartUpServlet.xml
web.xml.d/50servlet.d/330_FlashGateway.disabled
web.xml.d/50servlet.d/340_CFFormGateway.disabled
web.xml.d/50servlet.d/350_CfmServlet.xml
web.xml.d/50servlet.d/360_CFCServlet.xml
web.xml.d/50servlet.d/370_ServerCFCServlet.xml
web.xml.d/50servlet.d/380_CFRestServlet.disabled
web.xml.d/50servlet.d/390_RDSServlet.xml
web.xml.d/50servlet.d/400_JSDebugServlet.xml
web.xml.d/50servlet.d/410_CFFileServlet.xml
web.xml.d/50servlet.d/420_WSRPProducer.disabled
web.xml.d/50servlet.d/85sapphire_msb.xml
web.xml.d/60servlet-mapping.d/10_MessageBroker.disabled
web.xml.d/60servlet-mapping.d/14flex2.disabled
web.xml.d/60servlet-mapping.d/15FlashGateway.disabled
web.xml.d/60servlet-mapping.d/20_cfml.xml
web.xml.d/60servlet-mapping.d/45NativeSES.disabled
web.xml.d/60servlet-mapping.d/48RDS.xml
web.xml.d/60servlet-mapping.d/49JSDebug.xml
web.xml.d/60servlet-mapping.d/50jws.disabled
web.xml.d/60servlet-mapping.d/51cfr.disabled
web.xml.d/60servlet-mapping.d/52CFFormGateway.disabled
web.xml.d/60servlet-mapping.d/53CFFileServlet.xml
web.xml.d/60servlet-mapping.d/55API.disabled
web.xml.d/60servlet-mapping.d/56DenyHBM.xml
web.xml.d/60servlet-mapping.d/58CFForm.disabled
web.xml.d/60servlet-mapping.d/60Forbidden.xml
web.xml.d/60servlet-mapping.d/62WSRP.disabled
web.xml.d/60servlet-mapping.d/90sapphire_msb.xml
web.xml.d/62session.xml
web.xml.d/64welcome.xml
web.xml.d/69secure.disabled
web.xml.d/70taglib.d/10flex2.disabled
web.xml.d/70taglib.d/87sapphire_msb.xml
web.xml.d/92WebSphere.disabled
web.xml.d/99footer.xml

Notes:

  1. It’s easy to disable something. Rename it from *.xml to *.disabled
  2. Things can be in different packages. Seefusion, for example, has a couple files that can be in their own package. If the file is present, it’s in the config in the right spot. If it’s not, it won’t be in the web.xml at all.
  3. There aren’t commented out blocks of code in my web.xml

Which goes with the associated script (since tomcat can’t assemble these files on its own)

#!/bin/sh

find web.xml.d/ -type f -iname "*.xml" |sort | while read i; do
  if [ -f "$i" ]; then
    echo '<!-- '"$i"' -->'
    #echo
    cat $i
    echo ""
    #echo ""
  fi
done |fromdos > web.xml.tmp
xmllint --format --encode UTF-8 web.xml.tmp > web.xml

My point is that this structure could be implemented in Docker machines or vagrant configurations without any changes in Lucee. Personally I think this structure is more flexible than the structure implemented by Augeas, but that’s my opinion. (Plus it can be easily customized by the user) One could also argue that all these files could be collapsed after the container is built. (rm -rf the web.xml.d directory after the web.xml is generated, s/web.xml/lucee-web.xml.cfm/g; as well) Just like I would suggest that the last step of a docker build with commandbox be to delete commandbox. i.e. apt-get purge commandbox, apt-get autoremove, rm -rf ~/.box

Or perhaps cfadmin could behave differently if it’s being run from a command line context… Some systems (like postgresql) treat local access differently. I.e. hit the cfm page from a webpage, you need a password. Use REPL or CommandBox commandline access, and you have somewhat higher privileges.

Or, in ACF, you can make the administrator not have a password at all… by manipulating properties files. Drop out the properties, do your admin changes, put it back?

Or maybe Lucee can provide a command line cfg mgmt api? Enough to load the config management, parse the config file, make indicated changes, and save the file back?

Is the problem here that the .xml format for Lucee isn’t documented? If so, that should be fixed, and tools should be provided to migrate between xml formats if/when upgrades occur. (Just like ACF administrator imports prior cf10settings cf11settings etc) Easily done with XSLT or a quick script provided BY Lucee and checked into the repo when file formats change, and tested with unit tests. (CFML, Perl, Java, Groovy, or language du jour)

Is the problem here that the .xml format is monolithic and hard to maintain in a modular manner? If so, there are ways to deal with that outside Lucee, as illustratred.

Is the problem that the server has to be online to change the config through apis? If so, the question is are APIs the answer, or should Lucee provide a command line tool, or a minimal server that can just provide the cfadmin support, or is the answer to document the config file structure? (And once that’s done, don’t change it in a way that breaks backwards compatibility) Having an XSD, for example, would give some level of documentation and the ability to validate the config. (disclaimer: I didn’t actually check to see if such an XSD already exists)

Were such an XSD available, and migration tools… My next question would be - why isn’t an includable lucee.xml file checked into your source repo?

I love the commandbox way of doing things and I’d like to see support like what Brad outlines… Just tossing out ideas to make sure we’re implementing the right thing. Ultimately commandbox could work with any of the schemes above. Maybe all commandbox has to do is generate appropriate xml snippets to a documented schema.

8

Just to clarify…

It is more usual to programatically configure what you can. Those configuration options that can only be administered through the Admin are the issue. The approach in Docker :whale: is to configure the Admin in a running container, and then to capture the resulting XML configs. These are then checked into version control and copied into the Image as part of the build process.

At least that’s how we do it.

First priority has to be to get all things configurable in a way that doesn’t require starting the server or installing a CLI :slight_smile: Ideally, it would be great to have every element that can be programatically configured available as an option in the Application.cfc. To be fair 5.0-5.1 have made some considerable progress towards that goal.

An extremely important aspect of configuration, in Docker at least, is the ability to configure the environment by passing in Environment variables at run time.

PS. As an aside, it would always be best to COPY source last in the Dockerfile; ie. most volatile at the end

9

Unfortunately, while it removes CommandBox as a potential vector, it still leaves it in the layers that make up the resulting Docker :whale: image. As much as I admire Commandbox, I don’t think it should become the defacto standard for server configuration.

For me the problem is translating configuration from environment variables to the server instance.

Anything to do with XML needs to be either:

a) considered at build time; and so not open to configuration at run time, or;
b) administered via some sort of bootstrapping mechanism when the container starts

While b) is not an uncommon approach, I can’t help feeling the ability to configure the server through the Application.cfc would be ideal.

Perhaps we could have a greatly simplified XML config for compile time directives only, and a separate optional config file for run time configuration that can be completely overridden by the Application.cfc at application start.

Docker is certainly a simpler environment, primarily because we focus on a single web context per container instance. I suspect the need for server level and web context level changes in configuration might present additional complexity.

10

I’d respond inline but I don’t know how to work this forum. :-/

Anywho.

If commandbox is added and removed in the same build step, it shouldn’t exist in any of the docker layers.

It seems to me you’re looking for a runtime configuration solution for docker. (i.e. run with ENV variable, set config once the server starts) In which case, yes, if commandbox is the vehicle, you need to keep it in the container.

But you could easily run a script instead of Lucee directly - so you could run a shell script first, that reads the environment and fires off appropriate XSLT to manipulate on-disk XML before the server starts.

Or perhaps you use a volume-map to map to a source config folder and cp the lucee xmls first. (or wget them from a central location, or git checkout them from a managed location)

The docker concerns, since they’re runtime configuration… I’m not sure why you’d want to do the config before the server starts. (As you say, most of the time it’s done in Application.cfc) That gives the user the ability to define whatever config they want. You could even do what vmware does with ovfdep/vApps and just have an environment xml structure stuffed somewhere.

I had assumed the thread was more about offline-confg…I.e. commandbox’s concern is usually

  1. I just cloned something from github and need to configure the runtime server
  2. I just cloned something form github and want to install dependencies (extensions et al) into Lucee
  3. I am about to start a server and want to configure it before I do

Were I to do a docker container, I’d likely build a base image and then have variations on the theme for configuration… The config would be part of the image, not in the environment. Or I’d pull anything volatile from a database, etcd, Redis, or something where it could be managed.

Just my $.02

So, Geoff, in your case why don’t the current mechanisms available in Application.cfc work for you?

Or are we talking about Application = Web context, and therefore Application.cfc should have more (implicit) rights?

Or are we talking about having more server global config options overridable by Application?

11

For the most part they do. There are only a few remaining areas that are awkward (for our needs) and we have work arounds for these; for example, SMTP server config.

I’m not sure there is much we disagree on. Whatever the approach, it should not require anyone to ever login to the Admin; if I could ship containers without the admin installed i would.

We have a few legacy projects (not all Lucee based) that utilise things like envsubst for variable substitution, and bootstrap scripts for container starts. Perhaps these are awkward because the underlying processes config formats are awkward. I find Lucee’s current XML format awkward; maybe things just need simplification and documentation.

Some of my late night :sleeping: concerns include:

  • a clearly articulated separation of runtime and compile time directives
  • preference for Lucee to have an internal default configuration, that can be optionally overridden by a simple configuration file; ie. no config file needed or very short config files needed
  • all run time configuration to be programmatically configurable

Perfectly true, if you can fit it into a single step. But i still don’t think Commandbox should be the defacto standard for server configuration :wink:

12

Just stepping into this space what, 7 years later…
Has Lucee adopted any of these ideas or solved for this in any way by now?

13

HI @Datico I created the library CFConfig out of this conversation. It is a module for the CommandBox CLI which allows

  • export of CF config to JSON
  • import of JSON to CF server
  • JSON files support all possible system setting expansions in the form of ${FOO} so JSON files are safe to commit and can be re-used across environments
  • Comprehensive diffing settings between servers/JSON and export these results to HTML/PDF
  • support for both Adobe ColdFusion and Lucee Server in the same JSON format
  • Get and Set individual settings
  • A suite of CLI commands for manually creating datasources, event gateways, mail servers, caches, etc
  • Ability to import or export only parts of a JSON file by key name
  • Ability to overwrite or “append” config on top of existing config (such as adding DSN’s without removing existing)
  • Ability to export settings to JSON with automatic replacement of sensitive values with ${DB_HOST} style placeholders to fully automate extraction of settings

CFConfig can be used to manage any CF server’s config, even if it is installed on Tomcat, etc. CFConfig, being a CommandBox module, also has a deep integration to CommandBox servers such that simply running server start will do the following out of the box:

  • find and import .cfconfig.json, .cfconfig-web.json and/or .cfconfig-server.json files automatically by convention
  • Will import files not in the convention location or having the default name as specified in server.json
  • Is capable of handling a single web context or as many web contexts as you like for ModCFML support
  • FULL env var support for every possible Lucee setting which can be used to add a datasource, override just a hostname in a single Datasource definition or default the Lucee admin password on the fly without any JSON (or in addition to) which works great in Docker

Furthermore, the CFConfig (and dotenv) modules are installed standard in the ortussolutions/commandbox Docker Image so you can fully load in your CF config by convention right as the server starts up and it’s ready to go with no need for web-admin access.

Here are the docs for CFConfig

2 Likes
14

Lucee has adopted the configImport approach which @bdw429s pioneered and it is now supported it natively, via several methods

the latest release, 5.3.10.120 has some additional fixes for configImport()

there are some additional methods for deploying extensions / updates via the /deploy folder on startup

For warming up images for fast deployment, there LUCEE_ENABLE_WARMUP env var

And there are also startup listeners which can be used to programmatically configure your Lucee server using good old CFML

Don’t forget Lucee is open source, so anything you can do via the Lucee admin, can be done in cfml!

Lastly, for running CI with Lucee, we have developed the script-runner, which can be used to run test cases in CI, it’s headless, so there’s no http server, but it’s super quick

Pretty much all the Lucee repos use script-runner to run their tests using Github Actions

4 Likes