Elastic Search Application Performance monitoring for lucee

Tools and Libraries
1

In a few words,

For high node count installations, it’s quite costly to go through the existing commercial systems, as such, i took the same idea as lucee did a few years ago, and contributed a plugin to elastic’s apm agent. this allows to use an existing elastic cluster to gain app insights.

Out of necessity, i’ve battled with elastic-java-apm-agent to instrument the lucee server, the great news is that the latest version of the agent, can theoretically work on a “vanilla” version of lucee server (we’re currently running custom builds for other optimisation reasons). I can therefore link to the github.

This Implementation do instrument quite a lot, still a work in progress, but more test, more use always helps. Obviously, it requires to build the existing code, this is not that complex, and to simply start the server with the agent as java-agent.

Here is the link if anyone feels adventurous :

the branch you want is : feature/lucee-server-http-instrumentation

Currently implemented features:

  • outpgoing http with trace transmission,
  • Database and redis (via the existing plugins)
  • ressource operations
  • locks operations
  • all the application.cfc hooks
  • DbManager (connection retrieval and transactions operations)
  • Compilation spans
  • Java “CreateObject” layer
  • Image tag
  • _InternalRequest instrumentation as a sub transaction (i’m still struggling to separate the spancount limit).

I’m not certain a lot other operations are problematic, but, use case can help pinpoint to new propositions. Been running in production with a quite high traffic (largely over 1K transaction per minute) for ~1 year

Let me know if there is any interest.

Regards,
Xavier.

5 Likes
2

Hi Xavier

First of all let me thank you for this very cool work.
I also did some playing around with Elastic-APM a while ago for the same reason as yours (shit expensive existing APM solutions), and I really felt the need to have a custom plugin for Lucee.
Now I’m not as familiar with Lucee codebase to roll my own custom APM plugin from scratch, but your work definitely gives me the necessary platform + courage to do some more experiments and hopefully do some contributions in the plugin.

It would be really nice if you can help me(us, I’m sure other folks will also be interested in this) with the necessary setup, for development and testing.

3

Well,

Not sure what i can tell more, first you’ll need an elk setup with the elastic-apm ingester running, i won’t redo a tutorial. for the setup of an elk stack or the apm ingest point.

This will be a good starting point : Quick start | APM User Guide [8.6] | Elastic

For the linking it with lucee, I’ll first recommend you read the code to make sure i haven’t done anything nasty (i’ll repeat it if you want to run it in production), this is able to capture and will alter your http calls, so, gives basically the keys to your system.

To build the agent, well, basically, it’s something like :
clone the repository, checkout the branch:
feature/lucee-server-http-instrumentation which is a misnomer, because it went further.

./mvnw clean install -DskipTests=true -Dmaven.javadoc.skip=true # build only jar

You’ll find the .jar inside the build directory.

The last step would be to link those to your lucee,

I basically add this to tomcat’s setenv.sh


CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.service_name=${APM_SERVICE_NAME}"
CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.environment=${APM_ENVIRONMENT}"
CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.use_path_as_transaction_name=true"
CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.server_urls=${APM_ENDPOINT}"
CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.secret_token=${APM_TOKEN}"
CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.trace_methods_duration_threshold=0.1ms"
CATALINA_OPTS="$CATALINA_OPTS -Dlogging.level.co.elastic.apm=${APM_LOG_LEVEL}"
CATALINA_OPTS="$CATALINA_OPTS -javaagent:/usr/local/tomcat/lib/elastic-apm-agent.jar"

Let me know if this get’s you going.

Xavier.

1 Like
4

I already had the Elastic APM setup and working with default agent; I was asking more about the process to build and deploy process for the custom agent.

But, thanks, what you’ve explained is sufficient and will help everyone; I’ll give it a try over the weekend;