Is Anyone Using LogStash for Monitoring Logs From Their Production Servers?

Daniel_Jansen · February 25, 2015, 12:07pm

Title pretty much says it all: Is Anyone Using LogStash for Monitoring Logs

markdrew · February 25, 2015, 12:17pm

About to start using it for some clients so will let you know how we get on.On Wednesday, February 25, 2015 at 12:07:36 PM UTC, Daniel Jansen wrote:

Title pretty much says it all: Is Anyone Using LogStash for Monitoring
Logs From Their Production Servers?

I’m trying to find the best way to monitor my production environment and
make it quick and easy to debug when errors occur.

Just looking for advice to whether anyone is using it and any suggestions
on best practice or setup.

Thanks.

Dan

madmike_de · February 25, 2015, 12:28pm

Hi,

I’m using Elasticsearch and Kibana for searching in my database. I’ve written a small api wrapper if someone wants to use it.

The results with logstash (only testing here) are very impressive, I can absolutely recommend it. even as replacement/enhancement of BuglogHQ.

Best,

MichiVon: lucee@googlegroups.com [mailto:lucee@googlegroups.com] Im Auftrag von Mark Drew
Gesendet: Mittwoch, 25. Februar 2015 13:18
An: lucee@googlegroups.com
Betreff: [Lucee] Re: Is Anyone Using LogStash for Monitoring Logs From Their Production Servers?

About to start using it for some clients so will let you know how we get on.

On Wednesday, February 25, 2015 at 12:07:36 PM UTC, Daniel Jansen wrote:

Title pretty much says it all: Is Anyone Using LogStash for Monitoring Logs From Their Production Servers?

I’m trying to find the best way to monitor my production environment and make it quick and easy to debug when errors occur.

Just looking for advice to whether anyone is using it and any suggestions on best practice or setup.

Thanks.

Dan

–
You received this message because you are subscribed to the Google Groups “Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+unsubscribe@googlegroups.com mailto:lucee+unsubscribe@googlegroups.com .
To post to this group, send email to lucee@googlegroups.com mailto:lucee@googlegroups.com .
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/78923bd2-2458-47b8-b337-4621d0960c19%40googlegroups.com https://groups.google.com/d/msgid/lucee/78923bd2-2458-47b8-b337-4621d0960c19%40googlegroups.com?utm_medium=email&utm_source=footer .
For more options, visit https://groups.google.com/d/optout.

Chris_Blackwell · February 25, 2015, 12:48pm

yes we are.

we’re using ELK (Elasticsearch, logstash & kibana) for monitoring logs from
a lot of sources. For our internal webapps we have it reading from a
number of places.

our reverse proxies (apache httpd) write their logs directly in json
our apps run on embedded tomcat, those logs get sent to
/var/log/messages and that gets read
the apps use logbox, and as well as a filesystem appender we use an
appender that sends the logs as a json packet directly to redis

we use mod_unique_id in apache to tie front end and backend requests
together. The logbox appenders know how to get this value.

we use a minimal logstash agent config on each of the proxy and app servers
which ships the logs to a redis queue, we then have a central logstash
server that reads the queue, groks and transforms the logs (like geoip-ing
clients) and writes them into a small elasticsearch cluster.

if (when… our apps throw an error they present the user (these apps are
not public, they’re just internal) with a friendly message with an error id
in it. When they contact us for support we can stick that error id into
kibana and get back the log entry from the front end request and the
stacktrace from the app server.

its been really useful, and well worth the few days it took to get it up
and running.On 25 February 2015 at 12:07, Daniel Jansen <@Daniel_Jansen> wrote:

Title pretty much says it all: Is Anyone Using LogStash for Monitoring
Logs From Their Production Servers?

I’m trying to find the best way to monitor my production environment and
make it quick and easy to debug when errors occur.

Just looking for advice to whether anyone is using it and any suggestions
on best practice or setup.

Thanks.

Dan

–
You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/b06e1e69-901c-4882-8727-827d5c86e935%40googlegroups.com
https://groups.google.com/d/msgid/lucee/b06e1e69-901c-4882-8727-827d5c86e935%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

Igal · February 25, 2015, 4:48pm

+1 for ELK (or at least for E-K, LogStash is not that great on Windows
so I don’t use it)

I’m not sure how mod_unique_id works, but in nginx I use the standard
X-Request-Id header, then it’s accessible in Lucee via CGI.HTTP_X_REQUEST_ID

A better solution would be to generate the unique request id even if it
isn’t passed, as proposed in

Igal Sapir
Lucee Core Developer
Lucee.org http://lucee.org/On 2/25/2015 4:48 AM, Chris Blackwell wrote:

yes we are.

we’re using ELK (Elasticsearch, logstash & kibana) for monitoring logs
from a lot of sources. For our internal webapps we have it reading
from a number of places.

our reverse proxies (apache httpd) write their logs directly in json

our apps run on embedded tomcat, those logs get sent to
/var/log/messages and that gets read

the apps use logbox, and as well as a filesystem appender we use
an appender that sends the logs as a json packet directly to redis

we use mod_unique_id in apache to tie front end and backend requests
together. The logbox appenders know how to get this value.

we use a minimal logstash agent config on each of the proxy and app
servers which ships the logs to a redis queue, we then have a central
logstash server that reads the queue, groks and transforms the logs
(like geoip-ing clients) and writes them into a small elasticsearch
cluster.

if (when… our apps throw an error they present the user (these
apps are not public, they’re just internal) with a friendly message
with an error id in it. When they contact us for support we can stick
that error id into kibana and get back the log entry from the front
end request and the stacktrace from the app server.

its been really useful, and well worth the few days it took to get it
up and running.

On 25 February 2015 at 12:07, Daniel Jansen <@Daniel_Jansen mailto:Daniel_Jansen> wrote:
Title pretty much says it all: Is Anyone Using LogStash for
Monitoring Logs From Their Production Servers?

I'm trying to find the best way to monitor my production
environment and make it quick and easy to debug when errors occur.

Just looking for advice to whether anyone is using it and any
suggestions on best practice or setup.

Thanks.

Dan
-- 
You received this message because you are subscribed to the Google
Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to lucee+unsubscribe@googlegroups.com
<mailto:lucee+unsubscribe@googlegroups.com>.
To post to this group, send email to lucee@googlegroups.com
<mailto:lucee@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/b06e1e69-901c-4882-8727-827d5c86e935%40googlegroups.com
<https://groups.google.com/d/msgid/lucee/b06e1e69-901c-4882-8727-827d5c86e935%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
–
You received this message because you are subscribed to the Google
Groups “Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to lucee+unsubscribe@googlegroups.com
mailto:lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com
mailto:lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/CAB%3DtfTqhep2jLQpbAd8BQ3_wf5_AVJRQrxtmevDw02gDXR1Gpg%40mail.gmail.com
https://groups.google.com/d/msgid/lucee/CAB%3DtfTqhep2jLQpbAd8BQ3_wf5_AVJRQrxtmevDw02gDXR1Gpg%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

Igal · February 25, 2015, 5:33pm

it might be a nice thing to build in, especially if we can insert the
request id into lucee’s logging.
that’s exactly what I have in mind

between that and the Server.OS.Hostname (which is mentioned in the other
ticket, and will help in clustered environments) debugging hard-to-find
issues will become much easier.

Igal Sapir
Lucee Core Developer
Lucee.org http://lucee.org/On 2/25/2015 9:23 AM, Chris Blackwell wrote:

That’s exactly what it does Igal, just inserts a request id into the
headers, I forget the exact name - it’s probably configurable. We
check for the request id on our apps, and if it’s not present we do
generate one, as it’s passed further on down the chain in some of our
internal apis

it might be a nice thing to build in, especially if we can insert the
request id into lucee’s logging.

On 25 Feb 2015 16:49, “Igal @ Lucee.org” <@Igal mailto:Igal> wrote:

+1 for ELK (or at least for E-K, LogStash is not that great on
Windows so I don't use it)

I'm not sure how mod_unique_id works, but in nginx I use the
standard X-Request-Id header, then it's accessible in Lucee via
CGI.HTTP_X_REQUEST_ID

A better solution would be to generate the unique request id even
if it isn't passed, as proposed in
https://bitbucket.org/lucee/lucee/issue/100/support-x-request-id-header

Igal Sapir
Lucee Core Developer
Lucee.org <http://lucee.org/>

On 2/25/2015 4:48 AM, Chris Blackwell wrote:

yes we are.

we're using ELK (Elasticsearch, logstash & kibana) for monitoring
logs from a lot of sources.  For our internal webapps we have it
reading from a number of places.

  * our reverse proxies (apache httpd) write their logs directly
    in json
  * our apps run on embedded tomcat, those logs get sent to
    /var/log/messages and that gets read
  * the apps use logbox, and as well as a filesystem appender we
    use an appender that sends the logs as a json packet directly
    to redis 

we use mod_unique_id in apache to tie front end and backend
requests together.  The logbox appenders know how to get this value.

we use a minimal logstash agent config on each of the proxy and
app servers which ships the logs to a redis queue, we then have a
central logstash server that reads the queue, groks and
transforms the logs (like geoip-ing clients) and writes them into
a small elasticsearch cluster.

if (when..;-) our apps throw an error they present the user
(these apps are not public, they're just internal) with a
friendly message with an error id in it.  When they contact us
for support we can stick that error id into kibana and get back
the log entry from the front end request and the stacktrace from
the app server.

its been really useful, and well worth the few days it took to
get it up and running.

On 25 February 2015 at 12:07, Daniel Jansen <@Daniel_Jansen <mailto:@Daniel_Jansen>> wrote:

    Title pretty much says it all: Is Anyone Using LogStash for
    Monitoring Logs From Their Production Servers?

    I'm trying to find the best way to monitor my production
    environment and make it quick and easy to debug when errors
    occur.

    Just looking for advice to whether anyone is using it and any
    suggestions on best practice or setup.

    Thanks.

    Dan
    -- 
    You received this message because you are subscribed to the
    Google Groups "Lucee" group.
    To unsubscribe from this group and stop receiving emails from
    it, send an email to lucee+unsubscribe@googlegroups.com
    <mailto:lucee+unsubscribe@googlegroups.com>.
    To post to this group, send email to lucee@googlegroups.com
    <mailto:lucee@googlegroups.com>.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/lucee/b06e1e69-901c-4882-8727-827d5c86e935%40googlegroups.com
    <https://groups.google.com/d/msgid/lucee/b06e1e69-901c-4882-8727-827d5c86e935%40googlegroups.com?utm_medium=email&utm_source=footer>.
    For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the
Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to lucee+unsubscribe@googlegroups.com
<mailto:lucee+unsubscribe@googlegroups.com>.
To post to this group, send email to lucee@googlegroups.com
<mailto:lucee@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/CAB%3DtfTqhep2jLQpbAd8BQ3_wf5_AVJRQrxtmevDw02gDXR1Gpg%40mail.gmail.com
<https://groups.google.com/d/msgid/lucee/CAB%3DtfTqhep2jLQpbAd8BQ3_wf5_AVJRQrxtmevDw02gDXR1Gpg%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google
Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to lucee+unsubscribe@googlegroups.com
<mailto:lucee+unsubscribe@googlegroups.com>.
To post to this group, send email to lucee@googlegroups.com
<mailto:lucee@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/54EDFCEE.4060208%40lucee.org
<https://groups.google.com/d/msgid/lucee/54EDFCEE.4060208%40lucee.org?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google
Groups “Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to lucee+unsubscribe@googlegroups.com
mailto:lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com
mailto:lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/CAB%3DtfTo5LFOkk%3DQfzizJ0NHDw-Tz2_mqPiMvvPvxvW%2B0kvNUYw%40mail.gmail.com
https://groups.google.com/d/msgid/lucee/CAB%3DtfTo5LFOkk%3DQfzizJ0NHDw-Tz2_mqPiMvvPvxvW%2B0kvNUYw%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

James_Hooker · February 26, 2015, 12:20pm

Whoops - Gist is back at Logstash.cfm · GitHub

sorry about that!On Thursday, 26 February 2015 11:43:47 UTC, James Hooker wrote:

We have been using Logstash / Kibana for some time at mso in order to
track, and analyse errors from various services, including ColdFusion.

During a recent sprint, I was tasked with implementing a method to send
log entries from a Lucee instance to our central log server. You can find
my method at the following gist.

https://gist.github.com/Seidr/9e0fe93760a72497f5b3

You’ll need to build the JAR, and place it into your class path, as
defined in catalina.properties.

Would love some feedback - so far this appears to be working well for our
Lucee instances.

On Wednesday, 25 February 2015 12:07:36 UTC, Daniel Jansen wrote:

Title pretty much says it all: Is Anyone Using LogStash for Monitoring
Logs From Their Production Servers?

I’m trying to find the best way to monitor my production environment and
make it quick and easy to debug when errors occur.

Just looking for advice to whether anyone is using it and any suggestions
on best practice or setup.

Thanks.

Dan

mailme.gx · February 26, 2015, 9:58am

we are using logstash for some other log4j based files, although we have
not yet made a tomcat configuration we plan to do so soon, I find it a
great way to centralize logs of all the different services, although other
members of the team have other priorities and dont want to focus on this I
find it a big help to have a central log repository that is well indexed
and search-able.

GXOn Wednesday, February 25, 2015 at 2:07:36 PM UTC+2, Daniel Jansen wrote:

Title pretty much says it all: Is Anyone Using LogStash for Monitoring
Logs From Their Production Servers?

I’m trying to find the best way to monitor my production environment and
make it quick and easy to debug when errors occur.

Just looking for advice to whether anyone is using it and any suggestions
on best practice or setup.

Thanks.

Dan

James_Hooker · February 26, 2015, 11:43am

We have been using Logstash / Kibana for some time at mso in order to
track, and analyse errors from various services, including ColdFusion.

During a recent sprint, I was tasked with implementing a method to send log
entries from a Lucee instance to our central log server. You can find my
method at the following gist.

https://gist.github.com/Seidr/9e0fe93760a72497f5b3

You’ll need to build the JAR, and place it into your class path, as defined
in catalina.properties.

Would love some feedback - so far this appears to be working well for our
Lucee instances.On Wednesday, 25 February 2015 12:07:36 UTC, Daniel Jansen wrote:

Title pretty much says it all: Is Anyone Using LogStash for Monitoring
Logs From Their Production Servers?

I’m trying to find the best way to monitor my production environment and
make it quick and easy to debug when errors occur.

Just looking for advice to whether anyone is using it and any suggestions
on best practice or setup.

Thanks.

Dan

Daniel_Jansen · February 27, 2015, 12:20am

Unreal feedback everyone!

I just need to make some time to setup a test environment now. Sounds like
a really powerful tool to have in place.
Thanks for the resources documenting the setup - I’m sure I’ll be referring
to them often.

Regards,

Dan Jansen

P.S. Loving the buzz of activity of the Lucee community - there is a real
sense of an active community here.