BLOCKED on lucee.runtime.db.DCStack@1c1dfcc1

Jamie_Jackson · June 12, 2015, 6:02pm

Happy Friday!

We were load testing our app, and we saw that in FusionReactor, a bunch of
requests were hung (and couldn’t be terminated), and the load test server
became unresponsive.

The hung requests were stuck here:

“http-bio-8888-exec-46” Id=10795 BLOCKED on
lucee.runtime.db.DCStack@1c1dfcc1 owned by “http-bio-8888-exec-4” Id=72
java.lang.Thread.State: BLOCKED
at
lucee.runtime.db.DatasourceConnectionPool.getDatasourceConnection(Unknown
Source)
- waiting to lock lucee.runtime.db.DCStack@1c1dfcc1 owned by
“http-bio-8888-exec-4”
at lucee.runtime.db.DatasourceManagerImpl.getConnection(Unknown
Source)
at lucee.runtime.tag.Query.executeDatasoure(Unknown Source)
at lucee.runtime.tag.Query.doEndTag(Unknown Source)

Here’s the full thread dump:

gist.github.com

https://gist.github.com/jamiejackson/4b739ff5844423c1d9e0#file-gistfile1-txt-L296-L302

gistfile1.txt

"main" Id=1 RUNNABLE (in native)
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
        at java.net.ServerSocket.implAccept(ServerSocket.java:530)
        at java.net.ServerSocket.accept(ServerSocket.java:498)
        at org.apache.catalina.core.StandardServer.await(StandardServer.java:453)
        at org.apache.catalina.startup.Catalina.await(Catalina.java:777)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:723)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

This file has been truncated. show original

FWIW, this is Lucee 4.5.1.000 final with the stock MySQL driver.

Happen to know what’s going on here?

Thanks,
Jamie

micstriit · June 13, 2015, 6:09am

Can you raise a ticket for it, it is hard to say without having the line
numbers, newer versions will provide that info

MichaAm Freitag, 12. Juni 2015 schrieb Jamie Jackson :

Happy Friday!

We were load testing our app, and we saw that in FusionReactor, a bunch of
requests were hung (and couldn’t be terminated), and the load test server
became unresponsive.

The hung requests were stuck here:

“http-bio-8888-exec-46” Id=10795 BLOCKED on
lucee.runtime.db.DCStack@1c1dfcc1 owned by “http-bio-8888-exec-4” Id=72
java.lang.Thread.State: BLOCKED
at
lucee.runtime.db.DatasourceConnectionPool.getDatasourceConnection(Unknown
Source)
- waiting to lock lucee.runtime.db.DCStack@1c1dfcc1 owned by
“http-bio-8888-exec-4”
at lucee.runtime.db.DatasourceManagerImpl.getConnection(Unknown
Source)
at lucee.runtime.tag.Query.executeDatasoure(Unknown Source)
at lucee.runtime.tag.Query.doEndTag(Unknown Source)

Here’s the full thread dump:

gist:4b739ff5844423c1d9e0 · GitHub

FWIW, this is Lucee 4.5.1.000 final with the stock MySQL driver.

Happen to know what’s going on here?

Thanks,
Jamie

–
You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com
<javascript:_e(%7B%7D,‘cvml’,‘lucee%2Bunsubscribe@googlegroups.com’);>.
To post to this group, send email to lucee@googlegroups.com
<javascript:_e(%7B%7D,‘cvml’,‘lucee@googlegroups.com’);>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/CA%2BonWPdjXGE9PEtiH%3DnM%2Brp2meDcQBSi4ocgLCuhQWTiO7_UwQ%40mail.gmail.com
https://groups.google.com/d/msgid/lucee/CA%2BonWPdjXGE9PEtiH%3DnM%2Brp2meDcQBSi4ocgLCuhQWTiO7_UwQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

Jamie_Jackson · June 15, 2015, 5:50pm

Hi Micha,

As I was creating a ticket, production failed in the same way. Here’s the
ticket: [LDEV-396] - Lucee

Thanks,
JamieOn Sat, Jun 13, 2015 at 2:09 AM, Michael Offner <@Michael_Offner> wrote:

Can you raise a ticket for it, it is hard to say without having the line
numbers, newer versions will provide that info

Micha

Am Freitag, 12. Juni 2015 schrieb Jamie Jackson :

Happy Friday!

We were load testing our app, and we saw that in FusionReactor, a bunch
of requests were hung (and couldn’t be terminated), and the load test
server became unresponsive.

The hung requests were stuck here:

“http-bio-8888-exec-46” Id=10795 BLOCKED on
lucee.runtime.db.DCStack@1c1dfcc1 owned by “http-bio-8888-exec-4” Id=72
java.lang.Thread.State: BLOCKED
at
lucee.runtime.db.DatasourceConnectionPool.getDatasourceConnection(Unknown
Source)
- waiting to lock lucee.runtime.db.DCStack@1c1dfcc1 owned by
“http-bio-8888-exec-4”
at lucee.runtime.db.DatasourceManagerImpl.getConnection(Unknown
Source)
at lucee.runtime.tag.Query.executeDatasoure(Unknown Source)
at lucee.runtime.tag.Query.doEndTag(Unknown Source)

Here’s the full thread dump:

gist:4b739ff5844423c1d9e0 · GitHub

FWIW, this is Lucee 4.5.1.000 final with the stock MySQL driver.

Happen to know what’s going on here?

Thanks,
Jamie

–
You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/CA%2BonWPdjXGE9PEtiH%3DnM%2Brp2meDcQBSi4ocgLCuhQWTiO7_UwQ%40mail.gmail.com
https://groups.google.com/d/msgid/lucee/CA%2BonWPdjXGE9PEtiH%3DnM%2Brp2meDcQBSi4ocgLCuhQWTiO7_UwQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/CAG%2BEEBzuV%2Bf8S%3DRKktZng4a0bpLoqppaY46MrtOis8tpM7DmzA%40mail.gmail.com
https://groups.google.com/d/msgid/lucee/CAG%2BEEBzuV%2Bf8S%3DRKktZng4a0bpLoqppaY46MrtOis8tpM7DmzA%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

Jamie_Jackson · June 30, 2015, 7:39pm

I brought it up in https://cfml.slack.com today, too, but I’m hereby
looking to drum up some more interest in:
[LDEV-396] - Lucee

Thanks,
JamieOn Mon, Jun 15, 2015 at 1:50 PM, Jamie Jackson <@Jamie_Jackson> wrote:

Hi Micha,

As I was creating a ticket, production failed in the same way. Here’s the
ticket: [LDEV-396] - Lucee

Thanks,
Jamie

On Sat, Jun 13, 2015 at 2:09 AM, Michael Offner <@Michael_Offner> wrote:

Can you raise a ticket for it, it is hard to say without having the line
numbers, newer versions will provide that info

Micha

Am Freitag, 12. Juni 2015 schrieb Jamie Jackson :

Happy Friday!

We were load testing our app, and we saw that in FusionReactor, a bunch
of requests were hung (and couldn’t be terminated), and the load test
server became unresponsive.

The hung requests were stuck here:

“http-bio-8888-exec-46” Id=10795 BLOCKED on
lucee.runtime.db.DCStack@1c1dfcc1 owned by “http-bio-8888-exec-4” Id=72
java.lang.Thread.State: BLOCKED
at
lucee.runtime.db.DatasourceConnectionPool.getDatasourceConnection(Unknown
Source)
- waiting to lock lucee.runtime.db.DCStack@1c1dfcc1 owned by
“http-bio-8888-exec-4”
at lucee.runtime.db.DatasourceManagerImpl.getConnection(Unknown
Source)
at lucee.runtime.tag.Query.executeDatasoure(Unknown Source)
at lucee.runtime.tag.Query.doEndTag(Unknown Source)

Here’s the full thread dump:

gist:4b739ff5844423c1d9e0 · GitHub

FWIW, this is Lucee 4.5.1.000 final with the stock MySQL driver.

Happen to know what’s going on here?

Thanks,
Jamie

–
You received this message because you are subscribed to the Google
Groups “Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/CA%2BonWPdjXGE9PEtiH%3DnM%2Brp2meDcQBSi4ocgLCuhQWTiO7_UwQ%40mail.gmail.com
https://groups.google.com/d/msgid/lucee/CA%2BonWPdjXGE9PEtiH%3DnM%2Brp2meDcQBSi4ocgLCuhQWTiO7_UwQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/CAG%2BEEBzuV%2Bf8S%3DRKktZng4a0bpLoqppaY46MrtOis8tpM7DmzA%40mail.gmail.com
https://groups.google.com/d/msgid/lucee/CAG%2BEEBzuV%2Bf8S%3DRKktZng4a0bpLoqppaY46MrtOis8tpM7DmzA%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

Terry_Whitney · July 1, 2015, 1:46pm

I would bet its your MySQL Server hitting max connections.

post your my.cnfOn Friday, June 12, 2015 at 2:02:43 PM UTC-4, Jamie Jackson wrote:

Happy Friday!

We were load testing our app, and we saw that in FusionReactor, a bunch of
requests were hung (and couldn’t be terminated), and the load test server
became unresponsive.

The hung requests were stuck here:

“http-bio-8888-exec-46” Id=10795 BLOCKED on
lucee.runtime.db.DCStack@1c1dfcc1 owned by “http-bio-8888-exec-4” Id=72
java.lang.Thread.State: BLOCKED
at
lucee.runtime.db.DatasourceConnectionPool.getDatasourceConnection(Unknown
Source)
- waiting to lock lucee.runtime.db.DCStack@1c1dfcc1 owned by
“http-bio-8888-exec-4”
at lucee.runtime.db.DatasourceManagerImpl.getConnection(Unknown
Source)
at lucee.runtime.tag.Query.executeDatasoure(Unknown Source)
at lucee.runtime.tag.Query.doEndTag(Unknown Source)

Here’s the full thread dump:

gist:4b739ff5844423c1d9e0 · GitHub

FWIW, this is Lucee 4.5.1.000 final with the stock MySQL driver.

Happen to know what’s going on here?

Thanks,
Jamie

Terry_Whitney · July 2, 2015, 6:47pm

In RHEL 6 , bind mysql to localhost. Even if you have your firewall setup,
binding it seemingly adds a tiny bit of performance

Second, you can tweak the timeout settings

[mysqld]
interactive_timeout=40
wait_timeout=45

Those should be set to a second or so off from your application timeout
settings.

Additionally, set SElinux to disabled temporarily and then stress test your
application.On Friday, June 12, 2015 at 2:02:43 PM UTC-4, Jamie Jackson wrote:

Happy Friday!

We were load testing our app, and we saw that in FusionReactor, a bunch of
requests were hung (and couldn’t be terminated), and the load test server
became unresponsive.

The hung requests were stuck here:

“http-bio-8888-exec-46” Id=10795 BLOCKED on
lucee.runtime.db.DCStack@1c1dfcc1 owned by “http-bio-8888-exec-4” Id=72
java.lang.Thread.State: BLOCKED
at
lucee.runtime.db.DatasourceConnectionPool.getDatasourceConnection(Unknown
Source)
- waiting to lock lucee.runtime.db.DCStack@1c1dfcc1 owned by
“http-bio-8888-exec-4”
at lucee.runtime.db.DatasourceManagerImpl.getConnection(Unknown
Source)
at lucee.runtime.tag.Query.executeDatasoure(Unknown Source)
at lucee.runtime.tag.Query.doEndTag(Unknown Source)

Here’s the full thread dump:

gist:4b739ff5844423c1d9e0 · GitHub

FWIW, this is Lucee 4.5.1.000 final with the stock MySQL driver.

Happen to know what’s going on here?

Thanks,
Jamie