BLOCKED on lucee.runtime.db.DCStack@1c1dfcc1

Happy Friday!

We were load testing our app, and we saw that in FusionReactor, a bunch of
requests were hung (and couldn’t be terminated), and the load test server
became unresponsive.

The hung requests were stuck here:

“http-bio-8888-exec-46” Id=10795 BLOCKED on
lucee.runtime.db.DCStack@1c1dfcc1 owned by “http-bio-8888-exec-4” Id=72
java.lang.Thread.State: BLOCKED
at
lucee.runtime.db.DatasourceConnectionPool.getDatasourceConnection(Unknown
Source)
- waiting to lock lucee.runtime.db.DCStack@1c1dfcc1 owned by
“http-bio-8888-exec-4”
at lucee.runtime.db.DatasourceManagerImpl.getConnection(Unknown
Source)
at lucee.runtime.tag.Query.executeDatasoure(Unknown Source)
at lucee.runtime.tag.Query.doEndTag(Unknown Source)

Here’s the full thread dump:

FWIW, this is Lucee 4.5.1.000 final with the stock MySQL driver.

Happen to know what’s going on here?

Thanks,
Jamie

Can you raise a ticket for it, it is hard to say without having the line
numbers, newer versions will provide that info

MichaAm Freitag, 12. Juni 2015 schrieb Jamie Jackson :

Happy Friday!

We were load testing our app, and we saw that in FusionReactor, a bunch of
requests were hung (and couldn’t be terminated), and the load test server
became unresponsive.

The hung requests were stuck here:

“http-bio-8888-exec-46” Id=10795 BLOCKED on
lucee.runtime.db.DCStack@1c1dfcc1 owned by “http-bio-8888-exec-4” Id=72
java.lang.Thread.State: BLOCKED
at
lucee.runtime.db.DatasourceConnectionPool.getDatasourceConnection(Unknown
Source)
- waiting to lock lucee.runtime.db.DCStack@1c1dfcc1 owned by
“http-bio-8888-exec-4”
at lucee.runtime.db.DatasourceManagerImpl.getConnection(Unknown
Source)
at lucee.runtime.tag.Query.executeDatasoure(Unknown Source)
at lucee.runtime.tag.Query.doEndTag(Unknown Source)

Here’s the full thread dump:

https://gist.github.com/jamiejackson/4b739ff5844423c1d9e0#file-gistfile1-txt-L296-L302

FWIW, this is Lucee 4.5.1.000 final with the stock MySQL driver.

Happen to know what’s going on here?

Thanks,
Jamie


You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com
<javascript:_e(%7B%7D,‘cvml’,‘lucee%2Bunsubscribe@googlegroups.com’);>.
To post to this group, send email to lucee@googlegroups.com
<javascript:_e(%7B%7D,‘cvml’,‘lucee@googlegroups.com’);>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/CA%2BonWPdjXGE9PEtiH%3DnM%2Brp2meDcQBSi4ocgLCuhQWTiO7_UwQ%40mail.gmail.com
https://groups.google.com/d/msgid/lucee/CA%2BonWPdjXGE9PEtiH%3DnM%2Brp2meDcQBSi4ocgLCuhQWTiO7_UwQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

Hi Micha,

As I was creating a ticket, production failed in the same way. Here’s the
ticket: https://luceeserver.atlassian.net/browse/LDEV-396

Thanks,
JamieOn Sat, Jun 13, 2015 at 2:09 AM, Michael Offner <@Michael_Offner> wrote:

Can you raise a ticket for it, it is hard to say without having the line
numbers, newer versions will provide that info

Micha

Am Freitag, 12. Juni 2015 schrieb Jamie Jackson :

Happy Friday!

We were load testing our app, and we saw that in FusionReactor, a bunch
of requests were hung (and couldn’t be terminated), and the load test
server became unresponsive.

The hung requests were stuck here:

“http-bio-8888-exec-46” Id=10795 BLOCKED on
lucee.runtime.db.DCStack@1c1dfcc1 owned by “http-bio-8888-exec-4” Id=72
java.lang.Thread.State: BLOCKED
at
lucee.runtime.db.DatasourceConnectionPool.getDatasourceConnection(Unknown
Source)
- waiting to lock lucee.runtime.db.DCStack@1c1dfcc1 owned by
“http-bio-8888-exec-4”
at lucee.runtime.db.DatasourceManagerImpl.getConnection(Unknown
Source)
at lucee.runtime.tag.Query.executeDatasoure(Unknown Source)
at lucee.runtime.tag.Query.doEndTag(Unknown Source)

Here’s the full thread dump:

https://gist.github.com/jamiejackson/4b739ff5844423c1d9e0#file-gistfile1-txt-L296-L302

FWIW, this is Lucee 4.5.1.000 final with the stock MySQL driver.

Happen to know what’s going on here?

Thanks,
Jamie


You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/CA%2BonWPdjXGE9PEtiH%3DnM%2Brp2meDcQBSi4ocgLCuhQWTiO7_UwQ%40mail.gmail.com
https://groups.google.com/d/msgid/lucee/CA%2BonWPdjXGE9PEtiH%3DnM%2Brp2meDcQBSi4ocgLCuhQWTiO7_UwQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/CAG%2BEEBzuV%2Bf8S%3DRKktZng4a0bpLoqppaY46MrtOis8tpM7DmzA%40mail.gmail.com
https://groups.google.com/d/msgid/lucee/CAG%2BEEBzuV%2Bf8S%3DRKktZng4a0bpLoqppaY46MrtOis8tpM7DmzA%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

I brought it up in https://cfml.slack.com today, too, but I’m hereby
looking to drum up some more interest in:
https://luceeserver.atlassian.net/browse/LDEV-396

Thanks,
JamieOn Mon, Jun 15, 2015 at 1:50 PM, Jamie Jackson <@Jamie_Jackson> wrote:

Hi Micha,

As I was creating a ticket, production failed in the same way. Here’s the
ticket: https://luceeserver.atlassian.net/browse/LDEV-396

Thanks,
Jamie

On Sat, Jun 13, 2015 at 2:09 AM, Michael Offner <@Michael_Offner> wrote:

Can you raise a ticket for it, it is hard to say without having the line
numbers, newer versions will provide that info

Micha

Am Freitag, 12. Juni 2015 schrieb Jamie Jackson :

Happy Friday!

We were load testing our app, and we saw that in FusionReactor, a bunch
of requests were hung (and couldn’t be terminated), and the load test
server became unresponsive.

The hung requests were stuck here:

“http-bio-8888-exec-46” Id=10795 BLOCKED on
lucee.runtime.db.DCStack@1c1dfcc1 owned by “http-bio-8888-exec-4” Id=72
java.lang.Thread.State: BLOCKED
at
lucee.runtime.db.DatasourceConnectionPool.getDatasourceConnection(Unknown
Source)
- waiting to lock lucee.runtime.db.DCStack@1c1dfcc1 owned by
“http-bio-8888-exec-4”
at lucee.runtime.db.DatasourceManagerImpl.getConnection(Unknown
Source)
at lucee.runtime.tag.Query.executeDatasoure(Unknown Source)
at lucee.runtime.tag.Query.doEndTag(Unknown Source)

Here’s the full thread dump:

https://gist.github.com/jamiejackson/4b739ff5844423c1d9e0#file-gistfile1-txt-L296-L302

FWIW, this is Lucee 4.5.1.000 final with the stock MySQL driver.

Happen to know what’s going on here?

Thanks,
Jamie


You received this message because you are subscribed to the Google
Groups “Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/CA%2BonWPdjXGE9PEtiH%3DnM%2Brp2meDcQBSi4ocgLCuhQWTiO7_UwQ%40mail.gmail.com
https://groups.google.com/d/msgid/lucee/CA%2BonWPdjXGE9PEtiH%3DnM%2Brp2meDcQBSi4ocgLCuhQWTiO7_UwQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.


You received this message because you are subscribed to the Google Groups
“Lucee” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to lucee+unsubscribe@googlegroups.com.
To post to this group, send email to lucee@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/lucee/CAG%2BEEBzuV%2Bf8S%3DRKktZng4a0bpLoqppaY46MrtOis8tpM7DmzA%40mail.gmail.com
https://groups.google.com/d/msgid/lucee/CAG%2BEEBzuV%2Bf8S%3DRKktZng4a0bpLoqppaY46MrtOis8tpM7DmzA%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

I would bet its your MySQL Server hitting max connections.

post your my.cnfOn Friday, June 12, 2015 at 2:02:43 PM UTC-4, Jamie Jackson wrote:

Happy Friday!

We were load testing our app, and we saw that in FusionReactor, a bunch of
requests were hung (and couldn’t be terminated), and the load test server
became unresponsive.

The hung requests were stuck here:

“http-bio-8888-exec-46” Id=10795 BLOCKED on
lucee.runtime.db.DCStack@1c1dfcc1 owned by “http-bio-8888-exec-4” Id=72
java.lang.Thread.State: BLOCKED
at
lucee.runtime.db.DatasourceConnectionPool.getDatasourceConnection(Unknown
Source)
- waiting to lock lucee.runtime.db.DCStack@1c1dfcc1 owned by
“http-bio-8888-exec-4”
at lucee.runtime.db.DatasourceManagerImpl.getConnection(Unknown
Source)
at lucee.runtime.tag.Query.executeDatasoure(Unknown Source)
at lucee.runtime.tag.Query.doEndTag(Unknown Source)

Here’s the full thread dump:

https://gist.github.com/jamiejackson/4b739ff5844423c1d9e0#file-gistfile1-txt-L296-L302

FWIW, this is Lucee 4.5.1.000 final with the stock MySQL driver.

Happen to know what’s going on here?

Thanks,
Jamie

In RHEL 6 , bind mysql to localhost. Even if you have your firewall setup,
binding it seemingly adds a tiny bit of performance

Second, you can tweak the timeout settings

[mysqld]
interactive_timeout=40
wait_timeout=45

Those should be set to a second or so off from your application timeout
settings.

Additionally, set SElinux to disabled temporarily and then stress test your
application.On Friday, June 12, 2015 at 2:02:43 PM UTC-4, Jamie Jackson wrote:

Happy Friday!

We were load testing our app, and we saw that in FusionReactor, a bunch of
requests were hung (and couldn’t be terminated), and the load test server
became unresponsive.

The hung requests were stuck here:

“http-bio-8888-exec-46” Id=10795 BLOCKED on
lucee.runtime.db.DCStack@1c1dfcc1 owned by “http-bio-8888-exec-4” Id=72
java.lang.Thread.State: BLOCKED
at
lucee.runtime.db.DatasourceConnectionPool.getDatasourceConnection(Unknown
Source)
- waiting to lock lucee.runtime.db.DCStack@1c1dfcc1 owned by
“http-bio-8888-exec-4”
at lucee.runtime.db.DatasourceManagerImpl.getConnection(Unknown
Source)
at lucee.runtime.tag.Query.executeDatasoure(Unknown Source)
at lucee.runtime.tag.Query.doEndTag(Unknown Source)

Here’s the full thread dump:

https://gist.github.com/jamiejackson/4b739ff5844423c1d9e0#file-gistfile1-txt-L296-L302

FWIW, this is Lucee 4.5.1.000 final with the stock MySQL driver.

Happen to know what’s going on here?

Thanks,
Jamie