"unable to create new native thread" after high-volume runAsync()

dev support
1

OS: MacOS Catalina
Java Version: 1.8.0_172 (Oracle Corporation) 64bit
Tomcat Version: Not sure - it’s CommandBox.
Lucee Version: Lucee 5.3.7.47

I just ran into a funny little issue when playing around with runAsync(). If I run this code a few times (maybe 4 or 5 times, maybe 15 times), my server runs out of threads:

<cfscript>

	counter = createObject( "java", "java.util.concurrent.atomic.AtomicInteger" ).init();
	values = [].set( 1, 1000, "" );

	timer
		type = "outline"
		label = "Using Run Async Iteration"
		{

		futures = values.map(
			() => {

				return runAsync(
					() => {

						counter.incrementAndGet();

					}
				);

			}
		);

		futures.each( ( value ) => value.get() );

		echo( "Counter: #counter.get()#" );

	}

</cfscript>

After I run that page a few times, the Lucee server stops being able to create new threads and all page requests end with this:

Error (java.lang.OutOfMemoryError)
lucee.runtime.exp.NativeException: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:681)
at lucee.runtime.future.Future._then(Future.java:56)
at lucee.runtime.functions.thread.RunAsync.call(RunAsync.java:27)
at run_async_speed630.async_cfm$cf.udfCall(/run-async-speed/async.cfm:19)
at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:106)
at lucee.runtime.type.UDFImpl._call(UDFImpl.java:344)
at lucee.runtime.type.UDFImpl.call(UDFImpl.java:212)
at lucee.runtime.type.EnvUDF.call(EnvUDF.java:109)
at lucee.runtime.functions.closure.Map._inv(Map.java:312)

I was trying to compare speed of parallel iteration (.each( op, true, 20 )) to the speed of runAsync() to see if there was any difference. Ironically, parallel iteration doesn’t seem to have any issues at all; but, this code with runAsync() consistently drains the server of all threads.

2

Ben, can you try the new 5.3.8. RC?

There are quite a few optimisations relating to these type of problems

3

@bennadel just tried this with the 5.3.8.139-RC and I could run it fine without any problems

ps: your java version is almost old enough to drink!

4

Yes, I’ll see if I can figure out how to pull down the RC with CommandBox. I’m sure it’s just some arg for the start command. I’ll report back.

5

Is that what you’re looking for?

1 Like
6

@bennadel Yes, the cfengine param is the one. You can start any version of Lucee so long as it’s been pulled into ForgeBox (happens daily). The best part is, you can just use tab completion to find what you want. For example, if you type

server start cfengine=lucee@5.3.8

and hit tab, you’d see all the possible snapshots.
In this case, you can also just run this:

server start cfengine=lucee@5.3.8-RC
7

So cool!! I knew I could use cfengine= for lucee vs adobe; but, I’m never sure which versions are available.

8

Another way you can see all the possible versions other than tab complete, is just to go to the Lucee entry on ForgeBox and peruse the versions tab.

There is a search feature built in that allows you to filter versions based on semantic version ranges.

1 Like
9

No luck with the RC, still getting the native thread exhaustion error:

2021-01-27_06-50-18
2021-01-27_06-50-181220×981 171 KB

10

@bennadel I forget where, but I someone was talking about this same error the other day. If you have FusionReactor, can you take a look at Resrources > Threads and see how many threads are running on the JVM. Then go to Resources > Thread Activity and back up the chart to see what the trend of thread counts looks like.

This is just a guess, but it’s possible the single executor thread pools that Lucee uses for runasync aren’t being shutdown and de-allocated.

11

Actually, I just realized you had a code sample at the top. I tossed that in a 5.3.7 server I had running and hit refresh about 20 times. I wasn’t able to get an error, but I can see in FR that there are 1000 threads created for each request, and a portion of them say in the ‘waiting’ state for a few seconds before they are all removed. (The yellow parts of the graph) The code you have creates a pretty wasteful creation of threads I think, and depending on your memory spaces and JVM config, I could see a scenario in which calling that too often would create more threads than the JVM could handle.

Obviously, your code here is just for the sake of example, but I’d never use a pattern like that in real life. The cbstreams library for instance, allows you to loop async over a collection of any size, but with a fixed thread pool. Which is more less what the arrayEach in parallel will do as well.

image
image1171×538 68 KB

I’d be interested in seeing what this graph looks like on your machine leading up to the error you reported. And I’ll re-iterate-- the Lucee runAsync() BIF creates a NEW single executor thread pool for every execution! I don’t think it’s architected very well for any sort of regular usage IMO.

12

I was just curious to see if there was any substantive performance difference between using parallel iteration and runAsync() … and I figured the easiest way to test would be to create a scenario in which there was likely more requests than threads, to see which ones can drain and complete first.

To your point, I would never actually do this in “real life” :laughing:

1 Like
13

bug filed [LDEV-4397] - Lucee

1 Like