High CPU usage - due to scheduled tasks

dev support
1

Hi,

What is the best way to find out why the Lucee CPU usage is so high? 50% with Lucee 5.3.7.47 and 75% with Lucee 5.3.8.139-RC. I did a fresh restart of Lucee and all scheduled tasks are on pause. Nobody have access to the website on this server, so no activity at all and still, the CPU is high and stay high.

OS: Windows Server 2016 (10.0) 64bit
Java Version: 1.8.0_181 (Oracle Corporation) 64bit
Tomcat Version: Apache Tomcat/8.5.33
Lucee Version: 5.3.7.47 and 5.3.8.139-RC

Lucee 5.3.7.47 Overview

overview-5.3.7.47
overview-5.3.7.47758Ă—445 28.1 KB

5.3.8.139-RC Overview

overview-5.3.8.139-RC
overview-5.3.8.139-RC767Ă—475 24.4 KB

Thank you!

2

try renaming any \WEB-INF\lucee\scheduler.xml file(s), to completely rule out scheduled tasks

also my Performance Analyzer plugin for Lucee will show you which threads are running

1 Like
3

I removed the scheduler.xml file. There is no scheduled tasks showing in the web admin and the CPU is still at 75% on Lucee 5.3.8.139-RC.

I’ll try your plugin.

4

did you restart lucee?

5

Yes. And I didn’t visit my website and boom 75% CPU already.

I see the threads now with your plugin. Something particular to look at? . I see a lot of threads with status TIMED_WAITING and the stack mention ScheduledTaskThread.

6

Oh! Now I used the Lucee-Tomcat Service Control to restart, instead of the restart Lucee button in the admin, and the CPU is at 0%, and go at 7% when I visit a page, then 0%. Sound good! No more threads about ScheduledTaskThread.

7

Use FusionReactor and it will show you which threads in the JVM are consuming the most CPU. Simply click on “Resources” > “Thread Visualizer” then click the “CPU(ms)” column to sort and you’ll have your answer right away.

1 Like
8

Good idea for FusionReactor. I’ll take a look at that.

So I suppose my problem was provoked by my scheduled tasks. I’ll take them back and restart the same way I did to see if the CPU stay low. I know I worked on my scheduled tasks few days ago and I had problem with the characters encoding. I did a lot of try/fail and I reported a bug about it. So maybe something got stuck in the memory and the restart button in the admin don’t do as much as restarting Lucee from Lucee-Tomcat Service Control.

9

There’s some certain combinations of scheduled task intervals which get stuck in an infinite loop trying to find the next schedule :frowning:

1 Like
10

As soon as I put back my scheduled tasks and restart Lucee with Lucee-Tomcat Service Control, CPU at 50% again. Many threads about ScheduledTaskThread. I’ll try to add my scheduled tasks one by one to see which combination provoke that. Note that all scheduled tasks are on pause…

There’s some certain combinations of scheduled task intervals which get stuck in an infinite loop trying to find the next schedule

Is this bug has been already reported?

I wonder if I should use something else than Lucee to manage the scheduled tasks…

11

This is sort of a side-question, but the fact that the Lucee restart button didn’t stop those scheduled task threads is probably a bug in itself. The Lucee service restart actually shuts down the JVM and starts a fresh one. The Lucee restart button leaves the JVM and the servlet container (tomcat) and simply unloads and reloads the Lucee context in memory. IMO, that should include any scheduled task threads.

Is this bug has been already reported?

Yes. That’s how Zac knows about it :slight_smile:

12

https://luceeserver.atlassian.net/browse/LDEV-3214

1 Like
13

Thank you @Zackster and @bdw429s

I’ll wait for a patch and if the patch don’t come in time, I will track down the scheduled task that provoke it or use another tool to manage the scheduled tasks instead of Lucee.

14

If you could isolate which task(s) are causing the problem before it get addressed, that would be super useful…

15

I’ll do that with pleasure then!

16

Cool.

The admin restart bug may have been fixed in 5.3.8 btw

1 Like
17

I found one of the scheduled task. This one alone provoke a 25% CPU usage non-stop. When I try a different interval like 1000 it’s okay.

<?xml version="1.0" encoding="ISO-8859-1" standalone="no"?><schedule><task autoDelete="false" endTime="{t '23:59:59'}" hidden="false" interval="14400" name="To Infinity and Beyond" paused="true" port="443" proxyHost="" proxyPassword="" proxyPort="0" proxyUser="" publish="false" readonly="false" resolveUrl="false" startDate="{d '2021-01-01'}" startTime="{t '00:00:00'}" timeout="90000" unique="true" url="https://www.example.com/infinity/"/></schedule>
18

Use the profile feature in FR to profile the thread for a few seconds and then see what it is doing. The previous issues with scheduled tasks was due to an infinite loop int he code trying to figure out when to run the task next.

19

I just added in thread CPU time to my Performance Analyzer plugin v1.0.1.2 (available via admin)

image
image2252Ă—920 324 KB

2 Likes
20

Hi! I installed the new version of your plugin @Zackster.

As soon I add the scheduled task To Infinity and Beyond and restart Lucee, I see this :

thread
thread1198Ă—147 5.18 KB

After few seconds I see CPU time jumping at 286,281.25, then 316,234.38, then 332,750.00, and going up and up and up…

1 Like