Known Issue - An accumulation of threads and the memory they allocate can result in OOM

May 09, 2025 14:21
Updated

Symptoms

After an upgrade to 2025.1.0.0 monitoring of resources shows an accumulation of threads in the Timed_Waiting state and memory consumption growing ever upwards. If left unchecked the system will eventually exhaust all available memory.

Cause

When System_Mail_Send with an attachment from a file stored on an platform Agent, then the thread reading the file from the process server remains in the Timed_Waiting state.

Steps to reproduce:

Create a chain
Step 1: Run an ABAP program that produces a spool file.
Step 2: Use System_Mail_Send to email the spool file to a recipient.
There will be a thread created with the reported stack.

Workaround

No workaround is available. Monitor the system and restart before memory is exhausted.

Affected version

RunMyJobs 2025.1.0.0

Resolution

This issue has been fixed in 2025.1.0.4

Reference

RCORE-50297

Error

Threaddump will show following stack

"Thread-120" prio=9 tid=0x00000000000001fd cpu=4917890621 ns usr=2210000000 ns blocked 117906 for -1 ms waited 117913 for -1 ms
java.lang.Thread.State: TIMED_WAITING
at java.base@17.0.8/java.lang.Object.wait(Native Method)
- waiting to lock <0x0000000044ad3b02> (a java.io.PipedInputStream)
at java.base@17.0.8/java.io.PipedInputStream.awaitSpace(PipedInputStream.java:273)
at java.base@17.0.8/java.io.PipedInputStream.receive(PipedInputStream.java:231)
at java.base@17.0.8/java.io.PipedOutputStream.write(PipedOutputStream.java:150)
at java.base@17.0.8/java.io.OutputStream.write(OutputStream.java:127)
at com.redwood.scheduler.jobservice.remote.file.PlatformAgentServiceFile.lambda$getJobFileContentsViaMessage$0(PlatformAgentServiceFile.java:242)
at com.redwood.scheduler.jobservice.remote.file.PlatformAgentServiceFile$$Lambda$2147/0x00007fa1cb3073f8.run(Unknown Source)