When you hit OOM where host is not able to create any native thread while application is requesting it still, here are things you could check right away:
Check Default Max task set in the host : systemctl show --property=DefaultTasksMax
Check system wide thread setting at : cat /proc/sys/kernel/threads-max
Check number of processes and thread spawned by them : ps -elfT | wc -l
Check number of thread spawned by application that is effected : ps -p <applicationPID> -lfT | wc -l
Check your threads PID limit : systemctl -a | grep kernel.pid_max
Most importantly, since DefaultTasksMax is only set at system level for almost all distributions of linux and you are not willing to stretch this param to max possible value system wide, you can always set process/service specific task max by modifying TasksMax parameter on service configuration.
The TasksMax setting for any Unit/Service files can be configured as below:
Example: for sshd.service in /etc/systemd/system/sshd.service.d/override.conf file: [Service] TasksMax=20000
And you grep it using this:
systemctl status sshd.service | grep -e Tasks
// it is just an example for sshd service. You can edit <application>.service file to set that.