Open file descriptor limits and Marathon

When using Marathon for a large number of tasks, we can run into issues with the number of file descriptors used either by Marathon, or by other processes in the system that are competing with Marathon for resources.

Symptoms:

You may see either "Permission Denied" errors on file writes, or in some cases JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError".

Marathon will be killed, but it may not be able to write out a dump file: 

JVMDUMP030W Cannot write dump to file //core.########.######.####.####.dmp: Permission denied

Diagnosis:

Note: Commands based on CentOS. May vary depending on your distro, shell, etc. Using the command "cat /proc/sys/fs/file-max", inspect your current ulimit. Then, using "ulimit -u", you'll be able to see your current utilization.

Resolution:

To increase the limit, append the following config directive at the end of /etc/sysctl.conf file:

fs.file-max = 100000

Once set, use sysctl -p to reload changes.

Have more questions? Submit a request

Comments

Powered by Zendesk