When running Mesos for a long time or under frequent development work, the filesystem will become exhausted and refuse to make new files, with the error:
No space left on device
Common causes for this include:
- The filesystem has no available disk space. (determined using df -H)
To resolve this cause, remove files until there is enough space to restart Mesos-slave
- The /tmp folder has no available disk space (df -H /tmp)
By default, Mesos uses /tmp/mesos as the work_dir. To resolve this issue, either clear space in the /tmp directory, or set a different work_dir as described at the Mesos configuration page.
- The filesystem has no available inodes (df -i)
A filesystem can only hold a certain number of file references (inodes) before it runs out of capacity. As it's generally not possible to increase the number of inodes available in the filesystem, the resolution is to delete files. Slaves with a very active relationship with Docker can collect a large number of file chunks from image pulls in /var/lib/docker/tmp, for instance.