Disk, memory and swap on Linux

Two failures account for most of the "it was fine yesterday" tickets on a Linux server: the disk filled up, or something ran out of memory and was killed. Both take under a minute to confirm once you know which command answers which question, and both look like an unrelated application bug until you check.

This page covers reading the disk and memory tools, tracking down what filled a disk, clearing space safely, recognising the out-of-memory killer, and adding swap without making a game server worse.

How full is the disk

df -h

-h prints sizes in units people read rather than blocks. Each row is a filesystem, and the columns that matter are Avail and Use% against the Mounted on path.

Most of the rows are noise. tmpfs and devtmpfs entries live in memory, not on your disk, and /dev/shm and /run being small or busy is normal. The row to look at is the one mounted on /, plus any extra volume you attached deliberately. To skip straight to it:

df -h /
df -hT /var

-T adds the filesystem type, which is worth having when you are not sure whether a path is on its own volume or part of the root filesystem. Passing a path shows the filesystem that path lives on, so df -h /var/log answers "is the volume holding my logs full" without you working out the mount layout.

A full root filesystem does not usually announce itself. Services fail to write and then fail to start, databases stop accepting writes, logging stops, and package installs abort part-way. If several unrelated things break at once, check df -h before anything else.

Space free, but still "No space left on device"

If df -h shows room and writes still fail, the filesystem has run out of inodes, which are the records that hold file metadata. One per file, allocated when the filesystem was created, and a directory holding millions of tiny session or cache files can exhaust them while using almost no space.

df -i

Read IUse% the same way you read Use%. If it is at 100, the fix is deleting files, not freeing bytes, and the next section finds them. Counting files per directory is how you spot the culprit:

find /var/lib -xdev -type f | wc -l

What filled it

du measures directories rather than filesystems. Start at the top and walk down the biggest branch, rather than trying to guess:

du -xh --max-depth=1 / | sort -h

• -x stays on one filesystem, so an attached volume or a network mount does not get counted into the root total, and /proc does not waste your time.
• --max-depth=1 summarises each child directory instead of printing every file.
• sort -h puts the largest at the bottom, next to your prompt.

Then repeat inside whichever directory is largest:

du -xh --max-depth=1 /var | sort -h
du -xh --max-depth=1 /var/log | sort -h

Run it as root, or du silently skips directories it cannot read and the totals come out too small. On a big filesystem it takes a while, because it is reading the whole tree.

For a single answer rather than a walk:

du -sh /opt/fivem

If you would rather browse interactively, ncdu does the same walk with a navigable interface. It is in the standard repositories for both distribution families, see Installing and updating software.

Space held by deleted files

There is one case where du and df disagree, and it confuses people for hours: df says the disk is full, du cannot find the space. That happens when a file has been deleted while a process still has it open. The directory entry is gone, so du cannot see it, but the space is not released until the last process using it closes it or exits. Deleting a large log file that a running service is still writing to is the classic way in.

lsof +L1

That lists open files with no remaining name. Install lsof first if it is not present. The fix is to restart the process holding the file, which releases the space immediately, see Managing services with systemd.

It is also the reason not to delete a live log file. Empty it in place instead, which keeps the file the process has open:

truncate -s 0 /var/log/big-service.log

Clearing space safely

In rough order of how much you get for how little risk.

The journal. systemd's log has a size cap, but the default can be large on a big filesystem:

journalctl --disk-usage
journalctl --vacuum-size=200M
journalctl --vacuum-time=14d

Vacuuming removes old entries and keeps logging working. To stop it growing back, set SystemMaxUse=200M in /etc/systemd/journald.conf and restart systemd-journald.

The package cache. Downloaded packages are kept after installation and are safe to remove.

On Debian and Ubuntu:

apt clean
apt autoremove --purge

On RHEL, AlmaLinux and Rocky:

dnf clean all
dnf autoremove

autoremove also drops dependencies nothing needs any more. Read the list it prints before agreeing, because a package you installed by hand can look unwanted to it.

Old kernels. Each kernel keeps its own files in /boot and its modules in /lib/modules, and a small /boot partition fills after a few updates. On Debian and Ubuntu, apt autoremove --purge handles it. On the RHEL family, dnf keeps a fixed number of kernels, set by installonly_limit in /etc/dnf/dnf.conf, and lowering it and running dnf autoremove clears the rest.

Application logs that are not rotating. Anything writing to /var/log that did not come from a package usually has no rotation configured. Add a file in /etc/logrotate.d/ for it rather than deleting by hand every few weeks. Game server logs and crash dumps are the usual offenders, and a directory of crash dumps is worth reading before it is worth deleting.

Memory, and why free is nearly zero

free -h

The column people misread is buff/cache, so take the row in order:

• total is the RAM in the machine.
• used is memory held by processes.
• free is memory doing nothing at all.
• buff/cache is memory holding cached file contents and filesystem metadata.
• available is the estimate that matters: roughly how much a new process could get without the system having to swap.

Linux uses spare RAM as a disk cache, on the reasoning that unused memory is wasted memory. Reading a file leaves a copy in cache so the next read does not touch the disk. That cache is counted in buff/cache and is given up instantly when a process needs the space.

So free sitting near zero on a server that has been up for a week is normal and healthy, not a problem. It means the cache is doing its job. The number to watch is available: while that is comfortable, you have memory. When available approaches zero, you are genuinely short.

For the same reason, dropping caches is not a maintenance task. Writing to /proc/sys/vm/drop_caches is a benchmarking tool. On a live server it frees memory the kernel would have released anyway and makes the next reads slower.

What is using it

ps aux --sort=-%mem | head

That is the top consumers by resident memory. top and htop show the same thing live, and M sorts by memory inside top.

Per service, systemd tracks it directly:

systemd-cgtop
systemctl status fivem

The status output includes a Memory: line for the whole service, its child processes included, which is more useful than reading one process from ps when a game server forks workers.

The OOM killer

When the kernel cannot free enough memory to satisfy an allocation, it does not fail the allocation, it picks a process and kills it. That is the out-of-memory killer. It generally chooses the process using the most memory, which on your server is usually the exact thing you were trying to keep running.

The symptom is distinctive: a process disappears with nothing in its own log. No stack trace, no shutdown message, no error, because it was killed outright and got no chance to write anything. If your game server vanishes cleanly and its log ends mid-sentence, check for this before you go looking for a bug.

It is recorded in the kernel log, so the journal has it:

journalctl -k --since "2 hours ago" | grep -i -e "out of memory" -e "killed process"
dmesg -T | grep -i oom

dmesg -T prints human-readable timestamps rather than seconds since boot. The entry names the process that was killed and its memory use at the time.

systemd records it too, and systemctl status for the unit shows the main process exiting on a KILL signal rather than with an exit code. If the unit has Restart=on-failure or Restart=always, it comes straight back, which is why this can run for weeks as a mysterious restart every few days rather than as an outage. See Managing services with systemd.

The real fixes are, in order: use less memory, cap the service so it fails predictably instead of taking the machine down with it, or get more RAM. The middle one is a unit setting, MemoryMax=, which confines the service to its own limit so the kernel kills inside that service rather than choosing whatever looked biggest on the machine.

Swap

Swap is disk space the kernel uses to hold memory pages it has pushed out of RAM. It is not more memory. It is slower memory, by a factor large enough that the difference is qualitative rather than a percentage.

Where it helps: a server whose memory use is mostly cold, or one that briefly peaks above its RAM. Pages that are allocated and then never touched, which most long-running programs have plenty of, sit in swap harmlessly and leave more RAM for cache and for the pages that are actually in use. A small amount of swap can absorb a spike that would otherwise trigger an OOM kill.

Watch whether it is being used, rather than whether it exists. vmstat prints swap-in and swap-out rates in its si and so columns:

swapon --show
vmstat 1 10

Swap sitting occupied with si and so at zero is fine, those are cold pages that were moved out once. Continuous non-zero si and so while your server is busy is the pattern to act on.

Adding a swap file

A file works as well as a partition and can be removed again, which a partition cannot. Size it as a margin over your peak, not as a substitute for RAM, and check you have the disk space to spare with df -h first.

fallocate -l 2G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
swapon --show

chmod 600 is not optional. The file contains whatever was in memory, including credentials, and it must not be readable by anyone else.

If fallocate is not usable on your filesystem, write the file out instead, then carry on from chmod:

dd if=/dev/zero of=/swapfile bs=1M count=2048 status=progress

To keep it after a reboot, add this line to /etc/fstab:

/swapfile none swap sw 0 0

Test the fstab entry before you rely on it, because a bad line there can stop the server booting:

swapoff /swapfile
swapon -a
swapon --show

Making the kernel reluctant to use it

vm.swappiness controls how readily the kernel swaps pages out rather than shrinking the file cache. Lower means it prefers to keep process memory in RAM, which is what you want when latency matters more than cache hit rate:

sysctl vm.swappiness=10

Make it survive a reboot by writing it to a file in /etc/sysctl.d/, for example /etc/sysctl.d/99-swappiness.conf, containing vm.swappiness = 10.

Setting it to 0 does not disable swap, it makes the kernel avoid swapping until it is nearly out of memory, at which point an OOM kill is close anyway. If you want no swap at all, remove the swap file rather than setting swappiness to zero.

Removing a swap file

swapoff /swapfile
rm /swapfile

Delete the /etc/fstab line as well, or the next boot will try to enable a file that is not there.

When none of this explains it

If the disk keeps filling and you cannot find the writer, or memory use grows without any process accounting for it, note what you have already checked and open a ticket at Opening a ticket. Include the output of df -h, free -h, and the relevant journal lines, which saves a round trip.

Related