"No space left on device" is the ENOSPC error: a write failed because the filesystem has no room. It surfaces everywhere — package installs, log writes, database commits, Docker builds:
write error: No space left on device
OSError: [Errno 28] No space left on device
ERROR: failed to solve: write /var/lib/docker/…: no space left on device
Before you delete anything: diagnose first. Deleting the wrong file on a production server causes far more damage than a full disk. Every command in the diagnosis section below is read-only and safe to run.
Step 1: Find the Full Filesystem
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 40G 40G 0 100% /
/dev/sdb1 200G 12G 188G 6% /data
/ is full while /data is nearly empty — so nothing on /data is relevant. Note the mount point, because that is the only place worth looking.
Step 2: Check Inodes — Free Space Can Be a Lie
If df -h shows space available and you still get ENOSPC, you are out of inodes:
df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 2621440 2621440 0 100% /
IUse% at 100% with Use% low means inode exhaustion. Every file consumes exactly one inode regardless of size, and the total is fixed when the filesystem is created. Millions of tiny files do it: PHP session files, mail queues, cache fragments, unrotated per-request logs.
Find the directories holding the most files:
sudo find / -xdev -type f -printf '%h\n' 2>/dev/null | sort | uniq -c | sort -rn | head -20
Common offenders are /var/lib/php/sessions, /var/spool/postfix, and application cache directories. The only remedies are deleting files or recreating the filesystem with a higher inode count — you cannot add inodes to a live ext4 filesystem.
Step 3: Check for Deleted-but-Open Files
This is the case that confuses people most: someone deleted a huge log, df still shows the disk full, and nothing seems to help.
sudo lsof +L1
COMMAND PID USER FD TYPE SIZE/OFF NLINK NODE NAME
nginx 1234 root 5w REG 8589934592 0 12345 /var/log/nginx/access.log (deleted)
NLINK 0 means the directory entry is gone but a process still holds the file open — and the blocks stay allocated until that handle closes. Unlinking a file does not free space; closing it does.
Fix by restarting the holding process, or better, truncate in place next time:
sudo truncate -s 0 /var/log/nginx/access.log
Truncating frees the blocks immediately and keeps the file handle valid, so the writing process carries on without needing a restart. This is the correct way to empty a live log file.
Step 4: Find the Large Directories
sudo du -x -h --max-depth=1 / | sort -h | tail -20
-x keeps du on a single filesystem, so it will not wander into /proc, network mounts, or that 200 GB /data volume you already ruled out. Descend into the biggest result and repeat:
sudo du -x -h --max-depth=1 /var | sort -h | tail -10
sudo du -x -h --max-depth=1 /var/log | sort -h | tail -10
To find individual large files:
sudo find / -xdev -type f -size +500M -exec ls -lh {} \; 2>/dev/null
Reclaiming Space Safely
Work down this list in order — it runs from "regenerates itself automatically" to "think carefully first."
Safest: package manager caches
Nothing unique lives here; everything can be re-downloaded.
# Debian / Ubuntu
sudo apt-get clean # empties /var/cache/apt/archives
sudo apt-get autoremove # removes orphaned dependencies
# RHEL / Rocky / AlmaLinux / Fedora
sudo dnf clean all
Safe: systemd journal
Journal logs commonly grow to several gigabytes on servers that have been up a long time.
journalctl --disk-usage
sudo journalctl --vacuum-size=200M # keep the newest 200 MB
sudo journalctl --vacuum-time=7d # or keep the last 7 days
Make it permanent so it does not recur, in /etc/systemd/journald.conf:
[Journal]
SystemMaxUse=500M
sudo systemctl restart systemd-journald
See Where Are systemd journald Logs Stored? for how the journal is laid out on disk.
Usually safe: rotated logs
sudo du -sh /var/log/*
sudo find /var/log -name "*.gz" -mtime +30 -delete
Do not rm -rf /var/log/*. Deleting an active log file that a process holds open frees nothing (see Step 3), and removing the directories a service expects can stop it logging entirely — or stop it starting. Use truncate on active files and delete only rotated archives.
If logs are what filled the disk, the real fix is rotation. The Where Are Nginx Logs Stored? and Where Are Apache Logs Stored? articles cover where each service writes and how it rotates.
Care required: Docker
Docker is very often the answer on a build server.
docker system df # look before you prune
docker system prune # stopped containers, unused networks,
# dangling images, build cache
Warning:
docker system prune -a --volumesis a much bigger hammer.-aremoves every image not currently used by a running container, and--volumesremoves unused volumes — which is where databases and persistent application data live. Rundocker volume lsand confirm what you would lose before adding--volumes.
Last resort: reserved blocks
ext4 reserves 5% of the filesystem for root, which is 20 GB on a 400 GB volume.
sudo tune2fs -m 1 /dev/sda1 # reduce reservation to 1%
Reasonable on a dedicated data volume. Keep a reserve on the root filesystem — that margin is what lets you log in and run these commands when everything else has stopped.
Verify
df -h /
df -i /
Both Use% and IUse% should have dropped. Restart anything that failed while the disk was full — services that hit ENOSPC frequently stay in a broken state until restarted, even after space is available:
systemctl --failed
Prevention
- Alert at 80–85%, not 95%. Databases stop accepting writes and package operations fail long before a disk is truly full.
- Cap the journal with
SystemMaxUse. It is the most common silent consumer on long-lived servers. - Verify logrotate is actually running:
sudo logrotate -d /etc/logrotate.confdoes a dry run and shows what it would rotate. - Monitor inodes too. Most dashboards graph bytes and not inodes, which is exactly why inode exhaustion arrives as a total surprise.
- Put Docker on its own volume on build machines, so a runaway image cache cannot take down the operating system with it.