How to stop logs from filling your server's disk on Ubuntu or Debian

Before you start

• You need a user with sudo rights. See how to make one.
• The examples were tested on Ubuntu 24.04 and Debian 13. The site is called shop and lives in /var/www/shop. Change the names to yours.

1. See how full the disk is

df -h

Look at the line for / and the Use% column. Lines for tmpfs are in memory, not on disk, so you can skip them. On Ubuntu you may also see /snap/... lines at 100%. That's normal: snaps are read-only images and are always "full".

A disk can also run out of files before it runs out of space, for example with millions of small session or cache files. Check with:

df -i /

If IUse% is near 100%, look for a folder with a huge number of small files, not for big ones.

2. Find what uses the space

Show the biggest folders, largest last. -x keeps it on one disk:

sudo du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -n 10

Then go one level deeper into the biggest one, for example /var and then /var/log:

sudo du -h --max-depth=1 /var/log | sort -h | tail -n 10

Or list every file over 500 MB:

sudo find / -xdev -type f -size +500M -exec ls -lh {} + 2>/dev/null

Easier: ncdu shows the same thing as a list you can move through with the arrow keys. Press Enter to open a folder and q to quit:

sudo apt install ncdu
sudo ncdu -x /

The usual suspects: /var/log/journal (step 3), app logs such as Laravel's storage/logs and website logs (step 4), old backups, and the package cache in /var/cache/apt (sudo apt clean empties it).

3. Put a limit on the system journal

Ubuntu and Debian keep system logs in the journal. See how much space it takes:

sudo journalctl --disk-usage

By default it may use up to 10% of the disk, at most 4 GB. To set your own limit, make a small file of your own. Don't edit /etc/systemd/journald.conf itself, so package updates never overwrite your change:

sudo mkdir -p /etc/systemd/journald.conf.d
sudo nano /etc/systemd/journald.conf.d/size.conf

Put this in it:

[Journal]
SystemMaxUse=500M

Restart the journal and check that it picked up the new limit:

sudo systemctl restart systemd-journald
sudo journalctl -b -u systemd-journald | tail -n 1

The last line should say something like max 500.0M. The journal now removes its oldest entries on its own when it reaches the limit.

Need space right now? Shrink the journal at once. This deletes the oldest entries until it's under the size you give:

sudo journalctl --vacuum-size=200M

Or keep only recent entries with sudo journalctl --vacuum-time=2weeks. Both only remove older, closed journal files, so the result can be a bit above the number you asked for.

4. Rotate your app and website logs

logrotate is already installed and runs once a day. It renames a log to .1, starts a fresh one, packs older ones with gzip and deletes the oldest. System logs and the standard Nginx and Apache logs in /var/log/nginx/ and /var/log/apache2/ are already covered. Logs anywhere else need a rule of their own in /etc/logrotate.d/.

A Laravel app

Laravel writes to storage/logs/laravel.log, which grows forever (see step 7 of our Laravel guide). First check who owns the folder:

ls -ld /var/www/shop/storage/logs

Then make /etc/logrotate.d/shop. Put the owner and group you just saw on the su line:

/var/www/shop/storage/logs/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    su shop shop
    copytruncate
}

What each line does:

• daily and rotate 14: start a new log every day and keep 14 old ones. Use weekly for quiet sites.
• compress and delaycompress: pack old logs with gzip, but leave yesterday's unpacked, so it's easy to read.
• missingok and notifempty: no error when there's no log, and skip logs that are empty.
• su shop shop: work as the user and group that own the folder. Without it, logrotate refuses to touch a folder that someone other than root can write to. You'll see skipping ... because parent directory has insecure permissions.
• copytruncate: copy the log, then empty the original. See the next part.

copytruncate or create?

• create renames the log and makes a new empty one, for example create 0664 shop shop. It's the clean way, but only for programs that open the log again: either for every request, or when they get a signal. Programs that keep the file open keep writing into the renamed .1 file.
• copytruncate copies the log and then empties it, so the program can keep writing to the same file. Lines written during the copy can be lost, but it works with any program.

For Laravel, use copytruncate: queue workers run for a long time and keep the log open.

Using Laravel's daily log instead? With LOG_CHANNEL=daily in .env, Laravel makes one file per day and deletes old ones itself (14 days, or LOG_DAILY_DAYS). Then you don't need this rule, and you shouldn't use both.

A site's Nginx logs in their own folder

If a site writes its logs to, say, /var/log/nginx/shop/, the standard rule doesn't see them, because it only covers files right inside /var/log/nginx/. Make /etc/logrotate.d/nginx-shop:

/var/log/nginx/shop/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        invoke-rc.d nginx rotate >/dev/null 2>&1
    endscript
}

This is the same as Ubuntu's and Debian's own Nginx rule. Nginx keeps its logs open, so the postrotate lines tell it to open the new files. With sharedscripts, that happens once, not once per file.

Test the rule

A dry run shows what logrotate would do, without changing anything:

sudo logrotate -d /etc/logrotate.d/shop

Look for error: lines. The first time, it may say log does not need rotating. That's normal: logrotate has just seen this log for the first time and will rotate it on its next daily run. To rotate right now and see the result:

sudo logrotate -f /etc/logrotate.d/shop
ls -l /var/www/shop/storage/logs/

You should see an empty laravel.log and a laravel.log.1. After the next rotation, the older one becomes laravel.log.2.gz. For the Nginx rule, open the site afterwards and check that the new request lands in the new, empty access.log.

The daily run is a systemd timer. See when it runs next with systemctl list-timers logrotate.timer.

5. Empty a huge log safely

If one log is already huge and the disk is nearly full, empty it in place:

sudo truncate -s 0 /var/www/shop/storage/logs/laravel.log

The program keeps writing to the same file and the space is free at once. If you still need the newest lines, save them first:

sudo tail -n 5000 /var/www/shop/storage/logs/laravel.log | sudo tee /root/laravel-last.log >/dev/null

6. Space that doesn't come back: deleted files still in use

You deleted a big log, but df -h shows the same number. A program still has the file open, so Linux can't free the space yet. List those files:

sudo apt install lsof
sudo lsof +L1

Each line ends in (deleted). COMMAND and PID tell you which program holds it, SIZE/OFF how big it is. Restart that service and the space comes back, for example:

sudo systemctl restart php8.3-fpm

If you can't restart it right now, empty the file through the program's open handle instead. The number in the FD column (3w means 3) and the PID give the path:

sudo truncate -s 0 /proc/1234/fd/3

Replace 1234 and 3 with your numbers. Then fix the cause, usually a missing logrotate rule or one that uses create for a program that never reopens its log (step 4).

7. Get a warning before the disk is full

A small script can check the disks every 15 minutes and print a line when one is too full. Save it as /usr/local/bin/disk-alert:

#!/bin/sh
# Print a warning for every disk that is fuller than LIMIT percent.
LIMIT=85
df -P -x tmpfs -x devtmpfs -x squashfs | tail -n +2 | while read -r dev size used avail pct mount; do
  pct=${pct%\%}
  if [ "$pct" -ge "$LIMIT" ]; then
    echo "$(hostname): $mount is $pct% full"
  fi
done

Make it runnable and try it. It prints nothing while all disks are under 85%:

sudo chmod 755 /usr/local/bin/disk-alert
/usr/local/bin/disk-alert

To see what a warning looks like, run it once with a limit of 1%:

sed 's/^LIMIT=85/LIMIT=1/' /usr/local/bin/disk-alert | sh

Then run it from cron. Make /etc/cron.d/disk-alert:

MAILTO=you@example.com
*/15 * * * * root /usr/local/bin/disk-alert

Cron emails anything the script prints to MAILTO, but only if the server can send email. Check with sudo journalctl -u cron after a warning: No MTA installed, discarding output means the email never left. Then set up mail, or change the echo line so it sends the text to a chat or alert service you already use, for example with curl.

This is a simple start. It won't tell you if the server itself is down, so an outside check is still worth having.

If something goes wrong

• skipping ... because parent directory has insecure permissions: add the su line with the folder's owner and group (step 4).
• Old logs keep growing after rotation: the program still writes into .1. Use copytruncate, or add a postrotate that makes it reopen its log.
• The disk is full again a day later: something writes a lot very fast. Look at the newest lines of the biggest log to see why. Often it's the same error, again and again, or a debug setting left on, such as APP_DEBUG=true or LOG_LEVEL=debug.
• Logs never rotate: check that the timer runs with systemctl list-timers logrotate.timer, and see its last run with sudo journalctl -u logrotate.

Official sources

• logrotate manual (Ubuntu 24.04)
• journald.conf manual (systemd) and journalctl manual
• lsof manual (Ubuntu 24.04)
• Rotating log files (nginx.org)
• Logging (Laravel docs)
• ncdu

Next steps: work through the security checklist for a new PHP server, and make sure your backups don't fill the same disk.

ServerNest is being built to show each server's disk use and email you when a disk is almost full. The product is still in development. Join early access to try it first.

← Back to the security blog