Arch Linux - A stop job is running for MariaDB database server
The message in question as shown below made its appearance while I was trying to shut down my laptop. The job went on for about a minute before completing its task.
Thinking Arch had finally thrown med a curveball, I booted up the machine to have a look at the systemd journal.
# View logs by service: journalctl -u mysqld.service # Output - mysqld.service: State 'stop-sigterm' timed out. Killing. - mysqld.service: Main process exited, code=killed ... - mysqld: 150718 12:20:54 [Warning] mysqld: Disk is full writing './mysql-bin.~rec~' (Errcode: 28 "No space left on device")....
So yeah, the “Disk is full” warning did not require me to be sysadmin of the year to decipher. By looking at the file system I could confirm that / was using 100% of available space. To find out where all the space had gone, I issued the command:
du -Pshx /* 2>/dev/null # Output .. 20K /tmp 5.8G /usr 13G /var
13G under /var doesn’t seem right. That can’t be all logs…
Let’s see whats taking up most of the space:
du -Sh /var | sort -rh | head -n 5 # Output 11G /var/cache/pacman/pkg 2.0G /var/log/journal/43a7d67b84eb4c05bc77f4ab75096597 174M /var/tmp/kdecache-master 112M /var/lib/mysql 58M /var/tmp/kdecache-master/http
To quote the pacman section of the Arch wiki:
pacman stores its downloaded packages in /var/cache/pacman/pkg/ and does not remove the old or uninstalled versions automatically, therefore it is necessary to deliberately clean up that folder periodically to prevent such folder to grow indefinitely in size.
In short, I had unknowingly kept a years worth of package updates on disk, which in turn had accumulated to 11G and eventually hit the roof.
Following the book by using the paccache command (instead of deleting everything) freed up some much needed space.
I was actually aware of this “feature” but had forgotten all about it. I wont be making the same mistake again as filling up / will eventually cause serious harm to the system.