Your disk is full and empty

Chris Noxz

August 14, 2022

I recently had a Schrödinger’s cat experience with the root drive of my NAS. Suddenly, I wasn’t able to apply any system updates to the system while getting errors indicating my root drive was full, which was strange as a 32G drive should be more than sufficient for storing the system. When checking the file allocation using du -cksh --exclude /strg --exclude /fbkp / I could see that, yes, the system was only allocating about 1.7G:

1.7G    / 
1.7G    total 

To confirm that my disk indeed had unallocated space, I also checked the allocation using df and got a conflicting result:

Filesystem     1K-blocks     Used Available Use% Mounted on 
/dev/root       30386396 29719476         0 100% / 
… 

What was going on?

The next thing I tried was to delete file caches and logs from various places, and even after that, df still showed 100% full. At this point, I was very confused and tried many useless things before realizing what was going on. When checking the file allocation using du, I excluded two file paths, one for the main NAS storage partition and one for backing up the SD card of the Raspberry Pi, as they were mounted on different disks and partitions than the root one. After checking the log, I could see that before the backup of the SD card, the disk used for storing the backup had, for unknown reasons, been unmounted. After restarts, that same disk had been mounted again, effectively hiding the problem.

After unmounting that disk and listing the contents of the directory acting as the mount point, I saw what was allocating the disk space.

# umount /fbkp 
# ls -l /fbkp 
total 28031472 
-rw-r--r-- 1 root root 28704194560 Aug 11 06:52 sdcard.img 

The way I was backing up the SD card had a major flaw considering that the destination of the backup was on a mounted disk, or rather a memory stick. I hadn’t taken into account that the memory stick could either fail or, for some other reason, not be mounted while the backup was taking place. My solution was to first make sure that the disk was in fact mounted before performing the backup. I changed the cron job from this

… 
# create backup of sdcard 
0 6 * * * dd if=/dev/mmcblk0 of=/fbkp/sdcard.img 
… 

to this

… 
# create backup of sdcard 
0 6 * * * mount | grep '/dev/sd.1 on /fbkp' && dd if=/dev/mmcblk0 of=/fbkp/sdcard.img 
… 

Since then, the cron job has done what it was supposed to do. I was once again reminded of how a simple solution to a simple, but seemingly illogical problem, can seem to be impossible to find from an initial look.