noxz.tech/articles/your_disk_is_full_and_empty/index.www
1I recently had a Schrödinger's cat experience with the root drive of my NAS.
2Suddenly, I wasn't able to apply any system updates to the system while getting
3errors indicating my root drive was full, which was strange as a 32G drive
4should be more than sufficient for storing the system. When checking the file
5allocation using
6.ICD "du -cksh --exclude /strg --exclude /fbkp /"
7I could see that, yes, the system was only allocating about 1.7G:
8
9.CDS
10.COS
111.7G /
121.7G total
13.COE
14.CDE
15
16To confirm that my disk indeed had unallocated space, I also checked the
17allocation using
18.ICD "df"
19and got a conflicting result:
20
21.CDS
22.COS
23Filesystem 1K-blocks Used Available Use% Mounted on
24/dev/root 30386396 29719476 0 100% /
25\[u2026]
26.COE
27.CDE
28
29What was going on?
30
31.PIMG cat.png
32
33The next thing I tried was to delete file caches and logs from various places,
34and even after that,
35.ICD "df"
36still showed 100% full. At this point, I was very confused and tried
37many useless things before realizing what was going on. When checking the file
38allocation using
39.ICD "du" ,
40I excluded two file paths, one for the main NAS storage partition and one for
41backing up the SD card of the Raspberry Pi, as they were mounted on different
42disks and partitions than the root one. After checking the log, I could see
43that before the backup of the SD card, the disk used for storing the backup
44had, for unknown reasons, been unmounted. After restarts, that same disk had
45been mounted again, effectively hiding the problem.
46
47After unmounting that disk and listing the contents of the directory acting as
48the mount point, I saw what was allocating the disk space.
49
50.CDS
51.COS
52# umount /fbkp
53# ls -l /fbkp
54total 28031472
55-rw-r--r-- 1 root root 28704194560 Aug 11 06:52 sdcard.img
56.COE
57.CDE
58
59The way I was backing up the SD card had a major flaw considering that the
60destination of the backup was on a mounted disk, or rather a memory stick. I
61hadn't taken into account that the memory stick could either fail or, for some
62other reason, not be mounted while the backup was taking place. My solution was
63to first make sure that the disk was in fact mounted before performing the
64backup. I changed the cron job from this
65
66.CDS
67.COS
68\[u2026]
69# create backup of sdcard
700 6 * * * dd if=/dev/mmcblk0 of=/fbkp/sdcard.img
71\[u2026]
72.COE
73.CDE
74
75to this
76
77.CDS
78.COS
79\[u2026]
80# create backup of sdcard
810 6 * * * mount | grep '/dev/sd.1 on /fbkp' && dd if=/dev/mmcblk0 of=/fbkp/sdcard.img
82\[u2026]
83.COE
84.CDE
85
86Since then, the cron job has done what it was supposed to do. I was once again
87reminded of how a simple solution to a simple, but seemingly illogical problem,
88can seem to be impossible to find from an initial look.
