noxz.tech/articles/deduplication_of_my_nas/index.www
1After developing
2.URL //noxz.tech/software/acst acst ,
3I realized that the generated and stored checksum could be used for duplicate
4detection on my NAS. The detection could also be done fairly quickly. The
5result is, of course, dependent on checksums being created or corrected fairly
6recently as no checksums are being computed during the duplicate check.
7
8When testing this (keep in mind that my NAS is a Raspberry PI with disks
9connected over USB), I was able to detect duplicates in 100,000 files in 1.5
10seconds. That's fast enough for me.
11
12I implemented the feature into
13.ICD acst
14with the added argument
15.ICD -d ,
16using the
17.URL https://en.wikipedia.org/wiki/Merge_sort "merge sort"
18algorithm as a means for detecting duplicates.
19
20I also realized that having a feature for recursively traversing through a file
21tree can be considered an anti-feature when
22.ICD find
23could be used instead with faster results. So, I'm considering removing this
24feature in favor of a smaller code base.
