Replace dupes with hard links or shortcuts #

CloneSpy: GUI dedupe utility that can (among other things) replace duplicate files with shortcuts or hard links.
Dupemerge: CLI utility that searches for identical files on a logical drive and creates hard links among those files. Includes a number of speed enhancements: only files with the same size are compared, hashsums for same-sized files get calculated incrementally, and more. (via Anthony Frazier)
Duplicate File Hard Linker (DFHL): Open source CLI tool that replaces duplicate files with hard links. Byte-per-byte comparison ensures only identical files are hard linked.
Finddupe: Open source CLI tool for detecting, deleting, or hard linking duplicate files. Can also list all hardlinked files on a given volume (similar to Microsoft's Hlscan.exe). Finddupe computes a hash based on the first 32k of a file, only checking entire file if there is a match.

This KB article explains hard links:

A hard link is a folder entry for a file regardless of where the actual file data exists on that volume. Every file is considered to have at least one hard link. On NTFS volumes, each file can have multiple hard links; therefore, a single file can be displayed in many folders (or even in the same folder with different names). Because all of the links reference the same file, programs can open any of the links and modify the file. A file is deleted from the file system only after all of the links to it have been deleted. After you create a hard link, programs can use the link like any other file name. Note that Windows Explorer and a command prompt will show all linked files as being the same size, even though they all share the same data and do not actually use that amount of disk space.

This last bit can be confusing. Let's say you've got a folder with three duplicate files of 10MB each. After replacing the dupes with hard links, the folder size is still reported as 30MB by Windows Explorer and cmd.exe. However, the "Used space" reported by Windows Explorer for the volume will have decreased by 20MB.

/windows | Feb 20, 2011

Subscribe or visit the archives.