How to find duplicate files on a Mac, free

Duplicates accumulate in predictable ways. You download the same attachment three times. A photo import runs twice. A folder gets copied “just in case” before a reorganization that never happened. None of it is dramatic, and after a few years it is tens of gigabytes.

The problem is not finding candidates. It is being certain that two files are genuinely identical before deleting one.

Why name and size are not enough

Two files called invoice.pdf, both 240 KB, are probably the same file. Probably. They might also be two different invoices from the same template, which is exactly the case where deleting the wrong one matters.

Conversely, the same photo exported twice at different quality settings has different sizes and is arguably a duplicate you would want to know about, but no checksum will tell you so.

The only way to be certain two files are byte-for-byte identical is to hash their contents. Everything else is a guess.

The one-line version

macOS ships with shasum, which reads a file and produces a fingerprint. Identical contents give an identical fingerprint; any difference at all gives a completely different one.

To find duplicates in a folder and everything under it:

find ~/Downloads -type f -exec shasum {} + | sort \
  | awk '{ if ($1 == prev) { if (hold != "") { print hold; hold="" } print } else hold=$0; prev=$1 }'

Reading that from the left: find every file, hash each one, sort so identical hashes sit together, then print only the lines whose hash appears more than once.

The awk line is doing the last step by hand for a reason. Most guides write it as uniq -w40 -D, which compares just the first 40 characters — but -w is a GNU extension and macOS ships the BSD uniq, which has no -w flag at all. That command fails with uniq: invalid option -- w on every Mac. The awk above holds each line back until it sees whether the next one shares its hash, which needs no special flags and works everywhere.

The output is groups of files sharing a hash. Every group is a set of genuinely identical files, and you can delete all but one of each with confidence.

For a whole home folder, add a size filter first so you are not hashing a hundred thousand tiny files:

find ~ -type f -size +10M -exec shasum {} + | sort \
  | awk '{ if ($1 == prev) { if (hold != "") { print hold; hold="" } print } else hold=$0; prev=$1 }'

That covers where the space actually is. Duplicated small files are numerous and almost never worth the effort.

Reading the output

You get something like:

d3486ae9136e7856bc42212385ea797094475802  /Users/you/Downloads/report.pdf
d3486ae9136e7856bc42212385ea797094475802  /Users/you/Documents/Archive/report.pdf

Same hash, two paths, so one can go. Which one is a judgement call, and the rule I use is: keep the one in the folder where you would look for it, delete the one in Downloads.

Do not do this to your Photos library

Photos keeps its images inside a package at ~/Pictures/Photos Library.photoslibrary, along with a database describing them. Deleting files inside that package by hand corrupts the library, and the checksum method will absolutely find “duplicates” in there because Photos deliberately stores thumbnails and edited versions alongside originals.

Photos has its own duplicate detection, and it is good. In recent macOS versions, look for the Duplicates album in the sidebar. It finds near-duplicates as well as exact ones, and merging through that interface keeps the database consistent.

Same principle for Music, and for anything else that stores its content in a package.

The other kind: duplicate downloads

The fastest win is usually not duplicate detection at all. It is looking at what has been downloaded repeatedly:

ls -lhS ~/Downloads | head -30

Sorted by size, largest first. Installers, disk images and the same PDF downloaded four times with (1), (2), (3) appended. That naming pattern is worth a search of its own:

find ~ -name "* (1).*" -o -name "* copy.*" 2>/dev/null

Those are duplicates that macOS itself created, and they are named as such.

Making it faster on a large disk

Hashing reads every byte of every file, so pointing it at a full home folder is slow. The trick is to hash only files that could possibly be duplicates — ones sharing a size with something else.

find ~ -type f -size +1M -exec stat -f '%z %N' {} + 2>/dev/null | sort -n \
  | awk '{ n=$1; $1=""; line=substr($0,2); if (n == prev) { if (hold != "") { print hold; hold="" } print line } else hold=line; prev=n }'

stat -f '%z %N' prints each file’s size and path. Sorting numerically puts equal sizes together, and the awk prints only the ones that share a size with a neighbour. That is a candidate list, not an answer — two different files can be exactly the same size — but it is usually a small fraction of the disk.

Now hash only those:

find ~ -type f -size +1M -exec stat -f '%z %N' {} + 2>/dev/null | sort -n \
  | awk '{ n=$1; $1=""; line=substr($0,2); if (n == prev) { if (hold != "") { print hold; hold="" } print line } else hold=line; prev=n }' \
  | tr '\n' '\0' | xargs -0 shasum | sort \
  | awk '{ if ($1 == prev) { if (hold != "") { print hold; hold="" } print } else hold=$0; prev=$1 }'

The size pass throws away everything that cannot possibly be a duplicate; the hash pass proves the rest. On a home folder that is minutes instead of an hour, and the answer is identical.

Near-duplicates, which checksums cannot see

A checksum is binary: identical or not. It has nothing to say about the cases people most often mean by “duplicate”.

The same photo exported at two quality settings. A song ripped once as ALAC and once as AAC. A document saved as both .pages and .pdf. A screenshot and a cropped version of it. All of these are duplicates in the sense that you only need one, and none of them will ever share a hash.

There is no terminal one-liner for this, because deciding two visually similar images are the same image is a judgement, not a comparison. Photos does it well within its own library. For everything else it is manual, and it is usually not worth the hours.

Deleting without regret

Once you have your groups, resist deleting from the terminal directly. A mistyped path in a find -delete is unrecoverable, and there is no undo.

The safer sequence is to move candidates to a staging folder first:

mkdir -p ~/Desktop/dupes-review
mv "/path/to/the/copy" ~/Desktop/dupes-review/

Use the Mac for a week. If nothing has complained, empty the folder. This matters more than it sounds: duplicates are frequently duplicates because something depends on the second copy being exactly where it is.

The one rule that has never let me down: when two copies of the same file exist, keep the one in the folder you would think to look in, and delete the one in Downloads.

Where duplicates rank

Worth being honest about the size of the prize. On a Mac several years old, genuine byte-identical duplicates usually come to somewhere between 2 and 10 GB. That is real, and it is also smaller than local Time Machine snapshots, old iOS backups and forgotten large files, all of which are easier to clear.

If you are here because the disk is full, do those first and come back to this. The ordered version of that list puts everything in sequence.

When it is worth automating

The shasum approach is exact and free, and it is fine for a folder. Across a full disk it is slow, because it reads every byte of every file, and the output needs interpretation.

That is the trade a dedicated tool makes: CleanSpace hashes in the background, groups the results, and shows you which copy is where so you can pick without reading terminal output. The answer it reaches is the same one the command above reaches — it is the same technique — so if you only need to clear one folder, save yourself the download.