How to deduplicate rows in a CSV file with Python

If every row in your CSV is a genuine full duplicate, Python's standard library is enough — no pandas install needed. Here's a script that works on any file, keeping the first occurrence of each row and dropping the rest:

import csv, sys

def dedupe(path_in, path_out):
    seen = set()
    with open(path_in, newline="", encoding="utf-8") as f_in, \
         open(path_out, "w", newline="", encoding="utf-8") as f_out:
        reader = csv.reader(f_in)
        writer = csv.writer(f_out)
        for row in reader:
            key = tuple(row)
            if key not in seen:
                seen.add(key)
                writer.writerow(row)

if __name__ == "__main__":
    dedupe(sys.argv[1], sys.argv[2])

Run it with python3 dedupe.py input.csv output.csv.

When "the whole row" isn't the right definition of duplicate

Most real exports aren't that clean. Two rows are "the same customer" even though the phone number is formatted differently, or the email has different casing, or one row has a typo in the company name. To dedupe by a single column instead of the whole row, swap the key:

key = row[2].strip().lower()  # e.g. column index 2 is the email

That covers a lot of cases. It stops covering them the moment you need more than one rule at once — normalize a date format and dedupe by email and map five inconsistent header names to the ones your next tool expects. At that point you're not writing a five-line script anymore, you're writing and testing a small program, and that's the part people usually pay someone else to do.

← See the other two packages (website scraping, API/webhook bridges)