Lessons · Python · deduplicating records
Deduplicating records
set(records) fails on dicts because they are unhashable. Deduplicate by building a key that identifies a record, a tuple of normalised fields, and keeping the first row per key.
Hone is a place to practise programming. This is one of its lessons, written out in full and free to read without an account.
What it is for
Two signups from the same person with different capitalisation, the same order imported twice: cleaning duplicates is the first step of almost every data job, and the key is the whole decision.
How to think about it
Decide what makes two rows the same, normalise those fields (strip, lower), make a tuple, and keep a set of tuples seen so far. Write the key down in words before writing it in code.
Worked example
rows = [{"email": "A@x.io"}, {"email": "a@x.io "}, {"email": "b@x.io"}]Three rows, two people.seen = set()Keys met so far.
unique = []What we keep.
for r in rows:One row at a time.
key = r["email"].strip().lower()What 'same person' means here.
if key not in seen:First time for this key?
seen.add(key)Remember it.
unique.append(r)Keep the first row.
print(len(unique))2: the first two rows were one person.
Your turn
A key that treats names case-insensitively.
key = (r["first"].lower(), r["last"].())
Solve one with the tests running
The trap
set(rows) on a list of dicts raises TypeError: unhashable type. It is telling you it does not know what 'same' means, which is the question to answer, not to work around.