Deduplication that does not flinch at scale

What it does

• Remove duplicates by emailFree

  Case-insensitive, so [email protected] and [email protected] collapse into one. Rows with a blank email are kept, never merged together.

• Remove duplicates by IDFree

  For exports where the record key lives in an id column.

• Remove duplicates by email + phoneStarter

  Both fields must match before a row is dropped. Safer for partial data.

• Remove duplicates by any fieldStarter

  Pick the column yourself — sku, customer_id, whatever your export uses. If the column is not in the file, the job stops and tells you which columns it did find.

• Keep only active recordsStarter

  Filters on the status column and drops everything else.

• Remove empty recordsStarter

  Drops rows where every single field is blank.

How it behaves

Blank values are never merged

A row with no value in the deduplication field is kept as-is. Grouping every blank into one record would silently delete real data.

Your types survive

A code like 007 comes back as 007, not 7. Columns are read as text so nothing is reinterpreted on the way through.

Deduplication is global

Even when the file is split into blocks for speed, a duplicate is caught across the whole file, not just within its block.

You get the numbers

Every job reports rows analysed, duplicates removed, rows kept and the reduction percentage — so you can check the result, not just trust it.