What is Deduplication?

What it means

A CRM accumulates duplicates the way a garage accumulates cables. The same person arrives as a lead form fill on Tuesday, as a business-card CSV row in March, as an Instagram DM in June, and as a reply to a cold email in September. Every one of those arrivals creates a record, and none of them announce that they are the same human. Deduplication is the machinery that notices.

The academic name for this problem is record linkage or entity resolution, and the reason it is genuinely hard is that there is no universal key. If every person carried a stable identifier that every channel reported, deduplication would be a database constraint and this page would not exist. Instead you get partial, noisy, contradictory evidence: an email here, a phone number there, a display name that is a nickname, a company field somebody typed in ALL CAPS.

A working deduplication system runs in five stages. Normalise the data so that comparisons mean something. Generate candidate pairs cheaply, because comparing everything with everything does not scale. Score each candidate pair with weighted rules. Route the score to auto-merge, review or ignore. Then apply survivorship rules field by field so that the merge does not destroy information. Each stage below is the version we actually run.

Stage 1: normalisation

Comparison is only meaningful between values in the same shape. The normalisation pass rewrites every match-relevant field into a canonical form and stores it alongside the original. The original is never overwritten, because the operator typed it and it is evidence.

• Case and whitespace. Lowercase, trim, collapse internal runs of whitespace to one space.
• Diacritics. Transliterate to a base alphabet so that Şükrü Öztürk and Sukru Ozturk compare equal. This matters enormously in Turkish, German and Nordic datasets, where the same person is entered both ways depending on the keyboard in front of the operator.
• Phone numbers. Parse to E.164 using the workspace default country for numbers written without a country code. 0532 111 22 33 and +905321112233 are the same number and must normalise to the same string.
• Email. Lowercase the whole address. Then apply provider-scoped rules and only provider-scoped rules: Gmail treats dots in the local part as insignificant and ignores everything after a plus sign, so ali.veli+crm@gmail.com and aliveli@gmail.com reach the same mailbox. At most other providers a dot is a significant character and stripping it invents a different address. Apply the Gmail rule to Gmail and to Google Workspace domains you have confirmed, nowhere else.
• Company name. Strip legal suffixes such as Ltd, Ltd. Sti., A.S., GmbH, Inc, LLC and BV, strip punctuation, then compare. Acme Inc. and ACME are the same firm.
• Website. Reduce to a registrable domain: drop the scheme, drop www, drop the path and query. https://www.acme.com/contact?ref=x becomes acme.com.

Stage 2: blocking, or why this is not quadratic

Comparing every record with every other record is O(n²). At 50,000 contacts that is roughly 1.25 billion pairs, which is not a job you run nightly. Blocking is the standard escape: you partition records into buckets using cheap deterministic keys, and only score pairs that share a bucket. Records land in several buckets at once, and that redundancy is the point, because any one key can be missing or wrong.

The blocking keys we generate per record:

• The exact normalised email.
• The email domain plus the normalised surname.
• The last nine digits of the E.164 phone, which survives country-code entry errors.
• A double-metaphone code of the surname plus the first initial. Metaphone codes group names that sound alike across spellings, which is what catches Schmidt against Schmitt and Yılmaz against Yilmas.
• The platform plus the platform user id, for social identities.

A record with an email, a phone and a Telegram id therefore sits in five or six blocks. Recall is high because a pair only has to share one block to be considered, and cost stays linear-ish because the blocks are small.

Stage 3: weighted scoring

Each candidate pair is scored by summing the rules in the table below. Positive signals argue that the pair is one person, negative signals argue that it is two. The weights are ours, tuned against our own review-queue outcomes, and you should expect to retune them against your data rather than adopting them as physics.

The name comparison uses Jaro-Winkler, which is the right family of string distance for personal names because it weights matching prefixes more heavily than matching suffixes. People truncate and abbreviate the ends of names far more often than the beginnings.

Stage 4: thresholds and a worked pair

House defaults: 90 and above merges automatically, 60 to 89 enters a human review queue, and below 60 is left alone. Note what the 90 line implies: nothing reaches auto-merge on fuzzy evidence. Only an exact normalised email or an exact platform user id clears it on its own. That is deliberate, for reasons in the next section.

Work an actual pair. Record A and record B:

• A: Mehmet Yılmaz, m.yilmaz@acme.com, phone +905321112233 (confirmed by a WhatsApp delivery receipt), company Acme Ltd. Sti., city Istanbul.
• B: Mehmet Yilmaz, mehmet.yilmaz@acme.com, phone +905339998877 (confirmed by an SMS reply), company ACME, city Istanbul.

Scoring, out loud:

• Emails are different strings, so the +100 rule does not fire. Neither record carries a social id, so the platform rule does not fire either.
• Same email domain acme.com, and Jaro-Winkler on "mehmet yilmaz" against "mehmet yilmaz" after diacritic transliteration is 1.0, comfortably above the 0.92 threshold: +60.
• Identical normalised full name and same city: +25.
• Same normalised company (both reduce to acme) and same first name: +15.
• Two different confirmed phone numbers: -25.

Total: 60 + 25 + 15 - 25 = 75. That lands in the review band, which is exactly right, because a human can see what the scorer cannot. The reviewer is shown both records side by side with the differing fields highlighted, the last five interactions from each, and the rules that fired with their point values. Two plausible resolutions: this is one person who changed mobile operator and kept an old alias address, or these are a father and son at a family firm. The conversation history usually settles it in about four seconds, and that is a far better use of four seconds than unpicking a bad merge later.

Stage 5: survivorship

Merging is not choosing a winning row and deleting the other. It is choosing a winner per field. The rules we apply are in the second table below. Two of them deserve emphasis.

Never drop a channel identity. A Telegram user id or a WhatsApp number is not decoration, it is the only routable address for that channel. A merge that keeps one email and discards the other has silently made a conversation unreachable.

Most restrictive consent wins. If either record opted out of email, the survivor is opted out of email. Merging should never resurrect a suppressed contact, and a merge that quietly does so is a compliance incident, not a data-quality improvement. See double opt-in for how consent gets recorded in the first place.

Every merge writes an audit row containing the losing record's full JSON snapshot, the surviving id, the rules that fired, the score, the actor, and the ids of every child row that was repointed. That snapshot is what makes an unmerge possible. Without it, reversing a merge is reconstruction from memory.

Why the error costs are asymmetric

A missed duplicate is an annoyance. Somebody gets two emails, a report double-counts a lead, an operator sees two cards and sighs. Irritating, cheap, fixable later.

A wrong merge is a different category of event. You have just attached one person's message history, notes, deal value and phone number to another person's record. The next operator to open that contact reads a stranger's conversation. If your inbox replies in-thread, you may send that stranger's context back out to the wrong recipient. In a regulated context it is a personal-data breach, and it happened because a scorer was 78 percent confident and nobody was asked.

The asymmetry is the entire argument for a conservative auto-merge threshold and a review queue that a human actually works. Optimising for a clean-looking contact count is optimising the wrong number.

Prevention beats cleanup

Everything above is remedial. The cheaper work happens at write time.

• A unique index on (workspace, normalised email) turns duplicate creation into a constraint violation you handle, rather than a row you clean up in six months.
• Upsert on import. CSV import should match on the normalised key and update, not blindly insert. Show the operator the match count before the import runs, not after.
• Dedupe inbound webhooks. Every integration that creates contacts is a duplicate factory unless it resolves against the existing set first.
• Normalise at the form. Trimming and lowercasing an email in the browser prevents a class of duplicate that no amount of later matching fully recovers.

The cross-channel identity problem

This is the limit case, and it deserves an honest statement rather than a marketing one. A Telegram user id, an Instagram-scoped id, a WhatsApp phone number and an email address share no key whatsoever. None of them can be derived from another. No amount of scoring bridges that gap, because there is no evidence to score.

The join happens only when the person supplies it. They reply to an email from an address you already hold. They type their email into a bot flow. They book with the phone number that is already on the record. They authenticate. Every one of those is an event you should capture and store as a verified link, with its source and timestamp, because it is the only high-confidence bridge you will ever get.

What you must not do is bridge on display names. @mehmet_y on Telegram and mehmet.yilmaz@acme.com looks like a match to a human and is a coin flip in reality. Scoring that pair above the review threshold produces confident, invisible, wrong merges, which is the worst failure mode this system has.

Related concepts

• Contact enrichment: fills the fields that make matching possible, and creates duplicates of its own if it writes without resolving first.
• Unified inbox: the feature that breaks most visibly when deduplication is wrong, because threads split or fuse.
• Deliverability: duplicates raise complaint rate and corrupt suppression lists.
• Merge variable: a badly merged record is where the empty first name that ruins a send comes from.
• Omnichannel CRM: one contact record across channels is the promise deduplication has to deliver.
• Cold outreach: sending the same opener twice to one human is the most common visible symptom of a duplicate problem.

How Pinlyx handles it

Pinlyx normalises email, phone and name on write, keeps the original values intact, and enforces a unique index on the workspace plus normalised email. CSV import resolves against existing contacts and shows the match count before anything is written. The nightly pass blocks on email, phone tail, metaphone surname and platform id, scores the candidate pairs with the weighted rules above, merges automatically only at 90 and above, and files everything from 60 to 89 into a review queue that shows both records side by side with the firing rules and their point values. Merges are field-level, keep every channel identity, take the most restrictive consent state, and write an audit row with a full snapshot of the losing record so the merge can be reversed.