What is identity deduplication?

Identity deduplication is the process of detecting and consolidating records that, although they may appear to be different, belong to the same individual.

When an organisation manages thousands or millions of people, whether they are customers, patients, benefit recipients or application users, it is possible for the same individual to end up with more than one record in the system.

Identity deduplication identifies these duplicate records and determines whether they should be merged, flagged for review or prevented from creating a new record. In other words, it determines what should be done with them.

It is important not to confuse identity deduplication with data deduplication in the broader sense, a term that in computing is also used to refer to removing redundant copies of files or storage blocks. Here, the objective is not to save storage space, but to determine whether several records correspond to the same identity.

Why do identities become duplicated?

Duplicate records can appear because of data entry errors, changes to personal information, registrations made through different channels or deliberate attempts to create multiple identities.

The most common causes include:

• Data entry errors: a person enters their name differently, an employee makes a typo or a form does not validate the information correctly.
• Changes to personal information: someone changes their surname, address or telephone number and the system does not link the new record to the previous one.
• Registrations through different channels: the same person registers through an app, a website and a physical branch, with each channel creating a separate record.
• Deliberate creation of multiple identities: a person may attempt to create several accounts to obtain certain benefits or to conceal the relationship between different records.

This last situation is particularly difficult to resolve using text-based rules alone, precisely because the aim may be to prevent the data in the different records from matching.

What is the difference between an exact duplicate and a fuzzy duplicate?

An exact duplicate contains the same data, whereas a fuzzy duplicate refers to the same person even though their data appears differently.

This distinction matters because each type requires a different detection technique:

A real-world case that illustrates this difference is the National Data Repository in Nigeria, where fingerprints were used to detect potential duplicate records in the national HIV records system.

Between 2021 and 2024, more than 1.5 million records, of which 162,120 were identified as potentially duplicate, were analysed. The system also distinguished between perfect duplicates, where other personal details also matched, and imperfect duplicates that required additional review.

The case shows how biometrics can complement demographic information to detect when the same person appears multiple times in a database, even when their records are not identical.

How are duplicate identities detected?

Several detection layers can be combined, including exact matching, fuzzy matching, machine learning models and biometrics.

The appropriate technique depends on the volume of data, the quality of the information available and the level of accuracy required by the system.

Exact or deterministic matching

Exact matching compares specific fields, such as a national ID number, email address or telephone number, and looks for records containing identical values.

It is a fast and straightforward technique, but its limitation is that if a piece of data contains a typo, appears in a different format or has changed over time, the system may fail to detect that two records belong to the same person.

Fuzzy matching or probabilistic record linkage

Fuzzy matching calculates a similarity score between two records rather than requiring an exact match.

Some of the most commonly used algorithms include:

• Levenshtein distance: calculates how many editing operations, such as inserting, deleting or replacing characters, separate two text strings.
• Jaro-Winkler: gives greater weight to certain matches, particularly at the beginning of words, which can make it useful for names.
• Phonetic algorithms: such as Soundex or Metaphone, can identify words that may sound similar even when they are spelt differently.
• Fellegi-Sunter model: uses a probabilistic approach to weight different fields and determine the likelihood that two records belong to the same person.

In practice, these techniques can be combined to produce a more comprehensive similarity score.

Blocking or how to scale identity deduplication

Comparing every record with every existing record quickly becomes computationally expensive when a database contains millions of users.

Blocking reduces this problem by first creating groups of candidates that share certain characteristics, such as a postcode, part of a surname or a date of birth. A detailed comparison is then carried out only within those groups.

This significantly reduces the number of comparisons required without having to analyse every possible pair of records.

Machine learning models

Machine learning models can learn to identify patterns from examples that have previously been labelled as duplicates or non-duplicates.

Rather than manually determining how much weight each field should have, the model can learn which combinations of characteristics are most relevant when determining whether two records belong to the same person.

This approach can be used, for example, to prioritise cases that require human review or to progressively improve the identification of potential duplicates.

Biometric deduplication (1:N matching)

Biometric deduplication makes it possible to compare a person’s biometric characteristic against a database of identities to determine whether that person is already registered.

ABIS (Automated Biometric Identification Systems) have used this principle for years with fingerprints and other biometric characteristics. In a 1:N scenario, a biometric sample is compared against existing identities to answer a different question from traditional verification: does this person already exist in the system?

A 2024 study published on TechRxiv by Thoughtworks researchers specifically examines this approach using facial embeddings and similarity searches in a vector database. The study explores the use of ArcFace to generate biometric representations and Milvus to perform similarity searches at scale.

In this type of architecture, a facial image is transformed into a mathematical representation. Rather than comparing the images directly, the system compares these representations to determine which ones are sufficiently similar.

The study reports an accuracy of 99.79%, an F1 score of 89.44% and an FPIR and FNIR of 0.1% in its tests. These results correspond to that specific experiment and its evaluation conditions, so they should not be interpreted as universal figures for every deduplication system.

The authors also analyse how performance changes as the number of reference samples associated with the same identity increases. Their results show an improvement in accuracy when more reference images are used, highlighting the importance of the quality and quantity of biometric samples in these types of systems.

Clustering: finding networks of duplicates

When the objective is not to check a new record against historical data, but rather to analyse an entire database to identify duplicates that have not yet been detected, clustering techniques can also be used.

Instead of analysing records only in pairs, these techniques group similar records into clusters. This makes it possible to identify relationships between several records rather than only isolated duplicates.

What role does privacy play when biometrics are used for deduplication?

Biometrics can make it possible to identify duplicates with a high level of accuracy, but their use must be accompanied by mechanisms that ensure biometric information is protected throughout the entire process.

In biometric deduplication, a sample, such as a face or fingerprint, is compared against identities already stored in a database to determine whether that person is already registered. Therefore, in addition to matching accuracy, it is important to define how biometric data is stored, processed and protected.

A good example is Janus, a system developed by researchers from EPFL, CISPA and the International Committee of the Red Cross (ICRC) and presented at the IEEE Symposium on Security and Privacy in 2024. The project addresses the challenge of preventing the same person from registering multiple times to receive humanitarian assistance using biometrics, without exposing biometric databases during the deduplication process.

Janus can determine whether a person is already registered and returns only a binary result: whether or not a match exists, without revealing additional information about the identities stored in the database. To achieve this, the research explores different secure computing mechanisms, including secure multiparty computation (SMC), a hybrid approach based on homomorphic encryption and SMC, and trusted execution environments (TEEs).

The Janus case shows how current research is not focused solely on making biometric matching more accurate, but also on developing architectures that allow organisations to take advantage of this accuracy while keeping biometric information protected.

Privacy and biometrics are therefore not opposing concepts. The key is to design the deduplication process so that the necessary information can be used safely and in a controlled manner.

The hybrid approach used by Janus combines secure multiparty computation with homomorphic encryption, a technique that makes it possible to perform certain operations on encrypted data without first decrypting it. Applied to biometrics, this allows part of the comparison process to be carried out while keeping the information used protected.

In other words, the system can work with protected data and obtain the result of the operation without directly exposing the biometric information. The Janus research explores this approach alongside other secure computing architectures to find a balance between privacy and performance.

Which sectors need to detect duplicate identities?

Any organisation managing large volumes of identities may face the problem of duplicate records.

How do you measure whether an identity deduplication system is working well?

The appropriate metric depends on the type of deduplication. In 1:N biometric systems, two commonly used indicators are FPIR and FNIR.

• FPIR (False Positive Identification Rate): measures how often the system incorrectly identifies a person as a match.
• FNIR (False Negative Identification Rate): measures how often the system fails to identify a match that actually exists.

The decision threshold has a direct impact on both metrics. Changing it can reduce one type of error while increasing the other, so it should be established according to the characteristics and risks of the specific use case.

The TechRxiv study mentioned above obtained an FPIR and FNIR of 0.1% in its tests. As with any experimental result, these figures should be interpreted within the specific conditions of the study and should not be considered a guarantee applicable to every population or database.

In other record linkage or data deduplication scenarios, metrics such as precision, recall or F1 can also be used, particularly when working with non-biometric records.

Frequently asked questions about identity deduplication

DOSIER PRODUCTO

Descubre nuestra solución de verificación de identidad

Verifica la identidad de tus clientes en segundos a través del escaneo y validación de documentos de identidad y matching biométrico facial con prueba de vida.