Duplicate Document Detection for UK Mortgage Brokers

Duplicate Document Detection for UK Mortgage Brokers

Duplicate document detection means identifying when the same document has reached a case file more than once, whether as an identical file or as a rescan, photo or fresh export of the same page. For a mortgage broker, the reliable approach combines a file-level check for identical copies with a content-level check on document type, issuer, period and key amounts. Flagged copies should be kept, reviewed by a named person and logged, never deleted silently.

This guide explains what counts as a duplicate, the main detection methods, where each one fails, and a five-step process a brokerage can run on every case.

TL;DR

  • An exact duplicate has identical bytes. A near-duplicate says the same thing but was rescanned, photographed or re-exported, so its bytes differ.
  • File hashing catches exact copies. Comparing issuer, period, account and amounts catches the rest.
  • A December and a January statement from the same account look alike but are different documents. Field comparison handles this; image similarity does not.
  • Keep flagged duplicates, exclude them from the count of received documents, and have a named reviewer confirm each flag.
  • Record every decision so the file shows what was received, what was set aside and why.

Table of Contents

What counts as a duplicate document

Two categories matter in practice.

An exact duplicate is a second file whose bytes are identical to the first. One accounting-firm guide defines it this way and notes that a file hash reliably detects exact-byte duplicates: identical hash values imply identical contents.

A near-duplicate carries the same information in a different file. A technical note on near duplication lists rescans, PDFs saved at different times and email threads as items that can look identical to a reader but differ under computer analysis. In a mortgage file, typical examples are:

  • a payslip sent as a PDF and again as a phone photo;
  • a bank statement downloaded twice from online banking on different days;
  • a passport page scanned at two resolutions;
  • the same utility bill attached to two different applicants on a joint application.

A third category is easy to overlook: similar but distinct documents. Consecutive monthly statements share a layout, an account number and often a similar balance pattern. They are not duplicates, and a detection method that treats them as such creates its own delays.

Why duplicates build up in mortgage case files

Duplicates rarely signal bad intent. The same accounting-firm guide says they arise when clients resend a document, send it in two formats, or when the firm itself creates a copy while handling the file. Brokerages add their own causes:

  • Chasing emails. A client who is reminded about a missing payslip often resends everything.
  • Joint applications. Two applicants upload the same household bill or the same joint account statement.
  • Multiple channels. Some documents arrive by email, some through a portal and some by messaging app.
  • Manual re-saving. An adviser renames or re-exports a file, producing a copy with different bytes.

The cost is practical. Duplicates inflate the document count, hide gaps (a case can look complete because it has twelve files when only nine are distinct), and give underwriters extra material to question. If you are tackling the cause upstream, our guide to automated document collection for UK mortgage brokers covers how structured requests reduce repeat uploads.

Five methods for detecting duplicate documents

Most systems layer these methods, from cheapest and strictest to most flexible.

1. File hash comparison

A hash is a fixed-length fingerprint calculated from a file's bytes. If two files produce the same hash, they are the same file. One document-management vendor explains that its duplicate detection compares file content only, using MD5 checksums, and that metadata such as the file name can differ between matched copies.

This is fast and almost free to run. It is also blind to any change at all: a rescan, a re-export or a screenshot defeats it.

2. Hash of extracted text

Because OCR (optical character recognition) turns a page image into text, you can hash the text instead of the file. An open AWS sample pipeline for contract checks uses a SHA-256 hash of extracted text for exact matching. This survives some format changes, since a PDF and a clean re-export can yield the same text, but OCR noise on a poor scan will still change the hash.

3. Normalised text similarity

A more tolerant approach normalises the text first and then scores how alike two pages are. An open-source tool for document libraries describes a pipeline that lowercases text, strips punctuation and collapses whitespace, finds candidate pairs with MinHash and locality-sensitive hashing, then scores them with Jaccard similarity and fuzzy matching. The point for brokers is the principle: normalise, shortlist, then score, rather than comparing every page with every other page.

4. Perceptual image hashing

Perceptual hashes describe how a page looks rather than what its bytes are. The AWS sample mentioned above uses a perceptual hash per page image, compared by Hamming distance, with a threshold of 10 bits or fewer, roughly 85% similarity, to flag near-duplicates. This catches rescans of the same page well. It is weaker on documents that share a template, which brings us to the method that matters most for lending.

5. Field-based comparison

For financial documents, the most dependable test asks whether the key details match: document type, issuer, period, account or reference, and key amounts. The accounting-firm guide recommends exactly this for rescans and phone photos, and points out that a December statement and a January statement from the same account are different documents, which is where image similarity goes wrong and field comparison does not.

For a brokerage, a sensible set of fields is:

DocumentFields that define "the same document"
Bank statementBank, account number, statement period, opening and closing balance
PayslipEmployer, employee, pay date, gross and net pay
P60 or SA302Tax year, employer or UTR, total pay or income figure
Utility billSupplier, address, bill date, account reference
Photo IDDocument type, number, expiry date

A statement with the same account and period but a different closing balance is not a duplicate. It is a discrepancy, and it deserves a different kind of attention (see the section on tampering below).

Where detection goes wrong

Every method has failure modes. Knowing them lets you design review around them.

Missed duplicates (false negatives)

  • Hashes can differ even when content is the same. The M-Files guide warns that some formats store extra metadata that changes the MD5 value.
  • Poor scans produce OCR errors, so text and field comparisons miss the match. A code walkthrough on duplicate image detection notes that low resolution and noise cause false negatives and suggests preprocessing such as contrast enhancement, cropping and deskewing.
  • Concurrent uploads can slip through. Several systems check new files against what is already stored, so two identical files arriving at the same moment may not see each other.

False matches (false positives)

  • Consecutive statements and payslips share templates, so image similarity over-flags them.
  • Misread fields can make two different documents look alike.
  • Joint applicants can legitimately hold the same document, such as a joint account statement, which should be linked to both rather than discarded.

The practical response is a confidence-based review. Exact hash matches can be treated as high confidence. Near-matches go to a person with the reason shown, for example "same account, same period, different file".

What duplicates can reveal about a case

Duplicate detection and tampering detection overlap. If two files share an issuer, account and period but show different balances or deposits, the cause may be an innocent re-download after a correction, or an edited file. Either way, the lender will notice before you do unless the check happens first. Our article on document tampering detection for UK mortgage brokers explains what to look for in that case.

It also helps with gaps. A duplicated page can mask a missing one, so a statement that looks complete by page count may not be. See missing bank statement pages and what brokers should do for how to spot and resolve this.

Identity and address evidence has its own pattern. The same proof of address submitted for two applicants, or an expired ID resubmitted under a new file name, is common. The guides on identity document verification and proof of address verification cover those checks in more detail.

What duplicates can reveal about a case

A five-step process for handling suspected duplicates

Step 1: Check every new batch on arrival

Run comparisons each time a document arrives, not once at the end of the case. Late checks mean advisers review a pile instead of a single new item, and they let duplicates inflate the "received" count in the meantime.

Step 2: Run exact and content checks in order

Start with the file hash. Then compare extracted fields against documents already on the case: type, issuer, period, reference and key amounts. Only then apply any similarity scoring. This order gives the cheapest and most certain answers first.

Step 3: Flag the newer copy and keep it

Treat the earlier document as the original and flag the later one. One expense-management help page describes treating the older document as the original, flagging the newer one and deleting nothing automatically. Retention matters in a regulated setting: the copy may show what the client actually sent and when.

Step 4: Have a named reviewer confirm

Assign each flag to a person who confirms "duplicate", "different document" or "discrepancy". The accounting-firm guide advises a named reviewer for every flag. Confirmed duplicates are excluded from the received count and kept visible. Discrepancies go to the adviser for a client conversation.

Step 5: Record the decision

Store the outcome as data on the case: which document is the original, which is the copy, the reason and the reviewer. A guide on ingestion pipelines recommends recording dedupe decisions as metadata rather than silently discarding documents. In a brokerage this becomes part of the audit trail. The article on document approval workflow controls auditors can trace sets out what that trail should contain.

Tracking duplicates alongside document status

The most useful place to show duplicate status is the same checklist that shows what is received, missing, rejected or approved. A flagged duplicate then stays visible without counting towards the documents you expect. It also lets you answer a simple question at any moment: is this case complete in distinct documents, or only in files?

Worth measuring over time:

  • the share of uploads flagged as duplicates, by document type;
  • the most common reasons (client resend, joint applicant, format change);
  • the time between flag and review.

A high duplicate rate on one document type usually points to an unclear request, such as asking for "recent statements" without a period. Fixing the request lowers the rate more than any amount of detection tuning.

Perspective: why this belongs inside document collection

We see duplicate detection as one control among several, not a feature to bolt on. A broker does not need to be told that two files share a hash. A broker needs to know whether the case is complete, whether anything conflicts, and whether a lender will come back with a query.

That is why we build our checks around the document set as a whole. Casefile cross-checks documents such as bank statements, payslips, tax forms, ID and utility bills, and highlights discrepancies, including undeclared commitments and mismatched wage deposits, before they reach the lender. Duplicates, rescans and altered files all sit on that same line of review.

Our position is that collection, checking and approval should happen in one place, under the brokerage's own branding, with UK-only data residency and a clear audit trail. Small brokerages in particular cannot spend adviser hours comparing file names. Automation should do the comparison and leave the judgement to the adviser.

Darren

How Casefile supports document checking

Casefile is a cloud-based platform for UK mortgage brokers that automates document collection, cross-checks documents and flags discrepancies before they become lender queries. Document lists build themselves from the client's payment types, approval queues keep review organised, and your branding stays on the client-facing experience. Data stays in the UK, and you can read how in how Casefile handles your documents.

Gocasefile

To see how this fits your case flow, visit Casefile.

Reducing duplicate uploads at the source

Detection treats the symptom. These habits reduce the cause:

  1. Request by period. Ask for "statements for the three months to [date]" rather than "recent statements".
  2. Show what is received. When clients can see a document was accepted, they stop resending it.
  3. Use one channel. Route uploads through a single branded portal instead of email and messaging apps.
  4. Tailor the list. A self-employed applicant and a salaried applicant need different documents. Lists built from payment type avoid generic requests that prompt extras.
  5. Link joint documents. Allow one document to be attached to two applicants explicitly, so nobody uploads it twice.
  6. Give clear rejection reasons. "Page 3 missing" gets one new upload. "Not accepted" gets five.

Anti-money-laundering evidence adds its own sensitivity, because retained copies are part of your records. The AML document checks guide explains how to keep that evidence organised.

FAQ

What is the difference between a duplicate and a near-duplicate document?

A duplicate is a file with identical bytes to another, so a hash comparison finds it. A near-duplicate carries the same content in a different file, such as a rescan, photo or re-export. Finding it needs a comparison of text, fields or page images.

Should duplicate documents be deleted?

No. Flag the newer copy, keep it, and exclude it from the count of received documents. Keeping it preserves a record of what the client sent and supports your audit trail. A named reviewer should confirm each flag.

Can software confuse two consecutive bank statements for duplicates?

Image-based methods can, because the layout is the same. Field-based comparison avoids this by checking account, period and balances, so a December and a January statement are treated as different documents.

Can a poor-quality scan cause missed or false duplicate flags?

Yes. Low resolution and noise cause OCR to misread characters, which can hide a real match or create a false one. Asking clients for clearer uploads, and applying preprocessing such as deskewing and contrast adjustment, reduces both problems.

Is a duplicate a sign of fraud?

Usually not. Most duplicates come from clients resending documents after a chase. The exception is a pair that shares issuer, account and period but shows different figures, which should go to the adviser for review as a discrepancy.

Sources

This article was generated with AI, and whilst it is our expectation that the article is accurate, it should not be taken as advice.