# Four mistakes that ruin a dataset built from your documents

**Short answer.** The four mistakes are: leaving duplicate text, which the model
memorizes instead of generalizing; contaminating the evaluation set with
fragments of the training set; chunking documents before splitting them, which
leaks the same template into train and test; and leaving personal data in, which
the model can end up reciting. The third is the most dangerous because it throws
no error: it raises the metric and looks like learning.

---

All four are documented. All four have a fix. And all four show up almost every
time someone builds a first dataset out of their company's documents, because
public pipelines were designed for web corpora and office documents behave
differently: more template, more repetition, more personal data per page.

## 1. Duplicates: the model memorizes instead of learning

Lee et al., *Deduplicating Training Data Makes Language Models Better*
(ACL 2022, [arXiv:2107.06499](https://arxiv.org/abs/2107.06499)):

> «over 1% of the unprompted output of language models trained on these datasets
> is copied verbatim from the training data […] Deduplication allows us to train
> models that emit memorized text ten times less frequently and require fewer
> train steps to achieve the same or better accuracy.»

Two numbers for scale. In C4 they found **a 61-word sentence repeated more than
60,000 times**. CCNet ([arXiv:1911.00359](https://ar5iv.labs.arxiv.org/html/1911.00359))
measured duplicate paragraphs at **70% of the text** in its corpus.

And Carlini et al. ([arXiv:2202.07646](https://arxiv.org/abs/2202.07646)) show
that memorization grows log-linearly with how many times an example appears.
Deduplication is not only quality: it is privacy.

**In company documents the problem is worse**, not better. The same contract
template, the same letterhead, the same report in two versions, the same data
processing clause in every annex.

### What parameters people use

The consensus is *5-gram word shingles*. Everything else varies:

| Pipeline | Hashes | Threshold |
|---|---|---|
| The Pile | 10 | 0.5 |
| FineWeb | 112 (14 bands × 8) | 0.75 |
| text-dedup (default) | 240 | 0.7 |
| RefinedWeb / Lee et al. | 9,000 (20 × 450) | 0.8 |

RefinedWeb ([arXiv:2306.01116](https://arxiv.org/html/2306.01116v1)) explicitly
criticises The Pile's 10 hashes: they gave «lower deduplication rates and worse
model performance».

**The detail that costs an afternoon:** lowering the threshold is not enough if
the LSH banding does not follow. Banding decides which pairs get compared at all;
with fixed banding you can lower the threshold as much as you like and find not
one more duplicate, because those pairs are never compared.

## 2. Contamination: the metric stops measuring

If a fragment of the evaluation set is also in the training set, the score
measures what the model remembered, not what it learned.

GPT-3 ([arXiv:2005.14165](https://ar5iv.labs.arxiv.org/html/2005.14165),
appendix C) set the method:

> «We attempted to remove text occurring in benchmarks from training data by
> searching for **13-gram overlaps** […] We ignored 13-grams that matched more
> than 10 training documents, as inspection showed the majority of these to
> contain common cultural phrases, legal boilerplate, or similar content.»

That second sentence is the one that matters in company documents, where the form
dominates the text.

Llama 3 ([arXiv:2407.21783](https://arxiv.org/html/2407.21783v3)) uses 8-grams
with a token-fraction threshold today, and published how contaminated the known
benchmarks are: AGIEval 98%, BIG-Bench Hard 95%, HellaSwag 85%, GSM8K 41%. Tulu 3
found NuminaMath-TIR 11.3% contaminated against MATH.

**Known limit:** n-gram overlap does not see a paraphrase. A model trained on a
rewritten benchmark scores high and passes the filter.

## 3. Leakage through chunking: the silent one

This is the most common mistake and the least visible, because the result is a
**better** metric.

If you chunk the documents first and split the chunks afterwards, two fragments
of the same contract end up one in train and one in test. The model already saw
the template, the vocabulary and the format. It did not generalize: it
recognized.

The principle is in the scikit-learn cross-validation documentation:

> «we need to ensure that all the samples in the validation fold come from groups
> that are not represented at all in the paired training fold»

Kapoor and Narayanan, *Leakage and the Reproducibility Crisis in ML-based
Science* ([arXiv:2207.07048](https://ar5iv.labs.arxiv.org/html/2207.07048)),
catalogue it as an L3.2 leak and count **329 affected papers across 17 fields**.
Work on sliding windows quantifies the effect: naive splitting «can inflate
classification accuracy from a genuine ~20–60% to an artefactual 99.9%», same
data, same model.

Even The Pile admits it of itself: «it is still possible that some documents are
duplicated across the train/validation/test splits».

**The fix is simple:** split by document, not by chunk. And make the split stable
so you can regenerate the set without invalidating your previous evaluation: same
seed, same split, and adding a new document never moves the ones already
assigned.

**What splitting by document does not fix:** two *different* documents can be
nearly identical, like two versions of one contract. That is why steps 1 and 2
still matter.

## 4. Personal data the model ends up reciting

Carlini et al., *Extracting Training Data from Large Language Models*
(USENIX Security 2021, [arXiv:2012.07805](https://arxiv.org/abs/2012.07805)),
extracted hundreds of verbatim sequences from GPT-2, including «names, phone
numbers, and email addresses», **from sequences that appeared in a single
document** of the corpus. And larger models are more vulnerable than smaller
ones.

Public pipelines do less than it seems:

- **Dolma** ([arXiv:2402.00159](https://arxiv.org/html/2402.00159v2)) limits
  itself to «three kinds of PII that are detectable with high precision: email
  addresses, IP addresses and phone numbers». Above five occurrences it discards
  the whole document.
- **FineWeb** redacts only emails and public IPs.
- **Llama 3** removes whole domains heavy in PII, not the PII itself.
- **StarCoder** ([arXiv:2305.06161](https://ar5iv.labs.arxiv.org/html/2305.06161))
  trained a detector on 22,950 hand-annotated entities: EMAIL 99%, NAME 89%, and
  **generic identifier 53%, bad enough that they excluded it**.

That last number is the honest lesson: **nobody detects generic identifiers
reliably**, not even with human annotators. Any tool promising perfect redaction
is selling you something.

**What is reasonable:** redact by default, count how many substitutions were
made, warn if you turn it off, and keep it reversible with a mapping that stays
with you. An irreversible substitution to a fixed token destroys information you
may need; a reversible one lets you go back.

## Order matters

The four steps are not independent:

1. **Deduplicate before splitting.** Split first and near-duplicates land on both
   sides, and step 3 will not save you.
2. **Redact before exporting**, but **classify before redacting**: the
   placeholders erase exactly the words that reveal what type of document it is.
3. **Measure contamination last**, against the real evaluation set, with the
   literal matching fragment in the report so the finding can be checked rather
   than believed.

## How to check it on your documents

Nothing to install, in the browser: [PDF to AI data](/en/pdf-to-jsonl/) shows how
many duplicates were dropped, how many personal data items were replaced and how
the split came out.

In the terminal, each step separately with JSON output for scripts:

```bash
npx sirdas dedupe docs/*.pdf --threshold 0.8 --json
npx sirdas boilerplate docs/*.pdf
npx sirdas contamination train/*.pdf --against eval/*.pdf --n 13
```