Four mistakes that ruin a dataset built from your documents

All four are documented. All four have a fix. And all four show up almost every time someone builds a first dataset out of their company's documents, because public pipelines were designed for web corpora and office documents behave differently: more template, more repetition, more personal data per page.

#1. Duplicates: the model memorizes instead of learning

Lee et al., Deduplicating Training Data Makes Language Models Better (ACL 2022, arXiv:2107.06499):

«over 1% of the unprompted output of language models trained on these datasets is copied verbatim from the training data […] Deduplication allows us to train models that emit memorized text ten times less frequently and require fewer train steps to achieve the same or better accuracy.»

Two numbers for scale. In C4 they found a 61-word sentence repeated more than 60,000 times. CCNet (arXiv:1911.00359) measured duplicate paragraphs at 70% of the text in its corpus.

And Carlini et al. (arXiv:2202.07646) show that memorization grows log-linearly with how many times an example appears. Deduplication is not only quality: it is privacy.

In company documents the problem is worse, not better. The same contract template, the same letterhead, the same report in two versions, the same data processing clause in every annex.

#What parameters people use

The consensus is 5-gram word shingles. Everything else varies:

RefinedWeb (arXiv:2306.01116) explicitly criticises The Pile's 10 hashes: they gave «lower deduplication rates and worse model performance».

The detail that costs an afternoon: lowering the threshold is not enough if the LSH banding does not follow. Banding decides which pairs get compared at all; with fixed banding you can lower the threshold as much as you like and find not one more duplicate, because those pairs are never compared.

#2. Contamination: the metric stops measuring

If a fragment of the evaluation set is also in the training set, the score measures what the model remembered, not what it learned.

GPT-3 (arXiv:2005.14165, appendix C) set the method:

«We attempted to remove text occurring in benchmarks from training data by searching for 13-gram overlaps […] We ignored 13-grams that matched more than 10 training documents, as inspection showed the majority of these to contain common cultural phrases, legal boilerplate, or similar content.»

That second sentence is the one that matters in company documents, where the form dominates the text.

Llama 3 (arXiv:2407.21783) uses 8-grams with a token-fraction threshold today, and published how contaminated the known benchmarks are: AGIEval 98%, BIG-Bench Hard 95%, HellaSwag 85%, GSM8K 41%. Tulu 3 found NuminaMath-TIR 11.3% contaminated against MATH.

Known limit: n-gram overlap does not see a paraphrase. A model trained on a rewritten benchmark scores high and passes the filter.

#3. Leakage through chunking: the silent one

This is the most common mistake and the least visible, because the result is a better metric.

If you chunk the documents first and split the chunks afterwards, two fragments of the same contract end up one in train and one in test. The model already saw the template, the vocabulary and the format. It did not generalize: it recognized.

The principle is in the scikit-learn cross-validation documentation:

«we need to ensure that all the samples in the validation fold come from groups that are not represented at all in the paired training fold»

Kapoor and Narayanan, Leakage and the Reproducibility Crisis in ML-based Science (arXiv:2207.07048), catalogue it as an L3.2 leak and count 329 affected papers across 17 fields. Work on sliding windows quantifies the effect: naive splitting «can inflate classification accuracy from a genuine ~20–60% to an artefactual 99.9%», same data, same model.

Even The Pile admits it of itself: «it is still possible that some documents are duplicated across the train/validation/test splits».

The fix is simple: split by document, not by chunk. And make the split stable so you can regenerate the set without invalidating your previous evaluation: same seed, same split, and adding a new document never moves the ones already assigned.

What splitting by document does not fix: two different documents can be nearly identical, like two versions of one contract. That is why steps 1 and 2 still matter.

#4. Personal data the model ends up reciting

Carlini et al., Extracting Training Data from Large Language Models (USENIX Security 2021, arXiv:2012.07805), extracted hundreds of verbatim sequences from GPT-2, including «names, phone numbers, and email addresses», from sequences that appeared in a single document of the corpus. And larger models are more vulnerable than smaller ones.

Public pipelines do less than it seems:

• Dolma (arXiv:2402.00159) limits itself to «three kinds of PII that are detectable with high precision: email addresses, IP addresses and phone numbers». Above five occurrences it discards the whole document.
• FineWeb redacts only emails and public IPs.
• Llama 3 removes whole domains heavy in PII, not the PII itself.
• StarCoder (arXiv:2305.06161) trained a detector on 22,950 hand-annotated entities: EMAIL 99%, NAME 89%, and generic identifier 53%, bad enough that they excluded it.

That last number is the honest lesson: nobody detects generic identifiers reliably, not even with human annotators. Any tool promising perfect redaction is selling you something.

What is reasonable: redact by default, count how many substitutions were made, warn if you turn it off, and keep it reversible with a mapping that stays with you. An irreversible substitution to a fixed token destroys information you may need; a reversible one lets you go back.

#Order matters

The four steps are not independent:

1. Deduplicate before splitting. Split first and near-duplicates land on both sides, and step 3 will not save you.
2. Redact before exporting, but classify before redacting: the placeholders erase exactly the words that reveal what type of document it is.
3. Measure contamination last, against the real evaluation set, with the literal matching fragment in the report so the finding can be checked rather than believed.

#How to check it on your documents

Nothing to install, in the browser: PDF to AI data shows how many duplicates were dropped, how many personal data items were replaced and how the split came out.

In the terminal, each step separately with JSON output for scripts:

npx sirdas dedupe docs/*.pdf --threshold 0.8 --json
npx sirdas boilerplate docs/*.pdf
npx sirdas contamination train/*.pdf --against eval/*.pdf --n 13