How to turn your PDFs into a training dataset without uploading them anywhere

#The problem nobody mentions until it bites

Fine-tuning a model on your own documents sounds like an afternoon of work. In practice there are only two roads today, and both charge a toll.

The first is uploading the PDFs to a cloud extraction service. Fast, good, and completely unusable the moment the documents carry client data: medical records, contracts, payroll, case files. It is not about trusting the vendor. You are still the data controller, and you just sent the data to a third party.

The second is wiring a Python pipeline designed for terabyte-scale web corpora. It works, but it was built for a different problem: cleaning Common Crawl, not turning forty contracts into a coherent training set.

The middle road is missing, and it is the one this article describes: the whole chain, running where the files already are.

#The five steps

#1. From PDF to structured text

A PDF is not text. It is a list of instructions for drawing glyphs on a page. Extracting it well means recovering the structure — what was a heading, what was a list, where a paragraph starts — because that structure is what later lets you chunk by section instead of by character count.

If the PDF is a scan there is no text layer and you need OCR. That OCR should run locally too: a scanned medical record is exactly the kind of file you do not want to upload.

Warning sign: if the extracted text has headings mixed into paragraphs, the chunking that follows will cut sections in half and every fragment will lose the context that made it useful.

#2. Redact, and be able to undo it

A model repeats what it sees. Carlini et al. (arXiv:2012.07805) extracted hundreds of verbatim training sequences from GPT-2, among them «names, phone numbers, and email addresses», from sequences that appeared in a single document. They also measured that larger models are more vulnerable than smaller ones.

Public pipelines do less than people assume. Dolma limits itself to three types detectable with high precision: emails, IPs and phone numbers. FineWeb redacts emails and public IPs. Llama 3 drops whole domains heavy in PII rather than the PII itself.

The honest lesson comes from StarCoder (arXiv:2305.06161), which trained a detector on 22,950 hand-annotated entities: 99% recall on emails, 89% on names and 53% on generic identifiers, bad enough that they excluded the category. Nobody detects generic identifiers reliably, not even with humans annotating.

Two practical consequences:

• Review. Automatic redaction is a first filter, not a guarantee.
• Make it reversible. Replacing with a consistent placeholder — the same value always becoming the same placeholder — keeps the text coherent and lets you recover the original with the mapping. Replacing with a fixed irreversible token, as Dolma and datatrove do, destroys information you may need later.

#3. Drop the duplicates

Lee et al., Deduplicating Training Data Makes Language Models Better (ACL 2022, arXiv:2107.06499):

«over 1% of the unprompted output of language models trained on these datasets is copied verbatim from the training data […] Deduplication allows us to train models that emit memorized text ten times less frequently and require fewer train steps to achieve the same or better accuracy.»

In C4 they found a 61-word sentence repeated more than 60,000 times. CCNet measured duplicate paragraphs at 70% of its corpus text. A folder of company contracts is a worse case, not a better one: the same template, the same letterhead, the same «this document is confidential» footer.

The standard method is MinHash over 5-gram word shingles, with LSH so you are not comparing every pair. Parameters vary: The Pile used 10 hashes and a 0.5 threshold; FineWeb 112 hashes and 0.75; RefinedWeb explicitly criticises The Pile's 10 for producing «lower deduplication rates and worse model performance».

Detail that costs time: lowering the threshold is not enough if the LSH banding does not follow. With fixed banding the pairs are never compared, so a lower threshold finds nothing new.

#4. Split by document, never by chunk

This is the most common mistake and the least visible, because the result is a better metric.

If you chunk the documents first and split the chunks afterwards, two fragments of the same contract end up one in train and one in test. The model already saw the template, the vocabulary and the format. The scikit-learn documentation says it in one sentence:

«we need to ensure that all the samples in the validation fold come from groups that are not represented at all in the paired training fold»

Kapoor and Narayanan (arXiv:2207.07048) catalogue this as an L3.2 leak and count 329 affected papers across 17 fields. Work on sliding windows quantifies the effect: naive splitting «can inflate classification accuracy from a genuine ~20–60% to an artefactual 99.9%», same data, same model.

The split should also be stable: the same seed always gives the same result, and adding a new document never moves the ones already assigned. Otherwise every regeneration silently invalidates your previous evaluation.

#5. Export with the right field names

Every trainer expects a different schema and the error shows up late, in file validation. The four shapes that cover almost everything:

If you are also splitting, export train.jsonl, validation.jsonl and test.jsonl separately, and keep the provenance of every record next to the set. Provenance is the first thing you will be asked about when the model says something strange.

#What this road cannot do

Worth saying before you find out halfway through the project.

• It does not generate «smart» question-answer pairs from prose. That requires calling a model with your documents' content, which is exactly what this approach avoids.
• It does not build preference pairs for DPO, for the same reason: you need two generated answers.
• It does not read handwriting or turn charts into data. That needs a vision model.

What you can do without a model is more than it looks: a table yields question-answer pairs verifiable by code — what something cost, how many rows, the maximum of a column — which is exactly the shape reinforcement learning with verifiable rewards uses. DeepSeek-R1 (arXiv:2501.12948) explains why they chose code-checkable rewards over a reward model: «the neural reward model may suffer from reward hacking in the large-scale reinforcement learning process».

#How to do it today

In the browser, nothing to install: drop the folder of PDFs into PDF to AI data and download the JSONL.

In your terminal, for thousands of documents:

npx sirdas dataset docs/*.pdf --format chat --split --hf -d dataset/
npx sirdas dedupe docs/*.pdf --threshold 0.8 --json
npx sirdas contamination train/*.pdf --against eval/*.pdf --n 13

Both roads use the same engine and produce the same result. Neither opens a network connection carrying your content.