From PDF to AI training data
PDF documents turned into a clean, anonymized corpus with provenance, in your browser.
Sent to servers: 0 BNothing left your device
About this tool
The tools that turn PDFs into text end at Markdown. The ones that clean corpora start at JSONL and are Python pipelines built for terabytes of web data. The bridge between them did not exist, and it is exactly where provenance, the split and the anonymization trail live.
Every record that comes out says which file and which section it came from, and how many personal items were replaced. Trainers ignore fields they do not know, so leaving it in costs nothing and lets you trace the example behind a strange answer.
How it works
- 1
Drop the PDFs.
- 2
Pick the format and the cleanup options.
- 3
Download the JSONL with provenance inside.
Questions
- What is the "sirdas" field on each line?
- Provenance: document, chunk number, heading, tokens and how many personal items were replaced. If one example causes trouble, you know which page it came from.
- Does it work with scanned PDFs?
- It needs selectable text. Run each document through local OCR first, which also runs in your browser.
- Can I drop the provenance?
- Yes, from the CLI with --no-meta. On the web it is included because it is cheap and it stops you having a corpus nobody can audit.