Train an AI model on my own documents
The step nobody explains: preparing the data properly, on your machine, before spending an hour of GPU.
About this tool
"I want a model that knows my domain" sounds like an afternoon and takes three days, nearly all of it on the data. What decides the result is not the training: it is what you feed it.
Three mistakes recur and none of them announces itself. Duplicated text gets memorized instead of generalized. The letterhead and footer that ride along in all hundred documents take up a third of the corpus and teach nothing. And splitting chunks at random between train and test leaves the same template on both sides, so the metric comes out beautiful and false.
This tool handles all three, tells you how much of each it found, and changes nothing without showing you first.
How it works
- 1
Drop all your documents.
- 2
Read the warning about how much of the text is repeated template.
- 3
Tick the train/validation/test split and download the three files.
Questions
- Why is the split by document and not by chunk?
- Because two chunks of the same contract on opposite sides mean the model already saw the template, the vocabulary and the format before the test. It is a silent leak: the only symptom is that the metric improves.
- What are "template paragraphs"?
- The ones identical across several documents: the letterhead, the confidentiality footer, the data-protection clause. Removing them always keeps the first occurrence, so the text stops being repeated rather than disappearing.
- Can I do this with client documents?
- That is why this exists. Nothing is uploaded and anonymization is on. Even so, review the result: detection is rule-based and can miss an unusual format.