Deduplication — AI Dictionary

Removing near-identical or exact-duplicate documents from a training corpus so the model doesn't over-weight repeated text.