Apple Study Maps Scaling Laws for Mixing Scarce and Generic Pretraining Data

Apple researchers ran 2,000+ training runs to chart the tradeoff between scarce target data and generic data in pretraining mixtures.