Chinchilla: compute-optimal scaling
Chinchilla, 70B parameters trained on 1.3T tokens, outperforms the 280B Gopher at the same training compute.
We ask the question: “What is the optimal model size and number of training tokens for a given compute budget?” To answer this question, we train models of various sizes and with various numbers of tokens, and estimate this trade-off empirically. Our main finding is that the current large language models are far too large for their compute budget and are not being trained on enough data.
Official source
An empirical analysis of compute-optimal large language model trainingdeepmind.google · published