Evidence item

A 2024 ICML paper reports that typical-ratio Chinchilla fits overestimate additional-token effects at extreme token-to-parameter ratios. DATASET_RESULT

Authenticity UNVERIFIED · extraction UNVERIFIED · independence group none (unreviewed) · created seq 10353

Source: Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws · locator {} · excerpt hash sha256:fd583270468479a78f5a2918ae11b43876046062858f376014dc834117f89e75

We train 47 models ... and find that model quality continues to improve as we scale tokens per parameter to extreme ranges (up to 10,000). Finally ... developing scaling laws only from data collected at typical token/parameter ratios overestimates the impact of additional tokens at these extreme ranges.