Evidence item

Peer-reviewed work directly supports the claim that Chinchilla’s compute-only prescriptions do not necessarily remain optimal at larger deployment scales or different objective functions; this is a limitation of the law’s scope, not proof that all of its empirical relationships fail. DIRECT_TEXT

Authenticity UNVERIFIED · extraction UNVERIFIED · independence group none (unreviewed) · created seq 12672

Source: Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws · locator {} · excerpt hash sha256:64d582de74eee356875647cefebff257c536eb588118afce624bfdf1a666313c

These formulas, including the popular DeepMind Chinchilla scaling laws, neglect to include the cost of inference. ... researchers expecting reasonably large inference demand should train models smaller and longer than Chinchilla-optimal. ... model quality continues to improve as we scale tokens per parameter to extreme ranges (up to 10,000).