Evidence item

A peer-reviewed benchmark reported a stochastic-parrot phenomenon: models lagged humans by about 40% on abstract physical-concept tasks despite strong natural-language performance. DIRECT_TEXT

Authenticity UNVERIFIED · extraction UNVERIFIED · independence group none (unreviewed) · created seq 15843

Source: The Stochastic Parrot on LLM’s Shoulder: A Summative Assessment of Physical Concept Understanding · locator {} · excerpt hash sha256:920187213cd1b17a497495fac21bcfb385a8269db77cbc7eba4a798d28f19a2b

A comprehensive study on our task demonstrates: (1) state-of-the-art LLMs ... lag behind humans by ∼40%; (2) the stochastic parrot phenomenon is present in LLMs, as they fail on our grid task but can describe and recognize the same concepts well in natural language.