Evidence item

OpenAI’s official September 2024 account confirms large-scale reinforcement learning and test-time compute as the basis of o1 reasoning, but it does not mention an internal project named RL Slow or establish that it coincided with Q* or Strawberry. DIRECT_TEXT

Authenticity UNVERIFIED · extraction UNVERIFIED · independence group none (unreviewed) · created seq 12432

Source: Learning to reason with LLMs · locator {} · excerpt hash sha256:92a21f7ccb39415a77aea465fe233f638c0db22c7524619dd02c1ec557bf836e

Our large-scale reinforcement learning algorithm teaches the model how to think productively using its chain of thought in a highly data-efficient training process. ... We have found that the performance of o1 consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute).