Contribution seq 22828

Payload

{
  "excerpt": "The authors evaluated Paper2Agent on three corpora: 100 computational-biology papers, 26 data- and discovery-focused papers, and 10 non-biology computational papers spanning AI, statistics, econometrics, game theory and astrophysics. Among the 100 computational-biology papers, 74 were successfully agentified, yielding 599 proposed tools, of which 593 passed automated validation. Failure modes included missing executable code, missing data or model artefacts, environment or dependency failures and non-generalizable scripts. On 300 tutorial-derived benchmark questions, Paper2Agent with Sonnet 4 achieved 91.2 ± 1.6% accuracy, compared with 80.3 ± 2.3% for Claude Code with direct repository access using Sonnet 4 and 86.3 ± 1.1% using Sonnet 4.6. Query cost and latency were lower for Paper2Agent. Across 42 execution-based tasks from 10 non-biology computational papers, it achieved 98.1 ± 0.8% accuracy across five independent runs. Across 26 data- and discovery-focused papers, the resource layer achieved 89.0 ± 3.1% accuracy on 100 synthesis questions, outperforming a browser-use baseline while being cheaper and faster. The system correctly rejected 100% of out-of-scope queries in a permuted paper-question benchmark.",
  "locator": {
    "section": "Large-scale evaluation of Paper2Agent"
  },
  "source_id": "693b0bdd-a05f-8f29-8347-03162e00dbfc",
  "excerpt_hash": "sha256:0763c959f009c0e2a617b4188c2c13d3e698a6c3f1fa14013fcfb872108ec307",
  "locator_type": "TRANSCRIPTION"
}

Rows created

• SourceLocation f9222f56-1372-86b0-b7ee-88e451b60a75

Status history

Audits

None.

Audit sampling (audit-policy-v0.1): probability 1.0000, sampled yes; inputs {"n":"0.00","mean":"0.5000","domain":"general","task_type":"MANUAL","principal_tier":"PSEUDONYMOUS","downstream_count":0,"outcome_is_unusual":false}