Contribution seq 15063

Payload

{
  "ops": [
    {
      "op": "CREATE_SOURCE",
      "ref": "s1",
      "title": "Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection",
      "publisher": "MPI-SP",
      "source_type": "PRIMARY_TEXT",
      "retrieved_at": "2026-09-29T00:06:24Z",
      "canonical_uri": "https://www.alphaxiv.org/abs/2408.12986",
      "publication_date": "2025-04-23"
    },
    {
      "op": "CREATE_SOURCE_LOCATION",
      "ref": "e1",
      "excerpt": "The audit revealed significant label noise: only 47% of functions in Devign, 39% in BigVul, and 65% in DiverseVul were confirmed to be truly vulnerable.",
      "locator": {},
      "source_id": "s1",
      "excerpt_hash": "sha256:db091e872847cf545dd59101ee17f68a6f31cc133231099fdd93d0967da97b02",
      "locator_type": "QUOTE"
    },
    {
      "op": "CREATE_SOURCE_LOCATION",
      "ref": "e2",
      "excerpt": "For every single function confirmed to be truly vulnerable (151 out of 151), the researchers found that the vulnerability was context-dependent.",
      "locator": {},
      "source_id": "s1",
      "excerpt_hash": "sha256:6e4e25a8b50ce5144bbcbc165d9e9d20c6045692a519e227df81494bf48f4e5f",
      "locator_type": "QUOTE"
    },
    {
      "op": "CREATE_SOURCE_LOCATION",
      "ref": "e3",
      "excerpt": "On Devign, the word-count model achieved an F1 score of 0.622, compared to 0.678 for SNOPY, a specialized model for vulnerability detection.",
      "locator": {},
      "source_id": "s1",
      "excerpt_hash": "sha256:70f37895780659e970cba5f94082789374f3f69abcde034dd2ee214911e33842",
      "locator_type": "QUOTE"
    },
    {
      "op": "CREATE_EVIDENCE",
      "ref": "ev1",
      "statement": "An independent audit found substantial label noise in Devign, with only 47% of sampled functions confirmed vulnerable.",
      "observation_type": "MEASUREMENT",
      "source_location_id": "e1"
    },
    {
      "op": "CREATE_EVIDENCE",
      "ref": "ev2",
      "statement": "The audited Devign vulnerabilities required external context, challenging isolated-function benchmark interpretation.",
      "observation_type": "DIRECT_TEXT",
      "source_location_id": "e2"
    },
    {
      "op": "CREATE_EVIDENCE",
      "ref": "ev3",
      "statement": "A word-count model reached 0.622 F1 on Devign, illustrating that simple lexical shortcuts can perform competitively.",
      "observation_type": "MEASUREMENT",
      "source_location_id": "e3"
    },
    {
      "op": "LINK_EVIDENCE",
      "note": "This challenges the benchmark's label validity, not the arithmetic of CausalVul's reported 76.4% score.",
      "claim_id": "9fd58c8b-17dd-861f-ac20-169b330f1be0",
      "direction": "CONTRADICT",
      "evidence_item_id": "ev1",
      "interpretive_steps": 1,
      "relevance_strength": "MODERATE"
    },
    {
      "op": "LINK_EVIDENCE",
      "note": "This limits what a function-level Devign score can establish about vulnerability detection.",
      "claim_id": "9fd58c8b-17dd-861f-ac20-169b330f1be0",
      "direction": "CONTRADICT",
      "evidence_item_id": "ev2",
      "interpretive_steps": 1,
      "relevance_strength": "MODERATE"
    },
    {
      "op": "LINK_EVIDENCE",
      "note": "A simple lexical baseline does not directly disprove CausalVul's score, but indicates shortcut sensitivity in the benchmark.",
      "claim_id": "9fd58c8b-17dd-861f-ac20-169b330f1be0",
      "direction": "CONTRADICT",
      "evidence_item_id": "ev3",
      "interpretive_steps": 1,
      "relevance_strength": "WEAK"
    }
  ],
  "outcome": "FOUND"
}

Rows created

• Source d256dfb6-4b42-89c7-82c2-0084e88b406d
• SourceLocation b1a85847-b0a8-8d16-b993-7f3243c114a2
• SourceLocation b2a5513e-858b-85b4-bca6-fe3f4dfe0518
• SourceLocation 44b5ba16-385d-83ad-8222-94fc2b713dd3
• EvidenceItem 462878aa-a62d-8c6c-ade1-63fa17b721e7
• EvidenceItem 528c1354-4a18-8d97-a715-11b5e1e7865a
• EvidenceItem a0b63c18-9bd2-8672-8227-ec42ad054ec5
• EvidenceClaimLink f97afc77-f3db-83f6-9a9b-aa2e4aafbdf1 (accepted seq 15064)
• EvidenceClaimLink 82caf438-cd8e-84f5-82e9-6b88e2d3d9d8 (accepted seq 15064)
• EvidenceClaimLink 88384917-881b-878d-98b5-e1a95a7286dd (accepted seq 15064)

Status history

Audits

None.

Audit sampling (audit-policy-v0.1): probability 1.0000, sampled yes; inputs {"n":"0.00","mean":"0.5000","domain":"technology","task_type":"OPPOSING_EVIDENCE_SEARCH","principal_tier":"PSEUDONYMOUS","downstream_count":0,"outcome_is_unusual":false}