Claim

Under project-level group-aware evaluation on the Devign vulnerability-detection benchmark, CausalVul achieved 76.4% F1, 18.0 points above its fine-tuned CodeBERT backbone and 4.8 points above the UniXcoder baseline.

Type QUANTITATIVE · truth-evaluable: yes · status at snapshot: ACTIVE · created seq 14751 · qualifiers {"sampling":"Reported benchmark performance under the paper's project-level group-aware evaluation protocol; not a guarantee of real-world vulnerability-detection performance."}

27 words; atomic claims are usually under 25a conjunction joins clauses that may be separately evaluablea comma-separated series may list several propositions

Referenced 116 times: checked 1, looked up by assistants 0, viewed 89, shared 26. A count of attention, not evidence.

Topics

Security · 1efd24fe AI · 1efd24fe

Assessment

Leans supported

Score: 0.7029 under ledger-default@0.3.0 at snapshot 27745

3 of 4 review checks done · 1 counted · 2 by the author only

4 independent evidence lineages · review checks 1 of 4 · stability MEDIUM

Main unresolved issue: 5 counted evidence items have no independence review; repeats of one origin may be counted as several coverage:9fd58c8b-17dd-861f-ac20-169b330f1be0

Not yet independently audited.Evidence points both ways.Every quoted passage was confirmed on the page when Galedra fetched it.

Why?

Strongest counted support: The abstract reports CausalVul’s Devign F1 score and improvements over CodeBERT and UniXcoder under group-aware evaluation. weight 2.000000 evidence

Strongest counted contradiction: An independent audit found substantial label noise in Devign, with only 47% of sampled functions confirmed vulnerable. weight 0.510000 evidence

Suppressed as dependent: The quoted passage directly reports CausalVul's Devign F1 and both stated comparative improvements.

Review gaps: qualifiers reviewed, opposing search done, independence reviewed

What would most change this: One independent, direct measurement that contradicts this claim would move it from Leans supported to Leans contradicted.

Hide calculation

0.7029 under ledger-default@0.3.0 at snapshot 27745 · state Leans supported · Review checks: 1 of 4 · stability MEDIUM · 5 sources, 4 independent lineages

Model ledger-default@0.3.0 (the default model, not the answer) · snapshot 27745 · trace sha256:fac295db4d8573896ac2df9fe75ba47b70bed8bf3344761bd664d99a3fd12147

Last changed at entry seq 15064; nothing logged since has borne on this claim.

How the number was reached

Each counted link is weighed by multiplying the factors the model declares; a factor it does not declare is not shown. Every factor name below links to what that number means.

Link	Working	Weight	Counted as
b24b7658 support	relevance DIRECT 2.0 × observation MEASUREMENT 1.0 × interpretation 0 steps, 1 − 0.15 × 0 1.000000 × authenticity UNVERIFIED 1.0 × extraction UNVERIFIED 1.0 × provenance INDEPENDENT 1.0 × edition NAMED_OR_UNRELATED 1	2.000000	0.000000 (dependent strongest only, kept 162a9bd3)
e2d3d9d8 contradict	relevance WEAK 0.2 × observation MEASUREMENT 1.0 × interpretation 1 step, 1 − 0.15 × 1 0.850000 × authenticity UNVERIFIED 1.0 × extraction UNVERIFIED 1.0 × provenance INDEPENDENT 1.0 × edition NAMED_OR_UNRELATED 1	0.170000	0.170000
5a7286dd contradict	relevance MODERATE 0.6 × observation MEASUREMENT 1.0 × interpretation 1 step, 1 − 0.15 × 1 0.850000 × authenticity UNVERIFIED 1.0 × extraction UNVERIFIED 1.0 × provenance INDEPENDENT 1.0 × edition NAMED_OR_UNRELATED 1	0.510000	0.510000
4aafbdf1 contradict	relevance MODERATE 0.6 × observation DIRECT_TEXT 0.9 × interpretation 1 step, 1 − 0.15 × 1 0.850000 × authenticity UNVERIFIED 1.0 × extraction UNVERIFIED 1.0 × provenance INDEPENDENT 1.0 × edition NAMED_OR_UNRELATED 1	0.459000	0.459000
162a9bd3 support	relevance DIRECT 2.0 × observation MEASUREMENT 1.0 × interpretation 0 steps, 1 − 0.15 × 0 1.000000 × authenticity UNVERIFIED 1.0 × extraction UNVERIFIED 1.0 × provenance INDEPENDENT 1.0 × edition NAMED_OR_UNRELATED 1	2.000000	2.000000

prior 0.50 → log-odds 0.000000 + 0.861000 (the counted weights) = 0.861000 → probability 0.7029

Under neighbouring settings of the same model: 0.6463 to 0.7539, a spread of 0.1076.

Half-even rounding, six places for weights and four for the probability. The same inputs under the same model reproduce this exactly; that is what the trace hash above is for. How a score is worked out explains every factor, why it is there, and what the model sets it to.

Your view

Registered views

Nobody has registered a view yet.

Registered views are personal belief, not evidence: they never change Galedra's assessment. Affiliations are self-declared and shown only for groups of 5 or more.

Checked as part of

Statements this claim was broken out of. The claim is scored on its own; a statement's reading only counts its claims.

“A 2026 Scientific Reports study introduced CausalVul, which reached 76.4% F1 on the Devign vulnerability-detection benchmark and showed a smaller performance...” · 2 claims · 28 September 2026

Evidence for

The abstract reports CausalVul’s Devign F1 score and improvements over CodeBERT and UniXcoder under group-aware evaluation.

From Causal-aware training with counterfactual supervision and structural causal models improves the robustness of LLM-based vulnerability detection Read 2026-09-28 Relevance DIRECT Steps 0 Counted

The passage directly states the benchmark score and comparative improvements.
The quoted passage directly reports CausalVul's Devign F1 and both stated comparative improvements.

From Causal-aware training with counterfactual supervision and structural causal models improves the robustness of LLM-based vulnerability detection Read 2026-09-28 Relevance DIRECT Steps 0 Counted

The passage explicitly states every quantitative element of the claim.

Evidence against

The audited Devign vulnerabilities required external context, challenging isolated-function benchmark interpretation.

From Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection Read 2026-09-29 Relevance MODERATE Steps 1 Counted

This limits what a function-level Devign score can establish about vulnerability detection.
A word-count model reached 0.622 F1 on Devign, illustrating that simple lexical shortcuts can perform competitively.

From Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection Read 2026-09-29 Relevance WEAK Steps 1 Counted

A simple lexical baseline does not directly disprove CausalVul's score, but indicates shortcut sensitivity in the benchmark.
An independent audit found substantial label noise in Devign, with only 47% of sampled functions confirmed vulnerable.

From Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection Read 2026-09-29 Relevance MODERATE Steps 1 Counted

This challenges the benchmark's label validity, not the arithmetic of CausalVul's reported 76.4% score.

Qualifications

None.

Suppressed as dependent

Evidence	Group	Kept instead
0e018c10		162a9bd3

Inferences

Concluded from

None.

Used as a premise in

None.

An inference is interpretation, not evidence: it never changes the assessment of its conclusion. Each premise is a claim with its own evidence.

Threads

None. Threads are for disagreements about how a determination was made — a statement whose figures are not in the passage it rests on, two sources that may share an origin. Something about the world belongs in the evidence above.

Claim edges

None. Edges are stored and displayed; v0.1 does not propagate probability across them.

Review checklist

Review checks: 1 of 4

3 checks by this claim's own author, alongside the independent review above. 1 OPPOSING EVIDENCE SEARCH · 1 QUALIFIER CHECK · 1 EVIDENCE VERIFICATION. A check by the person who recorded a claim does not raise its review coverage.

Check	Done	By
qualifiers reviewed	no
opposing search done	no
independence reviewed	no
primary source reviewed	yes	b24b7658, e2d3d9d8, 5a7286dd, 4aafbdf1, 162a9bd3

Summary

Leans supported. The main support is: The abstract reports CausalVul’s Devign F1 score and improvements over CodeBERT and UniXcoder under group-aware evaluation. b4804680

Against that, contradiction: A word-count model reached 0.622 F1 on Devign, illustrating that simple lexical shortcuts can perform competitively. ad054ec5

1 further item repeats the same origin and is not independent evidence. 0e018c10

A search for evidence in the opposing direction found something; see the counted evidence. bb4b30b8

Review checks: 1 of 4. coverage:9fd58c8b-17dd-861f-ac20-169b330f1be0

Generated by stub-v0.1; every sentence cites graph ids.

Contribution history

seq	event	detail
14751	APPENDED
14752	ACCEPT	AUTOMATIC_AFTER_VALIDATION
14763	LINK_EVIDENCE	by 1efd24fe (accepted)
14781	TASK_RESULT	by 1efd24fe (accepted)
15063	TASK_RESULT	by 1efd24fe (accepted)

Score trace

{
  "claim": "9fd58c8b-17dd-861f-ac20-169b330f1be0",
  "links": [
    {
      "kept": "fa2477b1-d7c8-8065-b675-3af2162a9bd3",
      "link": "435931f2-1e10-8d28-bfbd-be01b24b7658",
      "group": null,
      "reason": "dependent_strongest_only",
      "edition": "NAMED_OR_UNRELATED",
      "evidence": "4dc3fd9f-e2fc-8eb7-9b64-a25d0e018c10",
      "direction": "SUPPORT",
      "magnitude": "2.000000",
      "relevance": "DIRECT",
      "extraction": "UNVERIFIED",
      "provenance": "INDEPENDENT",
      "observation": "MEASUREMENT",
      "source_type": "PRIMARY_TEXT",
      "authenticity": "UNVERIFIED",
      "audit_confirmed": false,
      "effective_weight": "0.000000",
      "interpretive_steps": 0
    },
    {
      "link": "82caf438-cd8e-84f5-82e9-6b88e2d3d9d8",
      "group": null,
      "edition": "NAMED_OR_UNRELATED",
      "evidence": "a0b63c18-9bd2-8672-8227-ec42ad054ec5",
      "direction": "CONTRADICT",
      "magnitude": "0.170000",
      "relevance": "WEAK",
      "extraction": "UNVERIFIED",
      "provenance": "INDEPENDENT",
      "observation": "MEASUREMENT",
      "source_type": "PRIMARY_TEXT",
      "authenticity": "UNVERIFIED",
      "audit_confirmed": false,
      "effective_weight": "0.170000",
      "interpretive_steps": 1
    },
    {
      "link": "88384917-881b-878d-98b5-e1a95a7286dd",
      "group": null,
      "edition": "NAMED_OR_UNRELATED",
      "evidence": "462878aa-a62d-8c6c-ade1-63fa17b721e7",
      "direction": "CONTRADICT",
      "magnitude": "0.510000",
      "relevance": "MODERATE",
      "extraction": "UNVERIFIED",
      "provenance": "INDEPENDENT",
      "observation": "MEASUREMENT",
      "source_type": "PRIMARY_TEXT",
      "authenticity": "UNVERIFIED",
      "audit_confirmed": false,
      "effective_weight": "0.510000",
      "interpretive_steps": 1
    },
    {
      "link": "f97afc77-f3db-83f6-9a9b-aa2e4aafbdf1",
      "group": null,
      "edition": "NAMED_OR_UNRELATED",
      "evidence": "528c1354-4a18-8d97-a715-11b5e1e7865a",
      "direction": "CONTRADICT",
      "magnitude": "0.459000",
      "relevance": "MODERATE",
      "extraction": "UNVERIFIED",
      "provenance": "INDEPENDENT",
      "observation": "DIRECT_TEXT",
      "source_type": "PRIMARY_TEXT",
      "authenticity": "UNVERIFIED",
      "audit_confirmed": false,
      "effective_weight": "0.459000",
      "interpretive_steps": 1
    },
    {
      "link": "fa2477b1-d7c8-8065-b675-3af2162a9bd3",
      "group": null,
      "edition": "NAMED_OR_UNRELATED",
      "evidence": "781d4515-eed3-8c36-9bb5-f2c2b4804680",
      "direction": "SUPPORT",
      "magnitude": "2.000000",
      "relevance": "DIRECT",
      "extraction": "UNVERIFIED",
      "provenance": "INDEPENDENT",
      "observation": "MEASUREMENT",
      "source_type": "PRIMARY_TEXT",
      "authenticity": "UNVERIFIED",
      "audit_confirmed": false,
      "effective_weight": "2.000000",
      "interpretive_steps": 0
    }
  ],
  "model": "ledger-default@0.3.0",
  "prior": "0.50",
  "variants": {
    "spread": "0.1076",
    "permissive": "0.7539",
    "conservative": "0.6463"
  },
  "code_hash": "sha256:c61a6fdf35a754a0d7e958df149036a083c0eaeb7d2d2e52704ec32b0c36def7",
  "contested": true,
  "stability": "MEDIUM",
  "claim_type": "QUANTITATIVE",
  "config_hash": "sha256:86768038080b8ab5233f1e4b7aba10af05b5bb44caa432095150b90e81209809",
  "probability": "0.7029",
  "provisional": true,
  "evidence_sum": "0.861000",
  "snapshot_seq": 15064,
  "prior_log_odds": "0.000000",
  "support_groups": 1,
  "model_dependent": false,
  "review_coverage": "0.25",
  "truth_evaluable": true,
  "assessment_state": "LEANS_SUPPORTED",
  "review_checklist": {
    "qualifiers_reviewed": {
      "by": [],
      "ok": false
    },
    "opposing_search_done": {
      "by": [],
      "ok": false
    },
    "independence_reviewed": {
      "by": [],
      "ok": false
    },
    "primary_source_reviewed": {
      "by": [
        "435931f2-1e10-8d28-bfbd-be01b24b7658",
        "82caf438-cd8e-84f5-82e9-6b88e2d3d9d8",
        "88384917-881b-878d-98b5-e1a95a7286dd",
        "f97afc77-f3db-83f6-9a9b-aa2e4aafbdf1",
        "fa2477b1-d7c8-8065-b675-3af2162a9bd3"
      ],
      "ok": true
    }
  },
  "contradict_groups": 3,
  "rounding_boundary": false,
  "posterior_log_odds": "0.861000",
  "not_applicable_reason": null,
  "independence_unreviewed": 5
}