ONC-AGI

Can an agent find the signal—and know when there is none?

ONC-AGI, a benchmark for biomarker discovery, asks one thing:
Here is a cohort with an outcome. Which measurements drive it? Return an ordered list, most likely first, or an empty list if nothing can be found.

Cubes: each patient's level of each candidate biomarker · Towers: how strongly each biomarker tracks the outcome

Why this task

Biomarker discovery produces many promising associations and few clinical tools. Poor methods, selective reporting and failed validation are recurring problems12. Four failures recur:

  • a correlate mistaken for a cause;
  • a confounder that lifts many measurements at once;
  • a variable measured after the outcome, which "predicts" it trivially;
  • a confident claim where the data support none.

ONC-AGI plants known mechanisms in cohort data, turning each failure into a test with a checkable answer. An agent that passes does the work of a careful biostatistician: it finds the cause, not its correlates, and says nothing when nothing is there.

Try one

A world from the benchmark's training set: 240 patients and 23 measurements. Each patient costs $108 with every assay. Buy patients, read the evidence, then name what drives the outcome, most likely first, or nothing.

  1. Buy patients

    $25,920 to spend; the reference study costs $17,280.

  2. List what drives the outcome

    Click the bars to add measurements, in order. Click again to remove one.

    Evidence, −log10 p
    Gene expression
    Copy number
    Protein
    Clinical
    Derived
    Lab
    After outcome

    Buy patients to see the evidence.

    One bar per measurement, square-root scale. Hollow bars were measured after the outcome. Without JavaScript the evidence after 90 patients is shown.

    Follow one discovery

    240 patients. 23 measurements. A fixed budget.

    The cohort240 patients × 23 measurementsndp3cqt1tjn8tnr4gwt2jdh4wcl7rws7gwg3xlg5shq1fsc9ppt1rqg2ltj7chj6pvq6bkb2dlq6lsd2bqx7njh5kbb1Planted driverNear-duplicate, also acceptedOutcomeeventsExpressionCopy no.ProteinClinicalDerivedLabAfter outcome−2.5+2.5 zRows sorted by outcome
    Every fourth patient of 240. Colour encodes the standardised value of each measurement.
    0%20%50%80%100%↑ Detected304560100160240Patients per cohort →80%Driver ndp3Near-duplicate wcl7Unrelated jdh4lower 95% bound ≥ 80%
    0%20%50%80%100%↑ Detected3060120240Patients per cohort →80%
    Setting the bar0246810|z|, strength of association →Threshold 3.0695% of no-signal cohorts stay below400 cohorts with no signal:the strongest |z| in eachDriver ndp3:200 cohorts of 240
    Marginal tests on 200 redraws of the outcome per cohort size, with 95% Wilson intervals; the red mark is where the driver's lower bound first clears 80%. The threshold comes from 400 no-signal outcomes.
    Training worldsequential access
    Patients available240 · $40 each to recruit
    Budget$25,920
    Measurements · assay price per patient
    Expressionndp3$2cqt1$2tjn8$2tnr4$2gwt2$2jdh4$2wcl7$2rws7$2gwg3$2xlg5$2
    Copy no.shq1$3fsc9$3ppt1$3
    Proteinrqg2$8ltj7$8
    Clinicalchj6$0.5pvq6$0.5
    Derivedbkb2$1dlq6$1
    Lablsd2$5
    After outcomebqx7$5njh5$5kbb1$5
    Submitan ordered list, or nothing
    Answer keyheld by the scorer
    Identifiers are masked. Kind, timing and price are visible; hollow chips were measured after the outcome.
    Evidence per measurement−log₁₀ p0246810Agent's line · Bonferroni 5%ExpressionCopy no.ProteinClinicalDerivedLabAfter outcome30 patients · $3,240List [ndp3]ndp3: −log10 p = 3.984ndp3cqt1: −log10 p = 0.974tjn8: −log10 p = 0.772tnr4: −log10 p = 0.854gwt2: −log10 p = 0.705jdh4: −log10 p = 0.215wcl7: −log10 p = 2.545rws7: −log10 p = 0.011gwg3: −log10 p = 0.011xlg5: −log10 p = 0.245shq1: −log10 p = 0.757fsc9: −log10 p = 0.825ppt1: −log10 p = 0.712rqg2: −log10 p = 0.857ltj7: −log10 p = 0.064chj6: −log10 p = 0.1pvq6: −log10 p = 0.053bkb2: −log10 p = 0.418dlq6: −log10 p = 0.141lsd2: −log10 p = 0.2bqx7: −log10 p = 0.138njh5: −log10 p = 1.082kbb1: −log10 p = 0.07860 patients · $6,480List [ndp3, wcl7]ndp3: −log10 p = 8.176ndp3cqt1: −log10 p = 0.593tjn8: −log10 p = 0.162tnr4: −log10 p = 0.954gwt2: −log10 p = 0.361jdh4: −log10 p = 0.228wcl7: −log10 p = 7.047wcl7rws7: −log10 p = 0.186gwg3: −log10 p = 0.34xlg5: −log10 p = 0.032shq1: −log10 p = 0.539fsc9: −log10 p = 0.989ppt1: −log10 p = 0.32rqg2: −log10 p = 0.379ltj7: −log10 p = 0.223chj6: −log10 p = 0.195pvq6: −log10 p = 0.063bkb2: −log10 p = 0.191dlq6: −log10 p = 0.054lsd2: −log10 p = 0.052bqx7: −log10 p = 0.014njh5: −log10 p = 1.859kbb1: −log10 p = 0.00390 patients · $9,720List [ndp3, wcl7] repeats → stopndp3: −log10 p = 9.193ndp3cqt1: −log10 p = 1.12tjn8: −log10 p = 0.474tnr4: −log10 p = 0.999gwt2: −log10 p = 0.152jdh4: −log10 p = 0.068wcl7: −log10 p = 8.199wcl7rws7: −log10 p = 0.163gwg3: −log10 p = 0.173xlg5: −log10 p = 0.109shq1: −log10 p = 0.63fsc9: −log10 p = 0.553ppt1: −log10 p = 0.124rqg2: −log10 p = 0.81ltj7: −log10 p = 0.39chj6: −log10 p = 0.217pvq6: −log10 p = 0.342bkb2: −log10 p = 0.008dlq6: −log10 p = 0.054lsd2: −log10 p = 0.009bqx7: −log10 p = 0.01njh5: −log10 p = 0.847kbb1: −log10 p = 0.096■ before the outcome□ after the outcome: cannot countred: over the agent's line
    Each square is one measurement, in fixed position. Outlines trace where it stood at the previous purchase.
    Submittedndp3wcl7Count equivalents oncesame clusterndp3Keep the top R = 1recoverable causesndp3Answer keyheld by the scorerndp3 · wcl7matchCorrect for chance01random list 0.21this list 1.00(1.00 − 0.21) ÷ (1 − 0.21)Find = 1.00Post-outcome measurements listednoneCost$9,720 spent$17,280 reference study56% of the reference study
    Same world, other answersMatches keyNo leakLists somethingFindThe agent's answerndp3, wcl71.00The near-duplicate alonewcl71.00Add a post-outcome measurementndp3, bqx70.00Return nothing[ ]0.00List everything, in random orderall 20 baseline measurements10% of orders−0.01 meanOn a world with no signal, only “return nothing” earns Restraint.
    Find is credit for the planted cause, corrected for what a random list of the same kinds of measurement would earn.
    1. Build
    2. Certify
    3. Receive
    4. Act
    5. Score
    0 / 240 patients 0 USD spent reference study 17,280 USD

    1 · Build a world

    Benchmark worlds preserve the correlations in real cancer cohorts. This one has 240 patients and 23 measurements.

    ndp3 drives the outcome. Its near-duplicate, wcl7, carries almost the same information. The task accepts either.

    Generate an outcome from the planted mechanism. The signal is known; which patients experience the outcome remains noisy.

    3 measurements are taken after the outcome. Such variables can leak the answer. Feature names are masked so the agent must work from the data.

    2 · Check what can be found

    Redraw the outcome hundreds of times to check whether the planted signal can be detected. Here the driver is detected reliably, in at least 80% of cohorts even at the lower end of its 95% interval, from about 60 patients; an unrelated measurement almost never is.

    The bar comes from worlds with no signal: the strength of association that 95% of them never reach. When the data cannot tell a measurement from the driver, it is accepted as equivalent.

    3 · Give the agent the task

    The agent receives:

    • how many patients exist;
    • every measurement, with its kind and whether it was taken after the outcome;
    • the price of recruiting a patient and of each assay;
    • a budget.

    The interface withholds the answer. The agent buys data before analysing it.

    4 · Acquire evidence

    This baseline recruits patients in batches and tests each measurement, correcting for multiple comparisons. Only measurements taken before the outcome can enter its answer.

    At 30 patients and 3,240 USD, one measurement crosses its threshold.

    At twice the sample size, the near-duplicate crosses too. More evidence supports the signal without separating the two measurements.

    At 90 patients, the list repeats. The baseline stops and submits [ndp3, wcl7], spending 9,720 of 25,920 USD.

    5 · Score the answer

    The scorer counts equivalent measurements once, then keeps only the first R: the number of recoverable causes, one here.

    Either twin matches the key. Credit is corrected for what a random list of the same kinds of measurement would recover.

    A post-outcome measurement invalidates the answer. Acquisition cost is compared with the 17,280 USD reference study; this run costs less and keeps its full credit.

    The same scorer, run on other answers: the near-duplicate alone earns full credit; a post-outcome measurement removes it; silence misses this signal; listing everything depends on a lucky order.

    Across many worlds, a fifth with no signal at all, the headline is Discovery Score = Find × Restraint. Restraint rewards returning nothing when there is nothing, so always answering, never answering, guessing and padding all score zero.

    Figures as editable SVG: Cohort Detection Threshold Evidence Scoring Other answers

    What makes it hard

    Each mechanism is built so that a common shortcut fails. Six of them, shortcut against analysis:

    Respect time

    A measurement taken after the outcome tracks it closely.

    Hollow bars were measured after the outcome: they cannot count, even above the line.

    1251020Evidence, −log10 pxzn2xks9ttd6jwz1ldc3vwd7sjr2sjr2llq9ptj3qdn7dnq3qqx5xbb4tfd3xmj5hpd1ljr4lcg4ptl9lxh7lxh7 · after the outcomewzq9nqf8brc4multiple-testing line
    Shortcut
    Rank everything by association: lxh7 comes first, a leak, so Find falls to 0.
    Analysis
    Rank only what was measured before the outcome: sjr2, Find 1.
    240 patients.

    Separate cause from confounding

    A clinical subtype raises risk and lifts five genes.

    Dashed outlines: each gene with the subtype ignored. Solid bars: compared within each subtype.

    12510Evidence, −log10 psubtypeg1g2g3g4g4 · lifted by the subtypeg5x7x7 · acts aloneo1o2o3o4o5o6multiple-testing line
    Shortcut
    Screen genes on their own: g4, g2, g3 lead, all lifted by the subtype.
    Analysis
    Compare within each subtype: only x7 remains.
    360 patients.

    Look for joint effects

    The outcome follows two measurements together; neither matters alone.

    a1 →a2eventno eventEvidence (−log₁₀ p)a1 alone0.1a2 alone0.3a1 × a226.2multiple-testing line
    Shortcut
    Screen one measurement at a time: nothing crosses the line.
    Analysis
    Test the pair jointly: a1 × a2 stands far above it.
    400 patients.

    Look across data types

    A copy-number change drives the outcome and switches on a gene.

    1251020Evidence, −log10 pc1c1 · copy numberc2c3c4e1e1 · switched on by c1e2e3e4e5e6e7e8multiple-testing lineCopy numberGene expression
    Shortcut
    Read only gene expression: e1, the gene it switches on.
    Analysis
    Read every data type: c1, the copy-number cause.
    300 patients.

    Respect how the data were pooled

    Two batches differ in both marker level and event rate.

    Batch ABatch Bmarker level →Pooled: higher marker, fewer events (−log₁₀ p 6.2)Within each batch: higher marker, more events (5.7)eventno event
    Shortcut
    Pool the batches: the marker looks protective.
    Analysis
    Compare within each batch: it raises risk in both.
    400 patients.

    Know when there is nothing

    A twin of a real world with every effect removed.

    Hollow bars were measured after the outcome.

    125Evidence, −log10 pwwr7lxz1mmq4zgj2dkq4mkf5sbg2dvb9vcv5mwh2jsw8lxb1bcq3fsm8qvf2vgf1cxx1qxc8mjq1rsg2bpv1jhz8jhz8 · strongest, still belowxzt7multiple-testing line
    Shortcut
    Always name the strongest measurement: jhz8, a false claim.
    Analysis
    Nothing crosses the line, so return nothing: the only answer that earns Restraint.
    240 patients.

    The benchmark also plants near-duplicates that count as one answer, hidden causes seen only through proxies, modules of genes acting together, mediators and colliders, and sequential worlds where every patient and assay costs money.

    What a score means

    • The agent recovered planted mechanisms in real correlation structure, against an oracle certificate, more often than chance.
    • It said "nothing" when nothing was there, and paid for data in proportion to the question.
    • The agent interface withholds the key. Public training answers remain available to developers.

    Run your agent

    Bring a method or an agent. Read the world card, choose actions, and submit an ordered list through the public engine.

    Needs Python 3.12 and uv. The smoke test takes about ten seconds.

    Or use the hosted ONC-AGI Arena: sign in with GitHub, bring a model or upload an agent, and see verified results on the live leaderboard.

    git clone https://github.com/BradSegal/ONC-AGI && cd ONC-AGI
    uv sync
    uv run onc-agi smoke
    Expected outputsmoke passed
    ONC-AGI smoke - 20 public-train worlds
    oracle             DS  1.000  [+1.00,+1.00]  Find  1.00  Restraint +1.00  Strict  1.00  Leak 0.00
    univariate_bh      DS  0.511  [+0.29,+0.75]  Find  0.72  Restraint +0.71  Strict  0.51  Leak 0.00
    seq_univariate_bh  DS  0.578  [+0.35,+0.84]  Find  0.82  Restraint +0.71  Strict  0.58  Leak 0.00
    stability          DS  0.407  [+0.21,+0.65]  Find  0.63  Restraint +0.65  Strict  0.41  Leak 0.00
    random             DS  0.000  [-0.00,+0.00]  Find  0.00  Restraint +0.00  Strict  0.00  Leak 0.00
    giant_list         DS  0.000  [-0.00,+0.00]  Find  0.00  Restraint +0.00  Strict  0.00  Leak 1.00
    always_empty       DS  0.000  [+0.00,+0.00]  Find  0.00  Restraint -0.17  Strict  0.00  Leak 0.00
    leak_exploiter     DS  0.000  [+0.00,+0.00]  Find  0.00  Restraint +0.00  Strict  0.00  Leak 1.00
    random_abstain     DS  0.000  [-0.08,+0.00]  Find  0.08  Restraint -0.40  Strict  0.00  Leak 0.00
    smoke passed