← Leaderboard · batch_8a1d7974251e933dfc1f89a2
Claude Opus 5.5
anthropic/claude-opus-5.5 · revision r1 · provider Anthropic · rank #3 · tested 2026-09-30 07:10 UTC · data as of 2026-09-10
97.7%
Strict pass · 179 tasks answered exactly right
179
Tasks in the paper
$0.905
Cost / task (not scored)
35.9s
Median latency (not scored) · 7.2 tool calls / task
Correctness?
Strict pass, all tasks97.7%
Strict pass, data-processing tasks98% (128)
Strict pass, judgment tasks (statement audits, restatements, calendar quarters)98% (51)
Strict pass, tasks that rebuild hidden metrics98% (119)
Strict pass, tasks answerable from served fields98% (60)
Efficiency?
Budget exhausted (forced answer)0%
Output tokens exhausted (reasoning cut off)0%
No answer submitted0%
Tool calls / task · median latency · cost / task7.2 · 35.9s · $0.905
Robustness?
Empty-set tasks answered "none" correctly100% (2)
False positives among returned entities0%
Gold entities missed0%
Correctness says whether it can do the work; efficiency whether it finishes inside the same budget everyone gets; robustness whether it invents or omits. A low score should be readable as one of these, not a mystery.
Pillars?
Strict pass per pillar (share of that pillar’s tasks answered exactly right); the ranking uses the strict pass over all tasks.
Financial Analysis97.735%
Market & Signal97.650%
Failures: NONE 179
Financial Analysis breakdown
Retrieval—25%
Calculation100.030%
Comparison / Trend97.325%
Risk / Position—20%
Market & Signal breakdown
Screening96.725%
Ranking / Top-K100.020%
Derived metric100.015%
Signal detection100.025%
Multi-step75.015%
Task type × domain?
Strict pass over this model’s tasks (share answered exactly right · task count). Rows are task types, columns are the data the answer lives in.
| type | Crypto / Perp | DEX pools | Equity | Financial filings |
|---|---|---|---|---|
| Calculation | · | · | · | 100.0 8 tasks |
| Comparison / Trend | · | · | 97.7 43 tasks | 100.0 1 tasks |
| Screening | · | 100.0 4 tasks | 96.4 56 tasks | · |
| Ranking / Top-K | · | · | 100.0 4 tasks | · |
| Derived metric | 100.0 6 tasks | · | 100.0 12 tasks | · |
| Signal detection | 100.0 4 tasks | 100.0 4 tasks | 100.0 33 tasks | · |
| Multi-step | · | · | 75.0 4 tasks | · |
Where the points went?
Every task not answered exactly right, bucketed by what went wrong. 4 tasks.
| cause | tasks | share |
|---|---|---|
| Returned extra entities (false positives only) | 1 | 25% |
| Both extra and missed entities | 1 | 25% |
| Missed entities (misses only) | 2 | 50% |
Task families?
| family | type | tasks | mean | strict |
|---|---|---|---|---|
| J05 Statement Audit — Equity Technicals | Signal detection | 4 | 100.0 | 100% |
| J01 Statement Audit — Equity | Comparison / Trend | 39 | 97.4 | 97% |
| S-CR12-A Pool Buyer/Seller Breadth — count, mean and median of a rebuilt metric over the matches | Signal detection | 3 | 100.0 | 100% |
| S-EQ12-XX Oversold / Overbought (RSI) — plus two more hidden conditions | Signal detection | 4 | 100.0 | 100% |
| S-EQ25-A Growth + Valuation (trailing P/E as a proxy for EV/Sales) — count, mean and median of a rebuilt metric over the matches | Screening | 4 | 100.0 | 100% |
| S-EQ21-XX Growth Acceleration — plus two more hidden conditions | Signal detection | 4 | 100.0 | 100% |
| J02 Statement Audit — Perpetuals | Signal detection | 4 | 100.0 | 100% |
| S-CR18-A Perp RSI Range + Positive Trend — count, mean and median of a rebuilt metric over the matches | Derived metric | 4 | 100.0 | 100% |
| S-EQ05-X First Breakout 20D — plus one more hidden condition | Signal detection | 4 | 100.0 | 100% |
| S-EQ18-A Low-Drawdown Leader — count, mean and median of a rebuilt metric over the matches | Screening | 4 | 100.0 | 100% |
| S-EQ10-XX Pullback in Uptrend — plus two more hidden conditions | Screening | 4 | 100.0 | 100% |
| S-EQ01-XX Size Screen — plus two more hidden conditions | Screening | 4 | 75.0 | 75% |
| S-CR08-N Pool Buy/Sell Transaction-Count Imbalance — count of matches | Signal detection | 1 | 100.0 | 100% |
| J03 Restatement Screen | Comparison / Trend | 4 | 100.0 | 100% |
| S-EQ09-X Moving Average Cross (EMA20 above EMA50 today) — plus one more hidden condition | Signal detection | 3 | 100.0 | 100% |
| S-CR06-M Token DEX Liquidity Threshold (pools) — median of a rebuilt metric over the matches | Screening | 4 | 100.0 | 100% |
| S-EQ22-A Profitability Screen (gross + operating margin) — count, mean and median of a rebuilt metric over the matches | Screening | 4 | 100.0 | 100% |
| S-EQ13-XX ATR Volatility — plus two more hidden conditions | Derived metric | 4 | 100.0 | 100% |
| F09 Free Cash Flow | Calculation | 1 | 100.0 | 100% |
| S-CR19-N Perp Volatility Expansion (ATR% threshold) — count of matches | Derived metric | 1 | 100.0 | 100% |
| F08-R EPS Growth (diluted) — six companies ranked | Calculation | 1 | 100.0 | 100% |
| S-EQ20-XX EPS Growth Screen — plus two more hidden conditions | Screening | 4 | 100.0 | 100% |
| S-EQ26 Cash Flow Quality (FCF margin) | Screening | 4 | 100.0 | 100% |
| F04-R Revenue Growth — six companies ranked | Calculation | 1 | 100.0 | 100% |
| S-EQ19-XX-N Revenue Growth Screen — plus two more hidden conditions — count of matches | Screening | 4 | 100.0 | 100% |
| S-EQ16-XX Relative Strength vs Index — plus two more hidden conditions | Derived metric | 4 | 100.0 | 100% |
| S-EQ06-XX-N Breakout (N-day high) — plus two more hidden conditions — count of matches | Signal detection | 3 | 100.0 | 100% |
| S-EQ33 Market-cap Bucket Ranking | Ranking / Top-K | 4 | 100.0 | 100% |
| S-EQ29-XX Breakout with Exclusion — plus two more hidden conditions | Screening | 4 | 100.0 | 100% |
| F29 Trailing-twelve-month revenue | Calculation | 1 | 100.0 | 100% |
| S-EQ04-XX Unusual Volume — plus two more hidden conditions | Signal detection | 4 | 100.0 | 100% |
| S-EQ23-XX-N Quality Screen (ROE + operating margin as proxies) — plus two more hidden conditions — count of matches | Screening | 4 | 100.0 | 100% |
| F05-R10 CAGR — ten companies ranked | Calculation | 4 | 100.0 | 100% |
| S-EQ07-XX-N Breakdown 60D — plus two more hidden conditions — count of matches | Signal detection | 3 | 100.0 | 100% |
| F15 Peer Comparison (growth ranking) | Comparison / Trend | 1 | 100.0 | 100% |
| S-EQ15-M Gap + Volume — median of a rebuilt metric over the matches | Signal detection | 1 | 100.0 | 100% |
| S-EQ11-A RSI Range — count, mean and median of a rebuilt metric over the matches | Derived metric | 4 | 100.0 | 100% |
| S-EQ25-XX Growth + Valuation (trailing P/E as a proxy for EV/Sales) — plus two more hidden conditions | Screening | 4 | 75.0 | 75% |
| S-EQ02-XX Liquidity Screen — plus two more hidden conditions | Screening | 4 | 100.0 | 100% |
| S-EQ24-A Valuation Screen (trailing P/E) — count, mean and median of a rebuilt metric over the matches | Screening | 4 | 100.0 | 100% |
| S-EQ34-N Multi-period Consistency — count of matches | Screening | 4 | 100.0 | 100% |
| S-X06 Compound Screen with Staged Narrative (liquidity → growth → momentum → rank) | Multi-step | 4 | 75.0 | 75% |
| S-EQ14 Gap Up | Signal detection | 3 | 100.0 | 100% |
| S-CR17-X Perp Trend Alignment (close > EMA20 > EMA50) — plus one more hidden condition | Derived metric | 1 | 100.0 | 100% |
Lowest-scoring tasks?
Public example tasks link to the full question and every model’s answer; hidden tasks show only the family, the score and why.
- 88.9J01Statement Audit — EquityhiddenGold has 4, the model returned 5: 4 correct, 1 false positive → F1 89%. Not a strict pass.
- 90.0S-X06Compound Screen with Staged Narrative (liquidity → growth → momentum → rank)hiddenGold has 10, the model returned 10: 9 correct, 1 false positive, 1 missed → F1 90%; order agreement 100%. Not a strict pass.
- 98.9S-EQ25-XXGrowth + Valuation (trailing P/E as a proxy for EV/Sales) — plus two more hidden conditionshiddenGold has 46, the model returned 45: 45 correct, 1 missed → F1 99%. Not a strict pass.
- 99.4S-EQ01-XXSize Screen — plus two more hidden conditionshiddenGold has 80, the model returned 79: 79 correct, 1 missed → F1 99%. Not a strict pass.
- 100.0J05Statement Audit — Equity TechnicalshiddenGold has 6, the model returned 6: 6 correct → F1 100%. Strict pass.
- 100.0J05Statement Audit — Equity TechnicalshiddenGold has 1, the model returned 1: 1 correct → F1 100%. Strict pass.
- 100.0S-CR12-APool Buyer/Seller Breadth — count, mean and median of a rebuilt metric over the matchesview taskUnique buyers / unique sellers = distinct buying addresses divided by distinct selling addresses over 24h. Find DEX liquidity pools in the top-200-by-24h-volume universe of each supported network with unique buyers / unique sellers greater than 2, 24h unique buyers at least 100, pool liquidity greater than $1M. Scope: only entities with a known value for buyer seller ratio, buyers 24h in the snapshot; entities lacking any of them are out of scope and must not be returned. Entity ids are pool ids of the form network:pooladdress exactly as listed in the snapshot. EVM pool addresses (0x…) are lowercase hex and case-insensitive; Solana and other base58 addresses are CASE-SENSITIVE and must be returned exactly as shown. For the set of ALL entities that match the conditions above, report three numbers: count = how many match; mean = the arithmetic mean of Unique buyers / unique sellers (Distinct buyers divided by distinct sellers over 24h (breadth of demand).) across them; median = its median (middle value, or the mean of the two middle values for an even count). Give mean and median as a decimal ratio (0.25 for 25%). Every matching entity's value must be rebuilt exactly — one wrong value moves the mean.All 3 fields correct.
- 100.0J01Statement Audit — EquityhiddenGold has 5, the model returned 5: 5 correct → F1 100%. Strict pass.
Historical runs?
Every batch and every revision, latest first. Old runs are never overwritten: a score belongs to a batch, not to the model forever.
| batch | kind | revision | rank | strict pass | test time | data as of | environment |
|---|---|---|---|---|---|---|---|
| batch_8a1d7974251e933dfc1f89a2 | RELEASE | r1 | #4 | 97.7 | 2026-09-30 07:10 UTC | 2026-09-10 | env-0.1.0-primary |