← Leaderboard · batch_8a1d7974251e933dfc1f89a2
google/gemini-3.8-flash
google/gemini-3.8-flash · revision r1 · provider Google · rank #5 · tested 2026-09-30 13:25 UTC · data as of 2026-09-10
94.4%
Strict pass · 179 tasks answered exactly right
179
Tasks in the paper
$0.158
Cost / task (not scored)
79.6s
Median latency (not scored) · 15.3 tool calls / task
Correctness?
Strict pass, all tasks94.4%
Strict pass, data-processing tasks96% (128)
Strict pass, judgment tasks (statement audits, restatements, calendar quarters)90% (51)
Strict pass, tasks that rebuild hidden metrics96% (119)
Strict pass, tasks answerable from served fields92% (60)
Efficiency?
Budget exhausted (forced answer)1%
Output tokens exhausted (reasoning cut off)0%
No answer submitted0%
Tool calls / task · median latency · cost / task15.3 · 79.6s · $0.158
Robustness?
Empty-set tasks answered "none" correctly100% (2)
False positives among returned entities0%
Gold entities missed2%
Correctness says whether it can do the work; efficiency whether it finishes inside the same budget everyone gets; robustness whether it invents or omits. A low score should be readable as one of these, not a mystery.
Pillars?
Strict pass per pillar (share of that pillar’s tasks answered exactly right); the ranking uses the strict pass over all tasks.
Financial Analysis90.335%
Market & Signal96.150%
Failures: NONE 179
Financial Analysis breakdown
Retrieval—25%
Calculation100.030%
Comparison / Trend88.525%
Risk / Position—20%
Market & Signal breakdown
Screening95.025%
Ranking / Top-K100.020%
Derived metric94.415%
Signal detection97.625%
Multi-step100.015%
Task type × domain?
Strict pass over this model’s tasks (share answered exactly right · task count). Rows are task types, columns are the data the answer lives in.
| type | Crypto / Perp | DEX pools | Equity | Financial filings |
|---|---|---|---|---|
| Calculation | · | · | · | 100.0 8 tasks |
| Comparison / Trend | · | · | 88.4 43 tasks | 100.0 1 tasks |
| Screening | · | 100.0 4 tasks | 94.6 56 tasks | · |
| Ranking / Top-K | · | · | 100.0 4 tasks | · |
| Derived metric | 100.0 6 tasks | · | 91.7 12 tasks | · |
| Signal detection | 100.0 4 tasks | 100.0 4 tasks | 97.0 33 tasks | · |
| Multi-step | · | · | 100.0 4 tasks | · |
Where the points went?
Every task not answered exactly right, bucketed by what went wrong. 10 tasks.
| cause | tasks | share |
|---|---|---|
| Missed entities (misses only) | 8 | 80% |
| Some fields wrong (multi-field answer) | 1 | 10% |
| Returned extra entities (false positives only) | 1 | 10% |
Task families?
| family | type | tasks | mean | strict |
|---|---|---|---|---|
| J05 Statement Audit — Equity Technicals | Signal detection | 4 | 100.0 | 100% |
| J01 Statement Audit — Equity | Comparison / Trend | 39 | 87.2 | 87% |
| S-CR12-A Pool Buyer/Seller Breadth — count, mean and median of a rebuilt metric over the matches | Signal detection | 3 | 100.0 | 100% |
| S-EQ12-XX Oversold / Overbought (RSI) — plus two more hidden conditions | Signal detection | 4 | 100.0 | 100% |
| S-EQ25-A Growth + Valuation (trailing P/E as a proxy for EV/Sales) — count, mean and median of a rebuilt metric over the matches | Screening | 4 | 100.0 | 100% |
| S-EQ21-XX Growth Acceleration — plus two more hidden conditions | Signal detection | 4 | 75.0 | 75% |
| J02 Statement Audit — Perpetuals | Signal detection | 4 | 100.0 | 100% |
| S-CR18-A Perp RSI Range + Positive Trend — count, mean and median of a rebuilt metric over the matches | Derived metric | 4 | 100.0 | 100% |
| S-EQ05-X First Breakout 20D — plus one more hidden condition | Signal detection | 4 | 100.0 | 100% |
| S-EQ18-A Low-Drawdown Leader — count, mean and median of a rebuilt metric over the matches | Screening | 4 | 100.0 | 100% |
| S-EQ10-XX Pullback in Uptrend — plus two more hidden conditions | Screening | 4 | 100.0 | 100% |
| S-EQ01-XX Size Screen — plus two more hidden conditions | Screening | 4 | 100.0 | 100% |
| S-CR08-N Pool Buy/Sell Transaction-Count Imbalance — count of matches | Signal detection | 1 | 100.0 | 100% |
| J03 Restatement Screen | Comparison / Trend | 4 | 100.0 | 100% |
| S-EQ09-X Moving Average Cross (EMA20 above EMA50 today) — plus one more hidden condition | Signal detection | 3 | 100.0 | 100% |
| S-CR06-M Token DEX Liquidity Threshold (pools) — median of a rebuilt metric over the matches | Screening | 4 | 100.0 | 100% |
| S-EQ22-A Profitability Screen (gross + operating margin) — count, mean and median of a rebuilt metric over the matches | Screening | 4 | 100.0 | 100% |
| S-EQ13-XX ATR Volatility — plus two more hidden conditions | Derived metric | 4 | 75.0 | 75% |
| F09 Free Cash Flow | Calculation | 1 | 100.0 | 100% |
| S-CR19-N Perp Volatility Expansion (ATR% threshold) — count of matches | Derived metric | 1 | 100.0 | 100% |
| F08-R EPS Growth (diluted) — six companies ranked | Calculation | 1 | 100.0 | 100% |
| S-EQ20-XX EPS Growth Screen — plus two more hidden conditions | Screening | 4 | 100.0 | 100% |
| S-EQ26 Cash Flow Quality (FCF margin) | Screening | 4 | 100.0 | 100% |
| F04-R Revenue Growth — six companies ranked | Calculation | 1 | 100.0 | 100% |
| S-EQ19-XX-N Revenue Growth Screen — plus two more hidden conditions — count of matches | Screening | 4 | 100.0 | 100% |
| S-EQ16-XX Relative Strength vs Index — plus two more hidden conditions | Derived metric | 4 | 100.0 | 100% |
| S-EQ06-XX-N Breakout (N-day high) — plus two more hidden conditions — count of matches | Signal detection | 3 | 100.0 | 100% |
| S-EQ33 Market-cap Bucket Ranking | Ranking / Top-K | 4 | 100.0 | 100% |
| S-EQ29-XX Breakout with Exclusion — plus two more hidden conditions | Screening | 4 | 100.0 | 100% |
| F29 Trailing-twelve-month revenue | Calculation | 1 | 100.0 | 100% |
| S-EQ04-XX Unusual Volume — plus two more hidden conditions | Signal detection | 4 | 100.0 | 100% |
| S-EQ23-XX-N Quality Screen (ROE + operating margin as proxies) — plus two more hidden conditions — count of matches | Screening | 4 | 100.0 | 100% |
| F05-R10 CAGR — ten companies ranked | Calculation | 4 | 100.0 | 100% |
| S-EQ07-XX-N Breakdown 60D — plus two more hidden conditions — count of matches | Signal detection | 3 | 100.0 | 100% |
| F15 Peer Comparison (growth ranking) | Comparison / Trend | 1 | 100.0 | 100% |
| S-EQ15-M Gap + Volume — median of a rebuilt metric over the matches | Signal detection | 1 | 100.0 | 100% |
| S-EQ11-A RSI Range — count, mean and median of a rebuilt metric over the matches | Derived metric | 4 | 100.0 | 100% |
| S-EQ25-XX Growth + Valuation (trailing P/E as a proxy for EV/Sales) — plus two more hidden conditions | Screening | 4 | 50.0 | 50% |
| S-EQ02-XX Liquidity Screen — plus two more hidden conditions | Screening | 4 | 100.0 | 100% |
| S-EQ24-A Valuation Screen (trailing P/E) — count, mean and median of a rebuilt metric over the matches | Screening | 4 | 75.0 | 75% |
| S-EQ34-N Multi-period Consistency — count of matches | Screening | 4 | 100.0 | 100% |
| S-X06 Compound Screen with Staged Narrative (liquidity → growth → momentum → rank) | Multi-step | 4 | 100.0 | 100% |
| S-EQ14 Gap Up | Signal detection | 3 | 100.0 | 100% |
| S-CR17-X Perp Trend Alignment (close > EMA20 > EMA50) — plus one more hidden condition | Derived metric | 1 | 100.0 | 100% |
Lowest-scoring tasks?
Public example tasks link to the full question and every model’s answer; hidden tasks show only the family, the score and why.
- 33.3S-EQ24-AValuation Screen (trailing P/E) — count, mean and median of a rebuilt metric over the matcheshidden1 of 3 fields correct; wrong: mean, median.
- 66.7J01Statement Audit — EquityhiddenGold has 2, the model returned 1: 1 correct, 1 missed → F1 67%. Not a strict pass.
- 79.0S-EQ25-XXGrowth + Valuation (trailing P/E as a proxy for EV/Sales) — plus two more hidden conditionshiddenGold has 46, the model returned 30: 30 correct, 16 missed → F1 79%. Not a strict pass.
- 80.0J01Statement Audit — EquityhiddenGold has 6, the model returned 4: 4 correct, 2 missed → F1 80%. Not a strict pass.
- 85.7J01Statement Audit — EquityhiddenGold has 4, the model returned 3: 3 correct, 1 missed → F1 86%. Not a strict pass.
- 88.9J01Statement Audit — EquityhiddenGold has 5, the model returned 4: 4 correct, 1 missed → F1 89%. Not a strict pass.
- 88.9J01Statement Audit — EquityhiddenGold has 5, the model returned 4: 4 correct, 1 missed → F1 89%. Not a strict pass.
- 97.6S-EQ25-XXGrowth + Valuation (trailing P/E as a proxy for EV/Sales) — plus two more hidden conditionsview taskFundamental ratios are not served as fields: rebuild each one from the raw SEC facts (get_equity_financials) exactly as defined in list_metrics.compute_yourself, using the latest filing at or before the data cutoff; a company whose filings lack the needed facts is out of scope. P/E here is market cap divided by trailing-twelve-month net income — the sum of the four latest single fiscal quarters of net_income from the filings (latest filing copy per quarter, consecutive quarters); names whose TTM net income is not positive have no P/E and are excluded. Find US-listed common equities (instruments without a reported market cap — warrants, units, rights, preferred series, notes — are not in the universe) with market cap greater than $10B, 20-day average dollar volume greater than $100M, latest-quarter revenue growth (YoY) greater than 20%, P/E (market cap / trailing-twelve-month net income from the filings) less than 30, 60-day return greater than 0%, turnover ratio greater than 0.002%. Scope: only entities with a known value for 20-day average dollar volume, latest-quarter revenue growth (YoY), P/E (market cap / trailing-twelve-month net income from the filings), 60-day return, turnover ratio in the snapshot; entities lacking any of them are out of scope and must not be returned. Return every matching ticker; order does not matter. Filing-version policy: every financial value is taken from the LATEST filing accepted by the SEC at or before the data cutoff that reports the requested period. If a later filing (before the cutoff) restates an earlier period, the restated value is the answer; the original filing's value is not. A fiscal-year value is taken only from an annual report (10-K / 20-F / 40-F, including amendments) — a quarterly report never supplies an annual value. Rebuild policy: metrics that are not served as fields (fundamental ratios, growth rates, valuation, technical signals) must be recomputed by you from the raw facts and daily bars the tools serve, following the definitions in list_metrics.compute_yourself to the letter (period selection, latest-filing copy, quarter arithmetic); do the arithmetic with run_calculation over the fetched data. The reference answer was computed from the same facts by the same definitions.Gold has 21, the model returned 20: 20 correct, 1 missed → F1 98%. Not a strict pass.
Historical runs?
Every batch and every revision, latest first. Old runs are never overwritten: a score belongs to a batch, not to the model forever.
| batch | kind | revision | rank | strict pass | test time | data as of | environment |
|---|---|---|---|---|---|---|---|
| batch_8a1d7974251e933dfc1f89a2 | RELEASE | r1 | #3 | 94.4 | 2026-09-30 13:25 UTC | 2026-09-10 | env-0.1.0-primary |