← Leaderboard · batch_8a1d7974251e933dfc1f89a2
x-ai/grok-4.7
x-ai/grok-4.7 · revision r1 · provider xAI · rank #7 · tested 2026-09-30 07:44 UTC · data as of 2026-09-10
82.2%
Strict pass · 179 tasks answered exactly right
179
Tasks in the paper
$0.293
Cost / task (not scored)
44.0s
Median latency (not scored) · 10.4 tool calls / task
Correctness?
Strict pass, all tasks82.2%
Strict pass, data-processing tasks92% (128)
Strict pass, judgment tasks (statement audits, restatements, calendar quarters)57% (51)
Strict pass, tasks that rebuild hidden metrics92% (119)
Strict pass, tasks answerable from served fields63% (60)
Efficiency?
Budget exhausted (forced answer)0%
Output tokens exhausted (reasoning cut off)0%
No answer submitted0%
Tool calls / task · median latency · cost / task10.4 · 44.0s · $0.293
Robustness?
Empty-set tasks answered "none" correctly100% (2)
False positives among returned entities1%
Gold entities missed3%
Correctness says whether it can do the work; efficiency whether it finishes inside the same budget everyone gets; robustness whether it invents or omits. A low score should be readable as one of these, not a mystery.
Pillars?
Strict pass per pillar (share of that pillar’s tasks answered exactly right); the ranking uses the strict pass over all tasks.
Financial Analysis61.735%
Market & Signal90.650%
Failures: NONE 179
Financial Analysis breakdown
Retrieval—25%
Calculation100.030%
Comparison / Trend54.825%
Risk / Position—20%
Market & Signal breakdown
Screening90.025%
Ranking / Top-K100.020%
Derived metric94.415%
Signal detection90.225%
Multi-step75.015%
Task type × domain?
Strict pass over this model’s tasks (share answered exactly right · task count). Rows are task types, columns are the data the answer lives in.
| type | Crypto / Perp | DEX pools | Equity | Financial filings |
|---|---|---|---|---|
| Calculation | · | · | · | 100.0 8 tasks |
| Comparison / Trend | · | · | 53.5 43 tasks | 100.0 1 tasks |
| Screening | · | 100.0 4 tasks | 89.3 56 tasks | · |
| Ranking / Top-K | · | · | 100.0 4 tasks | · |
| Derived metric | 100.0 6 tasks | · | 91.7 12 tasks | · |
| Signal detection | 75.0 4 tasks | 100.0 4 tasks | 90.9 33 tasks | · |
| Multi-step | · | · | 75.0 4 tasks | · |
Where the points went?
Every task not answered exactly right, bucketed by what went wrong. 32 tasks.
| cause | tasks | share |
|---|---|---|
| Both extra and missed entities | 7 | 22% |
| Missed entities (misses only) | 16 | 50% |
| Some fields wrong (multi-field answer) | 3 | 9% |
| Returned extra entities (false positives only) | 5 | 16% |
| Wrong number (outside tolerance) | 1 | 3% |
Task families?
| family | type | tasks | mean | strict |
|---|---|---|---|---|
| J05 Statement Audit — Equity Technicals | Signal detection | 4 | 75.0 | 75% |
| J01 Statement Audit — Equity | Comparison / Trend | 39 | 48.7 | 49% |
| S-CR12-A Pool Buyer/Seller Breadth — count, mean and median of a rebuilt metric over the matches | Signal detection | 3 | 100.0 | 100% |
| S-EQ12-XX Oversold / Overbought (RSI) — plus two more hidden conditions | Signal detection | 4 | 100.0 | 100% |
| S-EQ25-A Growth + Valuation (trailing P/E as a proxy for EV/Sales) — count, mean and median of a rebuilt metric over the matches | Screening | 4 | 100.0 | 100% |
| S-EQ21-XX Growth Acceleration — plus two more hidden conditions | Signal detection | 4 | 75.0 | 75% |
| J02 Statement Audit — Perpetuals | Signal detection | 4 | 75.0 | 75% |
| S-CR18-A Perp RSI Range + Positive Trend — count, mean and median of a rebuilt metric over the matches | Derived metric | 4 | 100.0 | 100% |
| S-EQ05-X First Breakout 20D — plus one more hidden condition | Signal detection | 4 | 75.0 | 75% |
| S-EQ18-A Low-Drawdown Leader — count, mean and median of a rebuilt metric over the matches | Screening | 4 | 50.0 | 50% |
| S-EQ10-XX Pullback in Uptrend — plus two more hidden conditions | Screening | 4 | 100.0 | 100% |
| S-EQ01-XX Size Screen — plus two more hidden conditions | Screening | 4 | 100.0 | 100% |
| S-CR08-N Pool Buy/Sell Transaction-Count Imbalance — count of matches | Signal detection | 1 | 100.0 | 100% |
| J03 Restatement Screen | Comparison / Trend | 4 | 100.0 | 100% |
| S-EQ09-X Moving Average Cross (EMA20 above EMA50 today) — plus one more hidden condition | Signal detection | 3 | 100.0 | 100% |
| S-CR06-M Token DEX Liquidity Threshold (pools) — median of a rebuilt metric over the matches | Screening | 4 | 100.0 | 100% |
| S-EQ22-A Profitability Screen (gross + operating margin) — count, mean and median of a rebuilt metric over the matches | Screening | 4 | 100.0 | 100% |
| S-EQ13-XX ATR Volatility — plus two more hidden conditions | Derived metric | 4 | 75.0 | 75% |
| F09 Free Cash Flow | Calculation | 1 | 100.0 | 100% |
| S-CR19-N Perp Volatility Expansion (ATR% threshold) — count of matches | Derived metric | 1 | 100.0 | 100% |
| F08-R EPS Growth (diluted) — six companies ranked | Calculation | 1 | 100.0 | 100% |
| S-EQ20-XX EPS Growth Screen — plus two more hidden conditions | Screening | 4 | 75.0 | 75% |
| S-EQ26 Cash Flow Quality (FCF margin) | Screening | 4 | 100.0 | 100% |
| F04-R Revenue Growth — six companies ranked | Calculation | 1 | 100.0 | 100% |
| S-EQ19-XX-N Revenue Growth Screen — plus two more hidden conditions — count of matches | Screening | 4 | 100.0 | 100% |
| S-EQ16-XX Relative Strength vs Index — plus two more hidden conditions | Derived metric | 4 | 100.0 | 100% |
| S-EQ06-XX-N Breakout (N-day high) — plus two more hidden conditions — count of matches | Signal detection | 3 | 100.0 | 100% |
| S-EQ33 Market-cap Bucket Ranking | Ranking / Top-K | 4 | 100.0 | 100% |
| S-EQ29-XX Breakout with Exclusion — plus two more hidden conditions | Screening | 4 | 100.0 | 100% |
| F29 Trailing-twelve-month revenue | Calculation | 1 | 100.0 | 100% |
| S-EQ04-XX Unusual Volume — plus two more hidden conditions | Signal detection | 4 | 100.0 | 100% |
| S-EQ23-XX-N Quality Screen (ROE + operating margin as proxies) — plus two more hidden conditions — count of matches | Screening | 4 | 75.0 | 75% |
| F05-R10 CAGR — ten companies ranked | Calculation | 4 | 100.0 | 100% |
| S-EQ07-XX-N Breakdown 60D — plus two more hidden conditions — count of matches | Signal detection | 3 | 100.0 | 100% |
| F15 Peer Comparison (growth ranking) | Comparison / Trend | 1 | 100.0 | 100% |
| S-EQ15-M Gap + Volume — median of a rebuilt metric over the matches | Signal detection | 1 | 100.0 | 100% |
| S-EQ11-A RSI Range — count, mean and median of a rebuilt metric over the matches | Derived metric | 4 | 100.0 | 100% |
| S-EQ25-XX Growth + Valuation (trailing P/E as a proxy for EV/Sales) — plus two more hidden conditions | Screening | 4 | 75.0 | 75% |
| S-EQ02-XX Liquidity Screen — plus two more hidden conditions | Screening | 4 | 100.0 | 100% |
| S-EQ24-A Valuation Screen (trailing P/E) — count, mean and median of a rebuilt metric over the matches | Screening | 4 | 75.0 | 75% |
| S-EQ34-N Multi-period Consistency — count of matches | Screening | 4 | 100.0 | 100% |
| S-X06 Compound Screen with Staged Narrative (liquidity → growth → momentum → rank) | Multi-step | 4 | 75.0 | 75% |
| S-EQ14 Gap Up | Signal detection | 3 | 100.0 | 100% |
| S-CR17-X Perp Trend Alignment (close > EMA20 > EMA50) — plus one more hidden condition | Derived metric | 1 | 100.0 | 100% |
Lowest-scoring tasks?
Public example tasks link to the full question and every model’s answer; hidden tasks show only the family, the score and why.
- 0.0J01Statement Audit — EquityhiddenGold has 1, the model returned 1: 0 correct, 1 false positive, 1 missed → F1 0%. Not a strict pass.
- 0.0J01Statement Audit — EquityhiddenGold has 1, the model returned 1: 0 correct, 1 false positive, 1 missed → F1 0%. Not a strict pass.
- 0.0J01Statement Audit — EquityhiddenGold has 4, the model returned 1: 0 correct, 1 false positive, 4 missed → F1 0%. Not a strict pass.
- 0.0S-EQ23-XX-NQuality Screen (ROE + operating margin as proxies) — plus two more hidden conditions — count of matcheshiddenExpected 3, got 2 (33% off) — outside tolerance.
- 0.0S-EQ18-ALow-Drawdown Leader — count, mean and median of a rebuilt metric over the matcheshidden0 of 3 fields correct; wrong: count, mean, median.
- 0.0S-EQ18-ALow-Drawdown Leader — count, mean and median of a rebuilt metric over the matcheshidden0 of 3 fields correct; wrong: count, mean, median.
- 40.0J01Statement Audit — EquityhiddenGold has 4, the model returned 1: 1 correct, 3 missed → F1 40%. Not a strict pass.
- 50.0J01Statement Audit — EquityhiddenGold has 2, the model returned 2: 1 correct, 1 false positive, 1 missed → F1 50%. Not a strict pass.
Historical runs?
Every batch and every revision, latest first. Old runs are never overwritten: a score belongs to a batch, not to the model forever.
| batch | kind | revision | rank | strict pass | test time | data as of | environment |
|---|---|---|---|---|---|---|---|
| batch_8a1d7974251e933dfc1f89a2 | RELEASE | r1 | #6 | 82.2 | 2026-09-30 07:44 UTC | 2026-09-10 | env-0.1.0-primary |