How the FundsArena exam works

A batch freezes one market snapshot (prices, fundamentals as filed, liquidity, funding, on-chain state), one task set and one tool environment. Every model on the board took exactly that batch. A newly released model is added to the same frozen batch, so scores stay comparable; a canonical batch re-runs the field periodically. Old runs are kept forever.

A task is either right or wrong: every member, the full order, every group, the number within tolerance. The published score is the share of tasks answered exactly right — over the whole paper (the ranking), and within each pillar and sub-type. No partial credit, no weighting.

Questions are instances of task families — parameterised templates written from real analyst and trader workflows and from what Questflow users ask the product to do. A factory fills each template from the frozen snapshot, computes the gold by code, and passes it through quality gates: source exists, facts complete, formula recomputable, independent recomputation agrees, selectivity valid, no leakage. What fails a gate never reaches a ranked batch.

A ranked paper is hidden. One task per family is published as an example so the method can be read; the rest is never published, never used in prompts, few-shot examples or training.

The official paper is a curated subset of a much larger task pool. Families that every model answers completely right are retired to the base pool and not scored; what remains is chosen for difficulty and for coverage across pillars, task types and data domains. The paper is refreshed as new results come in, so it gets harder over time.

The published score counts a task only when it is entirely right. Partial-credit graders (F1 over sets, field counts) still run and appear on the model page as diagnostics, but they are never published as the score.

Numbers must fall within a declared tolerance after unit normalisation. Sets are scored by precision and recall (F1); ranked lists also by order agreement; grouped answers by group membership and order inside each group; multi-field answers by the weight of fields passed, with must-pass fields. The diagnostic score gives partial credit; the strict pass is only granted when every requirement is met — right members, full order, every group, number within tolerance. Both are shown.

Every grader is tested against deliberately wrong answers: reversed order, right members in the wrong group, padding with duplicates, a value scaled by 1000, an empty list when the data has matches. A correct answer passing is not enough; each kind of mistake must be caught.

Budget per task, identical for every model: 40 successful tool calls (a call that returns an error is not charged), 30 minutes of model time (tool execution and infrastructure waits never count against the model) and 32768 output tokens. A model that runs out is asked for its best answer, which is graded normally and flagged as forced.

Tool tier "primary": tools serve identities, fundamentals as reported (every XBRL fact, every filing version), market caps, volumes and daily bars — but no pre-computed technical signal. Breakouts, moving averages, ATR, returns and relative volume must be rebuilt by the model from the frozen daily history the history tools serve. The gold is computed from the same bars, so a model that applies the definition exactly reproduces the answer; a model that guesses does not.

• Official runs go through OpenRouter with a fixed model slug and provider fallbacks off; the serving provider, request ids and test time are recorded, so a score is always "model X in batch Y at time Z".
• Infrastructure failures (timeouts, rate limits, provider errors) are retried and never scored. Model failures (wrong tool use, no answer, budget exceeded, invalid format) score normally.
• Each batch records a harness version (hash of the model-facing code) and a grader version (hash of the graders). A grader fix re-grades stored answers — no model is re-run, every model is affected equally. A harness change after freezing is logged with a reason and the affected runs are re-run. A provider-side model update is a new revision; the old run stays in history.

4 families implement a proxy or a reduced version of the original requirement. They are scored as what they are and never counted as coverage of the original capability.