qwen/qwen3.8-flash

Correctness?

Efficiency?

Robustness?

Correctness says whether it can do the work; efficiency whether it finishes inside the same budget everyone gets; robustness whether it invents or omits. A low score should be readable as one of these, not a mystery.

Task type × domain?

Strict pass over this model’s tasks (share answered exactly right · task count). Rows are task types, columns are the data the answer lives in.

Where the points went?

Every task not answered exactly right, bucketed by what went wrong. 5 tasks.

Task families?

Lowest-scoring tasks?

Public example tasks link to the full question and every model’s answer; hidden tasks show only the family, the score and why.

• 0.0
  S-CR08-NPool Buy/Sell Transaction-Count Imbalance — count of matchesview task

  Buy/sell transaction ratio = number of buy swaps divided by number of sell swaps over 24h (transaction COUNTS, not volume — v0.1 proxy for the PRD's buy/sell volume imbalance). Find DEX liquidity pools in the top-200-by-24h-volume universe of each supported network with buy/sell transaction ratio greater than 3, pool liquidity greater than $1M. Scope: only entities with a known value for buy sell txn ratio in the snapshot; entities lacking any of them are out of scope and must not be returned. Entity ids are pool ids of the form network:pooladdress exactly as listed in the snapshot. EVM pool addresses (0x…) are lowercase hex and case-insensitive; Solana and other base58 addresses are CASE-SENSITIVE and must be returned exactly as shown. How many entities match ALL of the conditions above? Answer with the integer count (0 if none).

  Expected 8, got 11 (38% off) — outside tolerance.
• 57.1
  S-EQ04-XXUnusual Volume — plus two more hidden conditionsview task

  Fundamental ratios are not served as fields: rebuild each one from the raw SEC facts (get_equity_financials) exactly as defined in list_metrics.compute_yourself, using the latest filing at or before the data cutoff; a company whose filings lack the needed facts is out of scope. Today's session volume relative to the trailing 20 complete sessions: Find US-listed common equities (instruments without a reported market cap — warrants, units, rights, preferred series, notes — are not in the universe) with market cap greater than $2B, 20-day average dollar volume greater than $50M, sector equal to Technology, relative volume (today / 20-day average) at least 2, fcf margin latest fy greater than 10%, turnover ratio greater than 0.002%. Scope: only entities with a known value for 20-day average dollar volume, sector, relative volume (today / 20-day average), fcf margin latest fy, turnover ratio in the snapshot; entities lacking any of them are out of scope and must not be returned. Return every matching ticker; order does not matter. Filing-version policy: every financial value is taken from the LATEST filing accepted by the SEC at or before the data cutoff that reports the requested period. If a later filing (before the cutoff) restates an earlier period, the restated value is the answer; the original filing's value is not. A fiscal-year value is taken only from an annual report (10-K / 20-F / 40-F, including amendments) — a quarterly report never supplies an annual value. Rebuild policy: metrics that are not served as fields (fundamental ratios, growth rates, valuation, technical signals) must be recomputed by you from the raw facts and daily bars the tools serve, following the definitions in list_metrics.compute_yourself to the letter (period selection, latest-filing copy, quarter arithmetic); do the arithmetic with run_calculation over the fetched data. The reference answer was computed from the same facts by the same definitions.

  Gold has 5, the model returned 2: 2 correct, 3 missed → F1 57%. Not a strict pass.
• 90.0
  S-X06Compound Screen with Staged Narrative (liquidity → growth → momentum → rank)hidden

  Gold has 10, the model returned 10: 9 correct, 1 false positive, 1 missed → F1 90%; order agreement 100%. Not a strict pass.
• 98.9
  S-EQ21-XXGrowth Acceleration — plus two more hidden conditionshidden

  Gold has 45, the model returned 46: 45 correct, 1 false positive → F1 99%. Not a strict pass.
• 98.9
  S-EQ25-XXGrowth + Valuation (trailing P/E as a proxy for EV/Sales) — plus two more hidden conditionshidden

  Budget ran out and the answer was forced. Gold has 46, the model returned 45: 45 correct, 1 missed → F1 99%. Not a strict pass.
• 100.0
  J05Statement Audit — Equity Technicalshidden

  Gold has 1, the model returned 1: 1 correct → F1 100%. Strict pass.
• 100.0
  S-CR12-APool Buyer/Seller Breadth — count, mean and median of a rebuilt metric over the matchesview task

  Unique buyers / unique sellers = distinct buying addresses divided by distinct selling addresses over 24h. Find DEX liquidity pools in the top-200-by-24h-volume universe of each supported network with unique buyers / unique sellers greater than 2, 24h unique buyers at least 100, pool liquidity greater than $1M. Scope: only entities with a known value for buyer seller ratio, buyers 24h in the snapshot; entities lacking any of them are out of scope and must not be returned. Entity ids are pool ids of the form network:pooladdress exactly as listed in the snapshot. EVM pool addresses (0x…) are lowercase hex and case-insensitive; Solana and other base58 addresses are CASE-SENSITIVE and must be returned exactly as shown. For the set of ALL entities that match the conditions above, report three numbers: count = how many match; mean = the arithmetic mean of Unique buyers / unique sellers (Distinct buyers divided by distinct sellers over 24h (breadth of demand).) across them; median = its median (middle value, or the mean of the two middle values for an even count). Give mean and median as a decimal ratio (0.25 for 25%). Every matching entity's value must be rebuilt exactly — one wrong value moves the mean.

  All 3 fields correct.
• 100.0
  J05Statement Audit — Equity Technicalshidden

  Gold has 6, the model returned 6: 6 correct → F1 100%. Strict pass.