Brains Methodology

The anchor

10/10 = Peak Fable 5, June 9–12 2026 — the operator's reference window, an operator-declared composite constant: protocol floor 10/10, IFEval-mini 100%, and a 20-second velocity reference. These are declared reference values, not retro-measured June results. The anchor never drifts with the market. When any brain sustains the 10.0 ceiling, the public JSON raises anchor.rebase_due and the anchor is re-based upward by an explicit operator decision — never silently.

IQ — the quality composite

IQ = 10 × ( 0.40 × floor/10  +  0.30 × IFEval%/100  +  0.30 × battery/10 )

Three independent lanes, all scored by deterministic code — no LLM judge anywhere. A lane missing fresh, fingerprint-matched evidence drops out and its weight redistributes proportionally, so a stale component can never silently pollute a live score. "IQ" is Portal's internal operational composite for real work, not a standardized intelligence test.

Protocol floor — 10 live probes, hourly

Every hour (07:00–22:00 PT) each brain answers one prompt containing ten protocol probes in one pass. The probes test the operator's real working protocols: instruction fidelity under competing constraints, exact-format echoes, negation handling, multilingual expectation checks (including Russian), refusal-shape correctness, and arithmetic-with-format traps. One point per probe passed by exact deterministic checks; probe parameters are randomized per run so the target moves, and regression fixtures pin every judge against past false-negative classes. The probe texts themselves stay private — see what stays private.

IFEval-mini — instruction following

A fixed 50-item Portal subset of the public IFEval task family, scored with the standard deterministic IFEval checkers (prompt-level pass), refreshed as a full sweep on release days. Published per card: pass/n, the percentage, and a Wilson 95% confidence interval:

center = (p + z²/2n) / (1 + z²/n),  half = z·√(p(1−p)/n + z²/4n²) / (1 + z²/n),  z = 1.96

The subset is fixed (item ids hash-pinned in the release manifest) so cards stay comparable across runs; it is a subset, so figures are not comparable to full public IFEval scores.

Business battery — real deliverables

Three real Portal production tasks — a weekly-forecast body, an HD-manual paragraph, and a Step-2 profile paragraph — each generated three times per brain: nine generations per card per release. Each artifact is scored 0–10 by deterministic structural judges (register rules, required sections, grounded-facts checks, banned-pattern screens) built from Portal's production quality gates. The battery enters IQ only when its methodology fingerprint and its private evidence-file hash match the installed release exactly.

Velocity & delivery

velocity = clamp(1, 10, 10 × 20000 / trailing-24h-median-latency-ms)

The clock includes the full delivery pipeline as configured — session setup, routing, thinking time — because that is what a user actually waits for. Delivery 24h = clean first-attempt completions over measured attempts; provider errors, timeouts, retries, refusals, and detected classifier reroutes all count against it. Our-side configuration failures are excluded (they measure us, not the model).

IQ/$ — quality per dollar

IQ/$ = (IQ / workload$) ÷ (IQ_fable / workload$_fable)     → Fable 5 1M Max = 1.00×

The workload is identical per card: one floor check (trailing-24h median cost) + the 50-item IFEval sweep + the 9-generation battery, priced at dated public list rates for each route (fast tiers where fast is used; the pricing table id is stamped in the JSON). Cursor-SDK cards carry no per-call bill, so their cost is a list-price estimate by construction. Only the indexed ratio is published; absolute dollars stay private. A failed run is cheap because it failed — so the ratio exists only on clean runs, and a cheap failure can never top the column. Not a financial return.

ONE — the default-choice number

ONE answers the operator's actual question: "which brain do I run by default, right now?" It reproduces rational satisficing choice: quality differences too small to distinguish from the board's own measurement noise should cost nothing, while a real quality deficit must dominate any price advantage. Formula one_v1_2026-07-10, pre-registered, computed hourly from published components only — anyone can recompute it from brains.json:

Q_i    = mean IQ over the last 3 canary runs (clean runs only)
B      = clamp( median_i( (max−min of the same 3-run window) / 2 ),  0.5, 1.0 )
d_i    = max(0, max(Q) − Q_i − B)          # deficit BEYOND the live noise band
G_i    = exp( −(4·d_i / B)² )              # smooth quality gate, no cliff
econ_i = 1 + ln(1 + IQ/$_i)                # log: price advantage is bounded
spd_i  = max(0.3, velocity_i / 10)         # linear: users feel wall-time 1:1
del_i  = delivery% / 100                   # (1.0 when unmeasured)
ONE_i  = 100 × (G·econ·spd·del)_i / leader # leader = 100

Why each functional form

• 3-run recency window. The operator's business runs now — the window is the last three hourly runs, so a brain that recovered an hour ago is rehabilitated within three, and yesterday's spike buys nothing.
• Noise band from the board's own data. The band is not an opinion; it is half the board-median spread of the same 3-run windows, re-derived hourly and clamped to [0.5, 1.0] — the floor is the operator's lived noise width (score quantization is ±0.1 and the business lane alone carries roughly ±0.5 at nine samples), so a calm hour can never turn half a point of noise into a cliff. Inside it, the thermometer cannot honestly tell two brains apart — so the gate is exactly 1 and dollars decide.
• Gaussian gate. Smooth at the band edge (a 0.01 IQ move never flips a rank discontinuously) and collapsing faster than any bounded price term can grow — beyond-noise deficits dominate price mathematically, not by tuning.
• Log economics. ln(1+IQ/$) gives cheapness diminishing returns: 10×→20× cheaper matters far less than 1×→2×. No price can buy through a real quality hole.
• Linear speed with a floor. Parallelism does not shorten one person's wait: a thousand users each wait for their own artifact, so per-artifact wall-time carries linear weight; the 0.3 floor keeps a slow but brilliant brain from being zeroed by speed alone.
• Ties: higher recent quality → narrower window spread → higher delivery → name A–Z. Missing IQ/$ or no clean window → the row is unranked (never imputed).

Constants (window 3, band clamp [0.5, 1.0], gate steepness 4, speed floor 0.3) are frozen in version one_v1; any change lands as a new version id with a changelog entry — never a silent retune. The release validator recomputes every published ONE from the same public components and fails the release on any mismatch.

★ Max stakes — the second lane

One ranking cannot honestly serve two jobs. ONE ranks the default lane (volume work, customer deliverables). The gold ★ MAX STAKES badge marks the separate answer to a different question: "which brain when quality outranks cost and time?" — the row with the highest recent dependable quality (the same 3-run Q, not a headline spike). It is a pointer, not a rank; it never moves ONE.

Embeddings — Portal vs a commercial vendor

Paired comparison on identical Portal-authored private fixtures: same documents, same relevance judgments, same 1024 dimensions, cosine similarity, same top-k, each engine at its vendor-recommended query/document settings. Four lanes: Code, English, Russian, Ukrainian.

• Primary endpoint: paired ΔnDCG@10 (Portal − vendor) with a bootstrap 95% CI (2,000 resamples, fixed seed). Recall@10 is reported, never verdict-bearing.
• Pre-registered verdict floor (frozen before the first scored vendor run): CI entirely above +0.02 → Portal ahead; entirely below −0.02 → vendor ahead; CI inside ±0.02 → comparable; anything else → unclear. Lanes under 50 queries are directional only, never verdict-bearing.
• Honesty guards: the full fixture is scored with no post-hoc item dropping; an engine failure scores zero for that query on that engine (availability is part of quality); the text lanes' haystack pairs golden anchors with ~12,000 hash-spread distractors; the code fixture uses first-sentence-of-docstring queries against name-anonymized, comment-stripped function bodies with undocumented functions as distractors — after a first, easier fixture saturated at 1.000 for both engines, its replacement was pre-registered in the run ledger before execution and published regardless of direction.
• Fail-closed publication: the exporter refuses to write anything beyond the aggregate schema, and the page runner independently re-validates the sidecar against the same allowlist before publishing. Raw fixtures, queries, and per-item results stay private.

Route fidelity — measuring the thing we claim to measure

• Portal rail cards (Fable 5, GPT-5.6 Sol, Grok 4.5 Fast, Portal M2-25 Fast, Portal M3 500K Fast) ride the commercial api.portal.ai door with exact response-model receipts and zero commercial fallback: a mismatched or missing receipt makes the card not measured — never silently substituted. Exact-pinned vendor ids additionally carry pre-execution route-fingerprint pins that fail closed on drift.
• Native benchmark exceptions (Opus, Composer) run through the Cursor SDK, which does not echo served-model identity; those rows are measured as requested configuration plus observed delivery and latency, and wear a † on the board.
• Portal cards measure Portal's end-to-end product route at the standard public site price — currently backed by named upstreams through Portal orchestration (disclosed per row). A Portal row and its upstream's row are different pipelines, not two independent observations of one model.

Release integrity

• Every IQ lane carries a methodology/config fingerprint (a hash over the runner, judges, fixtures, and per-card route config) plus a private evidence-file hash; evidence enters IQ only when both match the installed release.
• A release validator independently recomputes IQ, velocity, IQ/$, ONE, and the badge from evidence and published components, verifies evidence freshness against the release cutoff, checks runtime versions, scans the public JSON against a forbidden-key list, and only then issues a signed release receipt. Public access to fresh results is gated on that receipt.
• The public JSON is produced by a whitelist projection (sanitize_public): anything not explicitly allowlisted cannot ship, by construction.

What stays private — and the honest why

• Probe and task texts: a live benchmark dies the day its items leak — models memorize, scores inflate, the thermometer stops measuring. Keeping probes private is what keeps the numbers meaningful tomorrow.
• Prompts, responses, per-item evidence: business-battery tasks are built from Portal's production algorithms — publishing them would publish the product. Aggregates, sample sizes, dates, and formulas are public instead; per-item records are retained privately for substantiation.
• Absolute dollar costs: route economics are competitively sensitive; the indexed IQ/$ ratio carries the decision-relevant information without the ledger.
• Everything needed to audit is retained: fingerprinted evidence files, signed release receipts, and a private confidential audit package exist for every release — available to auditors under agreement, not to scrapers.

Disclosures

Portal AI Inc. designs and runs this benchmark. Portal's own commercial routes appear on the board and Portal has an interest in the results. Portal is not affiliated with, sponsored by, or endorsed by any third-party model provider named on the board; product names and trademarks identify the products tested and belong to their respective owners. Inclusion describes observed technical performance and does not state or expand any commercial license. Results are point-in-time observations from the disclosed harness and may vary with model updates, load, routing, and randomized probe parameters. Nothing on the board is a warranty, an earnings claim, or a recommendation to purchase or use any model. See Portal Terms and the FTC comparative-advertising policy.