State of the FleetA living public benchmark of the AI models behind a multi-agent fleet — versioned, objectively graded, and documented in the open. The question isn't “which model is best?” It's “which model do I route each job to?”

How the fleet reaches its models

There is no central router: each agent holds its own provider credentials and a tiered fallback chain, connecting directly to the model APIs it is allowed to use. The bubbles below are live request/response traffic along each agent's primary model path — this is the wiring the rest of the page benchmarks.

Reading top-to-bottom: Hardware (Mac Mini, Windows PC) hosts the agent containers, and every agent connects directly to its providers. Gia (Fable 5 primary) carries the full deck: direct Anthropic, Google, OpenAI, xAI and DeepSeek keys, plus OpenRouter as a routed path (how kimi-k3 and glm are reached) and local Ollama. Mia runs Gemini-flash with Anthropic and Ollama fallbacks; Tia is local-first and capped at T2 (Haiku / Gemini-flash); Cia is local-only by design — no cloud keys at all; Zia runs Sonnet on the Windows box. Solid edges are primary model paths, dashed grey edges are direct fallback tiers.

Winners — current run

Auto-computed from the latest non-superseded results only. “Free” means a model that truly runs at $0 on the local node — grok variants are pricing-unconfirmed and are never labeled free.

How each task shook out

A compact read per task: how many models were tested, how many passed the objective grader, and the fastest & cheapest model that got it right.

Cost, correctness, latency

Lightweight, dependency-free charts rendered inline.

Code task: latency spread (cloud)

Every model, every task

Current (non-superseded) results, sorted by median time. Local models are muted; local $0 marks truly-free models, pricing TBD marks unconfirmed grok pricing. Hover a ✓/✗ for grader detail.

Math — 37 models, ranked by speed

Code — 36 models, ranked by speed

Summarize — 36 models, ranked by speed

Creative — 37 models, ranked by speed

Vision — 18 models, ranked by speed

The protocol

Every model sees the same prompt per task. Cloud models are timed over N=3 trials (median reported, p95 where available); local models are single-trial and their latency includes cold model-load. Grading is objective wherever possible.

Graders, in plain English

• Math: answer string contains 19 (apples-and-coffee budget problem).
• Code: extracted Python is AST-parsed and executed against 6 unit tests; pass = 6/6.
• Summarize: exact word count must land at 50 ±2 words.
• Creative: qualitative only — a 4-line metered poem, no clichés; not objectively graded.
• Vision: model output must contain 3 or three and red and fleet 19 (reads a generated image).

Hypotheses under test

• Is correctness still a differentiator, or has it collapsed (esp. on math)?
• What is the real cloud price floor for a correct answer?
• Are local models a viable free tier, or does latency disqualify them?
• Which provider/model should each job-type route to?

Nothing is hand-curated

Each result carries a run_key = fingerprint of model + task + task_version + grader_version + harness_version. A newer, higher-version result for the same model+task supersedes the old one. 142 of 567 logged results are currently superseded — preserved for audit, excluded from the dashboard.

What changed — harness

• harness v1 · 2026-06-11 — Initial harness: Anthropic + Google callers, serial, single-trial.
• harness v2 · 2026-06-13 — Added OpenAI/xAI/Ollama callers, parallel HTTP, local preflight. Fixed gpt-5 max_completion_tokens param.
• harness v3 · 2026-06-13 — Full output capture (no truncation), inline objective grading, multi-trial cloud timing (median + p95). Found claude-fable-5 needs OpenRouter route (404 on direct Anthropic).
• harness v3 · 2026-06-14 — Bugfix (no version bump): --trials value was leaking into task list causing KeyError crash before append (lost in-memory results). Parser now consumes the flag value and validates task names against manifest.
• harness v3 · 2026-06-28 — 2026-06-28 delta run: claude-fable-5 permanently unavailable (Anthropic returns 404, US export-control order 2026-06-13, msg redirects to Opus 4.8). glm-5.2 blocked this run: OpenRouter key credit cap reached (limit_remaining=0); glm-5.2 also lacks a vision endpoint on OpenRouter. 0 new valid results appended; no deploy.

What changed — graders

• code grader v1 · 2026-06-11 — Qualitative eyeball only — NOT rigorous.
• summarize grader v1 · 2026-06-11 — Qualitative eyeball only — NOT rigorous.
• code grader v2 · 2026-06-13 — Execute code vs 6 unit tests. Requires full output (harness v3). Supersedes all v1/v2 code results.
• summarize grader v2 · 2026-06-13 — Exact word-count check (50 ±2). Requires full output (harness v3). Supersedes all v1/v2 summarize results.
• vision grader v1 · 2026-06-13 — Strict digit match '3' — INVALID, miscounted word-numbers.
• vision grader v2 · 2026-06-13 — Accept word-numbers ('three'). Supersedes all grader-v1 vision results.

Run log

Documented in public, redacted by policy

This is a live experiment run on real infrastructure and shared openly. Infrastructure details (IP addresses, hostnames) are redacted with functional tags — <host-a>, <local-node> — so the methodology is fully reproducible without exposing the network. Benchmark data, prompts, graders, and version history are public; the wiring is not.

The earlier write-ups

The original June 11 fleet analysis (architecture, tokenomics, 7-model v1 benchmark) and the June 13 v2 evolution narrative are preserved for continuity.