Implementation skills benchmark ·

Ship the implementation
your customers actually get.

We gave the same AI agent the same Algolia job twice — once with only the docs and the client library, once with the Algolia implementation skills — and blind-graded both against the live index. With the skills, the agent shipped the analytics layer every time, fixed nearly twice the defects that never throw an error, and won 7 of 9 pairs on the models we recommend. One install. One line in your agent config.

Why teams run the skills

Your agent already knows how to call Algolia. The skills change what it considers done — and make it prove it against the live index before it says so.

Complete, not just working

AI-ready on day one

3/3events wired, skills
1/3events wired, bare

From a one-line brief, the skills agent shipped attributed click and conversion events every time — the data that Dynamic Re-Ranking, Personalization, A/B testing and NeuralSearch all run on. Without the skills it shipped them once and twice deferred them as "out of scope."

What you get: a store that starts earning its AI features at launch, not after a health check finds the gap.
Catches what never errors

Right, not just running

7.3silent defects fixed /10
4.3without the skills

Inert ranking chains, facets that can't be searched, price filters with no bounds, internal fields in every response, doubled events. They pass every demo. The skills check each one against live state and re-verify every fix.

What you get: the review a senior Algolia engineer would do, on every build and every audit.
Minutes, not money

Costs almost nothing

+32%agent time
<$1extra tokens per build

The skills are not faster — they make the agent do more. Across the recommended-tier pairs that was about a third more agent time and under a dollar of compute, set against the weeks a dead analytics layer or a leaked field costs to find later.

What you get: ten minutes of agent time buying weeks of production cleanup.
Ten minutes

It's three steps

1 · Installnpx skills add algolia/skills
Or the Claude Code marketplace, Cursor, Codex — anything that reads SKILL.md.
2 · One line in your agent config
"For any Algolia work, start with algolia-discovery-planning; to review an existing setup, start with algolia-audit."
3 · Ask for the work
"Add search to our store." "Audit our Algolia setup." The entry skill loads the rest and hands you a report with live evidence.
Quick start →

Which model should run them?

The skills are instructions; the model executes them. We ran the same tasks on four models so you don’t have to guess. Short version: the mid-tier models most teams already use get the most out of them.

How it's built

The design borrows from Algolia's Agentic Search Leaderboard: score what the agent actually does against live state, never what it says it did.

A real store, seeded300-product catalog on a dedicated Algolia app. For audits, 16 planted defects — most of them symptom-free.
Identical briefs, pinned modelByte-identical prompts per pair. One model per pair, pinned. "Add search to our store, make it good." — or "audit this."
Two arms, one differenceBare: client library, REST, public docs. Skills: the same plus the implementation skills and the CLI. Nothing else changes.
Live checks, blind gradersTen deterministic checks against the live index per run, then a 35- or 37-point rubric scored by a condition-blind grader told to ignore toolchain.
Claims verified, not readEvery scored claim is checked against live state — settings, queries, the rendered page, captured event payloads. Unsupported claims score zero.

What the skills change

On the recommended tier, the gap is not "smarter code." It is a different definition of done.

3/3 vs 1/3

Analytics wired and verified. From a one-line brief, the skills arm shipped attributed click and conversion events every time. The bare arm shipped them once — twice it deferred them as "out of scope."

7.3 vs 4.3/10

Silent defects fixed on an existing store. Tie-breakers, searchable facets, hierarchy, facet stats, leaked internal fields — the class of issue that never throws an error and never fails a demo.

9/10 from 4/10

After the fixes, unprompted. The benchmark found the skills weren't being invoked; we shipped algolia-audit and routing fixes; the free-choice re-run invoked it on its own and fixed 9 of 10.

Every model, every dimension

Summaries recompute from the pairs you filter below. Quality delta is a share of the rubric so build (/35) and audit (/37) pairs share a column without pooling raw points. "Working storefront" is what a shopper would see, checked in a browser after each run.

Every pair

Click a row for what decided it. Excluded rows are shown for completeness and never counted.

Modes. Autopilot: skills installed, agent chooses. Triggered: told to start with algolia-discovery-planning. Mandated: told to read two domain skills first. Skills-assisted: Codex with skills + CLI. Post-fix validation runs are listed in the methodology, not as pairs.

Methodology, limitations, updates

Controls and scoring

One Algolia application, one 300-product catalog (seeded, deterministic), one brief per task type. Each pair pins one model and runs the same brief twice: bare (client library, REST API, public docs) and skills (the same plus the implementation skills and the Algolia CLI). Prompts are byte-identical within a pair.

Two scoring layers. Live checks: ten deterministic assertions against the live index and rendered page after each run (settings shape, facet stats present, replicas actually sorted, retrieval hygiene, page renders, events captured). Blind rubric: 35 points for builds, 37 for audits, scored by a grader that does not know which arm it is grading and is instructed to verify every claim in the agent's write-up against live state. The two rubrics are never pooled.

What "working storefront" means

After each run the page is loaded in a browser. It counts as working if search requests fire, hit cards render with readable text, and there are no uncaught errors. Several runs on the small tier shipped pages that passed every settings check and rendered nothing — a wrong CDN filename for the Insights loader, or a seeded crash left in place — which is why this column exists and why the release-qa skill now has a mechanical page-smoke gate.

Known limitations
  • Sample size. Seven pairs on Claude Sonnet 5; two counted on GPT-5.6; two on Haiku 4.5; one on Opus 5. Variance runs both ways at this n — one adverse pair calibrates a pattern, it does not reverse it, and one favorable pair does not establish one.
  • One catalog, one vertical. Ecommerce only. Support, B2B and marketplace catalogs are the obvious next seeds.
  • Timing is agent time. Wall-clock and tokens as reported by the agent harness; Codex does not expose tokens. Skills cost more time in most pairs (typically +30–40%, up to +65% on Opus) — this benchmark measures correctness, not speed.
  • Graders are models. Condition-blind and told to verify against live state, with deterministic checks as a floor. Disagreements between grader and checker were resolved by re-checking live and are noted in the pair notes.
  • Platform and model are confounded on Codex. GPT-5.6 was only run through Codex; a second Codex model would separate the two.
Post-fix validation runs
Update policy and changelog

Every run is appended to benchmark/data.json with its date, model, mode, both arms' scores, effort, storefront outcome and the grader's decisive note; this page recomputes on build. Excluded runs stay in the data with a reason. Corrections restate the affected pair and are listed here.

  • 2026-09-17 — first public version: 13 pairs (7 Sonnet 5, 2 Haiku 4.5, 1 Opus 5, 3 GPT-5.6/Codex, one excluded), 4 post-fix validation runs.

Want your model or platform on the board? The harness runs the identical tasks on anything an agent platform exposes — contact the Algolia skills team.

Run the skills on your next implementation

One install, one line in your agent config. Start with algolia-discovery-planning for builds, algolia-audit for anything that already exists.

Algolia Implementation Skills · Benchmark