We gave the same AI agent the same Algolia job twice — once with only the docs and the client library, once with the Algolia implementation skills — and blind-graded both against the live index. With the skills, the agent shipped the analytics layer every time, fixed nearly twice the defects that never throw an error, and won 7 of 9 pairs on the models we recommend. One install. One line in your agent config.
Your agent already knows how to call Algolia. The skills change what it considers done — and make it prove it against the live index before it says so.
From a one-line brief, the skills agent shipped attributed click and conversion events every time — the data that Dynamic Re-Ranking, Personalization, A/B testing and NeuralSearch all run on. Without the skills it shipped them once and twice deferred them as "out of scope."
Inert ranking chains, facets that can't be searched, price filters with no bounds, internal fields in every response, doubled events. They pass every demo. The skills check each one against live state and re-verify every fix.
The skills are not faster — they make the agent do more. Across the recommended-tier pairs that was about a third more agent time and under a dollar of compute, set against the weeks a dead analytics layer or a leaked field costs to find later.
npx skills add algolia/skillsalgolia-discovery-planning; to review an existing setup, start with algolia-audit."The skills are instructions; the model executes them. We ran the same tasks on four models so you don’t have to guess. Short version: the mid-tier models most teams already use get the most out of them.
The design borrows from Algolia's Agentic Search Leaderboard: score what the agent actually does against live state, never what it says it did.
On the recommended tier, the gap is not "smarter code." It is a different definition of done.
Analytics wired and verified. From a one-line brief, the skills arm shipped attributed click and conversion events every time. The bare arm shipped them once — twice it deferred them as "out of scope."
Silent defects fixed on an existing store. Tie-breakers, searchable facets, hierarchy, facet stats, leaked internal fields — the class of issue that never throws an error and never fails a demo.
After the fixes, unprompted. The benchmark found the skills weren't being invoked; we shipped algolia-audit and routing fixes; the free-choice re-run invoked it on its own and fixed 9 of 10.
Summaries recompute from the pairs you filter below. Quality delta is a share of the rubric so build (/35) and audit (/37) pairs share a column without pooling raw points. "Working storefront" is what a shopper would see, checked in a browser after each run.
Click a row for what decided it. Excluded rows are shown for completeness and never counted.
Modes. Autopilot: skills installed, agent chooses. Triggered: told to start with algolia-discovery-planning. Mandated: told to read two domain skills first. Skills-assisted: Codex with skills + CLI. Post-fix validation runs are listed in the methodology, not as pairs.
One Algolia application, one 300-product catalog (seeded, deterministic), one brief per task type. Each pair pins one model and runs the same brief twice: bare (client library, REST API, public docs) and skills (the same plus the implementation skills and the Algolia CLI). Prompts are byte-identical within a pair.
Two scoring layers. Live checks: ten deterministic assertions against the live index and rendered page after each run (settings shape, facet stats present, replicas actually sorted, retrieval hygiene, page renders, events captured). Blind rubric: 35 points for builds, 37 for audits, scored by a grader that does not know which arm it is grading and is instructed to verify every claim in the agent's write-up against live state. The two rubrics are never pooled.
After each run the page is loaded in a browser. It counts as working if search requests fire, hit cards render with readable text, and there are no uncaught errors. Several runs on the small tier shipped pages that passed every settings check and rendered nothing — a wrong CDN filename for the Insights loader, or a seeded crash left in place — which is why this column exists and why the release-qa skill now has a mechanical page-smoke gate.
Every run is appended to benchmark/data.json with its date, model, mode, both arms' scores, effort, storefront outcome and the grader's decisive note; this page recomputes on build. Excluded runs stay in the data with a reason. Corrections restate the affected pair and are listed here.
Want your model or platform on the board? The harness runs the identical tasks on anything an agent platform exposes — contact the Algolia skills team.
One install, one line in your agent config. Start with algolia-discovery-planning for builds, algolia-audit for anything that already exists.