BENCHMARKS

How often does Synapse find the right code?

We asked 62 real questions about six codebases, including Flask, Express, Fastify and Hono, and checked every answer against the code by hand. Last run 31 October 2026.

84%Right file, first tryCode graph plus an AI check that reads the code
71%Code graph aloneRight file first, no AI at all
92%Right file in the top 3What your AI agent gets to read first
0False cleanup alarmsUsed code wrongly called unused

How we test

  • Real questions, the kind people ask on a first day in a codebase: “Where are sessions stored?”, “Where is CORS handled?”
  • Answers found by hand. Before asking, we read the code to find the file that answers it. A question can have more than one right file.
  • Pinned versions. Every repository is scanned at one fixed commit or release, so answers can't drift.
  • Run on every change. The test runs in our CI whenever the engine changes, and fails if accuracy drops.
  • “Not sure” is a miss. When the AI says it can't tell, we count it as wrong, even though the page then shows the closest matches.

What the numbers don't say

  • 62 questions is a start, not a final word. We add repositories and questions over time.
  • Two of the six repositories are our own, and we wrote the questions. We also fixed engine misses this test found, so the code graph numbers are partly tuned to these repositories. The next round adds repositories the engine has never seen.
  • Questions here ask for one place in the code. Broad questions (“how does auth work?”) get an overview instead and aren't scored.
  • The code graph covers TypeScript, JavaScript and Python today.

Every question

pallets/flask @ 3.1.311/13 right first
QuestionCode graphTop 3With AI
Where are sessions stored in cookies?
Where is config loaded from files and environment variables?
Where are blueprints registered?
Where is the flask command line?
Where are templates rendered?
Where is the request context pushed?
Where does the route decorator add URL rules?
Where are JSON responses serialized?
Where are class-based views?
Where is the test client?
Where are exceptions turned into error responses?
Where is url_for implemented?
Where are signals defined?
expressjs/express @ v5.3.03/6 right first
QuestionCode graphTop 3With AI
Where is res.json implemented?
Where is the app created?
Where are views rendered?
Where is the client IP read with trust proxy?
Where does app.listen start the server?
Where are cookies set on the response?
fastify/fastify @ v5.12.712/12 right first
QuestionCode graphTop 3With AI
Where are routes registered?
Where are lifecycle hooks like onRequest run?
Where is the request body parsed by content type?
Where is the 404 handler?
Where is the reply serialized and sent?
Where are request schemas validated?
Where are error responses built?
Where are decorators added?
Where does the server start listening?
Where is the logger created?
Where are request ids generated?
Where are HEAD routes added automatically?
honojs/hono @ 5ca49b013/16 right first
QuestionCode graphTop 3With AI
How are middleware handlers chained together?
Where is CORS handled?
Where are JWT tokens verified?
Where is the request body size limited?
Where are cookies parsed?
How does the regex router match routes?
Where is the 404 not found response created?
Where are thrown errors turned into responses?
Where is CSRF protection?
Where are server-sent events streamed?
Where is basic auth checked?
Where are static files served?
Where are responses compressed?
Where is the typed RPC client?
Where are query parameters parsed?
Where is a request timeout enforced?
Aadeshjain25/hotel-rate-comparator @ 4868c97 · our own7/7 right first
QuestionCode graphTop 3With AI
Where is routing handled?
Where would I add rate limiting?
Where is the hotel search API endpoint?
Where are environment variables read?
How is a hotel search cancelled?
Where are supplier results merged?
Where is the Temporal worker started?
Aadeshjain25/Synapse @ this repository · our own6/8 right first
QuestionCode graphTop 3With AI
Where is rate limiting configured?
Where are Razorpay payment signatures verified?
Where are GitHub webhook signatures verified?
Where is refresh token reuse detected?
Where does the PR bot post its comment?
Where is dead code detected?
Where are emails sent?
Where is the landing page hero?

Fewer tokens for your AI agent

Claude Code (Sonnet) did the same real tasks with and without a Synapse brief: 7 tasks per repository, 2 runs each, medians. Every edit compiled in both modes. Run on 20 October 2026.

location-inventory-reservation

public, 27 files, never tuned on

-58%

tokens · 3.44M → 1.46M

-48%

cost · $1.04 → $0.54

Synapse

our own, 410 files

-28%

tokens · 3.20M → 2.29M

-25%

cost · $1.07 → $0.80

A small sample: two repositories and tasks we wrote. Your savings depend on the repository and the task; planning tasks saved the most (up to 71% of the cost).

Try it on a repository you know

Any public GitHub repo, no sign-up. Tell us when it gets one wrong.

Explore a repo