← Back to Evals

Search tools · real agent work

Search API Eval v2

Which web search API helps a PR reviewer (Revera) and a coding agent (OpenCode)?

Run 5 providers + no-search baseline Agent / judge: Claude Opus 5.5

Brave, Exa, Parallel, Firecrawl and Tavily. Model settings: adaptive thinking, medium effort.

GitHub repo Full report

Small samples. Treat comparisons as directional. No blended score: each question is answered separately.

Revera · PR review

Search finds breaking-change bugs.

18/18bug-task runs

Search arms pooled vs. no search 0/6.

Recommended: keep Exa auto. Best rank-1 evidence on review-style lookups (11/12) and no strict false positives. Parallel fast is a close, faster, cheaper backup.

OpenCode · coding

No measurable gain from search on these 4 tasks.

4/4no search

= best search arm 4/4 in the quick pass.

Pick on retrieval and latency: Parallel fast (24/24 evidence@5, ~0.5 s per search).

Retrieval · held-out

Exa and Parallel tie at the top.

  • Exa24/24
  • Parallel24/24
  • Tavily22/24
  • Brave21/24
  • Firecrawl16/24

Only Firecrawl is clearly behind; the rest overlap within noise.

Swipe tables horizontally to compare providers.

Retrieval

Can it find the right evidence?

24 held-out questions, one search + read of the top result each. Evidence means the gold fact’s patterns matched provider-returned text, not merely that a familiar URL appeared.

How often is the evidence there?

Rank 1 vs. anywhere in the top 5. Whiskers: Wilson 95% interval.

Exa 23 / 24of 24
Parallel 20 / 24of 24
Tavily 21 / 22of 24
Brave 14 / 21of 24
Firecrawl 12 / 16of 24
Evidence retrieval counts and evidence@5 confidence intervals
ProviderQuestionsEvidence@1Evidence@5Wilson 95% interval
Exa24232486.2% to 100.0%
Parallel24202486.2% to 100.0%
Tavily24212274.2% to 97.7%
Brave24142169.0% to 95.7%
Firecrawl24121646.7% to 82.0%

Speed without losing evidence

Search only. Dot = p50; line extends to p90. Upper left is better.

50%75%100%0.5 s1.0 s2.0 s4.0 s8.0 sEvidence@5Search latency · log scaleExaParallelTavilyBraveFirecrawl
Search latency and evidence@5
ProviderEvidence@5p50 secondsp90 seconds
Exa24/244.4566.185
Parallel24/240.5310.624
Tavily22/243.3835.476
Brave21/240.5170.559
Firecrawl16/240.9421.763

Provider cost per query

Exa is metered; other prices are estimates. Model spend is separate.

Exa $0.0080 metered
Parallel $0.0020 estimate
Tavily $0.0177 estimate
Brave $0.0050 estimate
Firecrawl $0.0030 estimate

Where the difference comes from

Revera-like lookups separate providers. Traps overlap the two slices.

Evidence@1 / evidence@5 by slice; counts, not percentages
ProviderRevera-liken = 12OpenCode-liken = 12Trapsn = 6
Exa 11 / 1212 / 126 / 6
Parallel 8 / 1212 / 125 / 6
Tavily 9 / 1012 / 126 / 6
Brave 6 / 98 / 123 / 4
Firecrawl 4 / 58 / 112 / 3

Each cell: evidence@1 / evidence@5.

Inspect every question

Hover, focus or select a question to read it in full.

24 questions
1–5 = first evidence rank; stronger tint = earlier— = miss! = top-result read failed

Scroll for more providers; the question stays in view.

First evidence rank for every retrieval question and provider. A dash is a miss. An exclamation mark flags a read error.
QuestionExaParallelTavilyBraveFirecrawl
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:2 first evidence rank
Brave:3 first evidence rank
Firecrawl:—miss

r-pymongo-host-injectionsecurity advisory conditionsmedium

A multi-tenant Python service on pymongo 4.18.1 builds its MongoDB connection string by inserting a tenant-supplied hostname fragment into the host portion of the URI. Is it affected by the 2026 PyMongo connection-string host injection advisory, and which version fixes it?

Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:3 first evidence rank
Tavily:1 first evidence rank
Brave:—miss
Firecrawl:—miss
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:—miss
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:—miss
Firecrawl:—miss
Exa:1 first evidence rank
Parallel:3 first evidence rank
Tavily:—miss
Brave:1 first evidence rank
Firecrawl:—miss
Exa:1 first evidence rank
Parallel:2 first evidence rank
Tavily:1 first evidence rank
Brave:2 first evidence rank!
Firecrawl:2 first evidence rank
Exa:4 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:—miss
Firecrawl:—miss
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:5 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:2 first evidence rank
Tavily:—miss
Brave:1 first evidence rank
Firecrawl:—miss
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:2 first evidence rank
Firecrawl:4 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:4 first evidence rank
Firecrawl:—miss
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:2 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:2 first evidence rank!
Firecrawl:3 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:1 first evidence rank
Exa:1 first evidence rank
Parallel:1 first evidence rank
Tavily:1 first evidence rank
Brave:1 first evidence rank
Firecrawl:2 first evidence rank

A blinded LLM judge, shown the same search results and read page, agreed with the deterministic match (gold fact anywhere in that payload) on all 120 units: 115 supported, 5 insufficient, 0 contradicted.

No search call failed; 2 Brave reads failed in the shared reader. Failures stay in the denominators.

Revera

Does search improve a PR review?

2 real breaking-change bugs, 2 clean PRs. The grid includes confirmation reps; the table keeps the equal-sized quick pass separate.

Task-by-task outcomes

One marker per run, in recorded order: quick pass, then confirmation runs where available.

●Bug found / clean with no finding ○Bug missed ×Strict clean-control finding Small amber dot: an uncertain comment was also filed (not a strict finding)
revera task outcomes, including all confirmation runs
Task Exa Parallel Tavily Brave Firecrawl No search
rv-commander15-negatebugUpdate deployment CLI dependencies and add tags
Exa, quick pass: 1/1 bugs found; 0 strict false positives; 0 uncertain findingsExa, confirmation 1: 1/1 bugs found; 0 strict false positives; 0 uncertain findingsExa, confirmation 2: 1/1 bugs found; 0 strict false positives; 0 uncertain findings
Parallel, quick pass: 1/1 bugs found; 0 strict false positives; 0 uncertain findings
Tavily, quick pass: 1/1 bugs found; 0 strict false positives; 0 uncertain findings
Brave, quick pass: 1/1 bugs found; 0 strict false positives; 0 uncertain findings
Firecrawl, quick pass: 1/1 bugs found; 0 strict false positives; 0 uncertain findingsFirecrawl, confirmation 1: 1/1 bugs found; 0 strict false positives; 0 uncertain findingsFirecrawl, confirmation 2: 1/1 bugs found; 0 strict false positives; 0 uncertain findings
No search, quick pass: 0/1 bugs found; 0 strict false positives; 0 uncertain findingsNo search, confirmation 1: 0/1 bugs found; 0 strict false positives; 1 uncertain findingsNo search, confirmation 2: 0/1 bugs found; 0 strict false positives; 0 uncertain findings
rv-got16-http2-agentbugRefresh relay HTTP dependencies and poll timeout
Exa, quick pass: 1/1 bugs found; 0 strict false positives; 0 uncertain findingsExa, confirmation 1: 1/1 bugs found; 0 strict false positives; 0 uncertain findingsExa, confirmation 2: 1/1 bugs found; 0 strict false positives; 0 uncertain findings
Parallel, quick pass: 1/1 bugs found; 0 strict false positives; 0 uncertain findings
Tavily, quick pass: 1/1 bugs found; 0 strict false positives; 0 uncertain findings
Brave, quick pass: 1/1 bugs found; 0 strict false positives; 0 uncertain findings
Firecrawl, quick pass: 1/1 bugs found; 0 strict false positives; 0 uncertain findingsFirecrawl, confirmation 1: 1/1 bugs found; 0 strict false positives; 0 uncertain findingsFirecrawl, confirmation 2: 1/1 bugs found; 0 strict false positives; 0 uncertain findings
No search, quick pass: 0/1 bugs found; 0 strict false positives; 1 uncertain findingsNo search, confirmation 1: 0/1 bugs found; 0 strict false positives; 1 uncertain findingsNo search, confirmation 2: 0/1 bugs found; 0 strict false positives; 1 uncertain findings
rv-got16-migratecleanRefresh the snapshot monitor transport
Exa, quick pass: Clean control; 0 strict false positives; 0 uncertain findingsExa, confirmation 1: Clean control; 0 strict false positives; 0 uncertain findingsExa, confirmation 2: Clean control; 0 strict false positives; 1 uncertain findings
Parallel, quick pass: Clean control; 1 strict false positives; 1 uncertain findings
Tavily, quick pass: Clean control; 1 strict false positives; 0 uncertain findings
Brave, quick pass: Clean control; 1 strict false positives; 0 uncertain findings
Firecrawl, quick pass: Clean control; 0 strict false positives; 1 uncertain findingsFirecrawl, confirmation 1: Clean control; 0 strict false positives; 0 uncertain findingsFirecrawl, confirmation 2: Clean control; 1 strict false positives; 0 uncertain findings
No search, quick pass: Clean control; 0 strict false positives; 0 uncertain findingsNo search, confirmation 1: Clean control; 0 strict false positives; 0 uncertain findingsNo search, confirmation 2: Clean control; 0 strict false positives; 0 uncertain findings
rv-ky2-migratecleanUpdate queue-control HTTP client
Exa, quick pass: Clean control; 0 strict false positives; 0 uncertain findings
Parallel, quick pass: Clean control; 0 strict false positives; 0 uncertain findings
Tavily, quick pass: Clean control; 0 strict false positives; 0 uncertain findings
Brave, quick pass: Clean control; 0 strict false positives; 0 uncertain findings
Firecrawl, quick pass: Clean control; 0 strict false positives; 0 uncertain findings
No search, quick pass: Clean control; 0 strict false positives; 0 uncertain findings

Quick pass · equal task counts

First run only. Costs and wall time are means per review; model prices are estimates.

revera quick pass results and mean per-run costs
Provider / profileBugs foundStrict clean-
control findings		UncertainModel $Search $Wall s
Exa auto2/200$0.0726$0.0075metered40.0
Parallel fast2/211$0.1091$0.0018estimate49.1
Tavily advanced2/210$0.1108$0.0220estimate52.4
Brave web2/210$0.0947$0.0063estimate45.7
Firecrawl search2/201$0.1177$0.0040estimate49.4
No search baseline0/201$0.0391$0.0000not used39.6

Every strict false positive was the same comment on the clean rv-got16-migrate PR (that agent: {http2: false} disables session pooling). The premise is correct per the Got 16 docs; adjudicated as a reasonable non-blocking comment, not a hallucination.

No-search reviewers missed both bugs; on Commander 15 the no-search reviewer asserted the old behavior.

An Exa review used about 1.9x the no-search model cost, with similar wall time.

OpenCode

Does search help the repair pass?

4 dependency-upgrade repairs, scored by hidden tests run offline. Confirmation reps focus on oc-ai7.

Task-by-task outcomes

One marker per run, in recorded order: quick pass, then confirmation runs where available.

●All tests pass ○Task failed
opencode task outcomes, including all confirmation runs
Task Exa Parallel Tavily Brave Firecrawl No search
oc-ai7Restore the catalog assistant on the updated AI SDK
Exa, quick pass: All tests pass; 4/4 testsExa, confirmation 1: All tests pass; 4/4 testsExa, confirmation 2: Task failed; 3/4 tests
Parallel, quick pass: Task failed; 3/4 tests
Tavily, quick pass: Task failed; 3/4 tests
Brave, quick pass: Task failed; 3/4 tests
Firecrawl, quick pass: Task failed; 3/4 testsFirecrawl, confirmation 1: All tests pass; 4/4 testsFirecrawl, confirmation 2: Task failed; 3/4 tests
No search, quick pass: All tests pass; 4/4 testsNo search, confirmation 1: Task failed; 3/4 testsNo search, confirmation 2: All tests pass; 4/4 tests
oc-execa10Restore the release command runner after a dependency update
Exa, quick pass: All tests pass; 3/3 tests
Parallel, quick pass: All tests pass; 3/3 tests
Tavily, quick pass: All tests pass; 3/3 tests
Brave, quick pass: All tests pass; 3/3 tests
Firecrawl, quick pass: All tests pass; 3/3 tests
No search, quick pass: All tests pass; 3/3 tests
oc-ky2Repair the inventory API client after its HTTP dependency update
Exa, quick pass: All tests pass; 4/4 tests
Parallel, quick pass: All tests pass; 4/4 tests
Tavily, quick pass: All tests pass; 4/4 tests
Brave, quick pass: All tests pass; 4/4 tests
Firecrawl, quick pass: All tests pass; 4/4 tests
No search, quick pass: All tests pass; 4/4 tests
oc-ts7-configBring the warehouse typecheck configuration forward
Exa, quick pass: All tests pass; 4/4 tests
Parallel, quick pass: All tests pass; 4/4 tests
Tavily, quick pass: All tests pass; 4/4 tests
Brave, quick pass: All tests pass; 4/4 tests
Firecrawl, quick pass: All tests pass; 4/4 tests
No search, quick pass: All tests pass; 4/4 tests

Quick pass · equal task counts

First run only. Costs and wall time are means per task; model prices are estimates.

opencode quick pass results and mean per-run costs
Provider / profileAll passModel $Search $Wall s
Exa auto4/4$0.1049$0.0105metered69.4
Parallel fast3/4$0.1053$0.0013estimate59.8
Tavily advanced3/4$0.1046$0.0160estimate60.7
Brave web3/4$0.0890$0.0063estimate59.0
Firecrawl search3/4$0.1125$0.0027estimate55.6
No search baseline4/4$0.0590$0.0000not used54.5

Every lost run was on oc-ai7, with the same tsc error: the code used a non-generic ToolExecutionOptions, but it is generic in the installed ai@7.0.143 (in the quick pass, the web arms had copied it from docs). In repeat runs oc-ai7 was noisy for every arm (Exa 2/3, Firecrawl 1/3, none 2/3).

Search arms used about 1.8x the model spend of no search, without adding passes. One untested hypothesis: the model already knows these releases.

Recommendations

Keep search useful. Keep it optional.

Revera: keep Exa auto

Exa auto
  • Keep web search opt-in. Keep Exa REST type:auto with highlights + /contents reads.
  • Raise highlight length to 2,000–4,000 chars. Cutting to 1,000 drops Exa from 24/24 to 22/24 evidence@5.
  • Raise the page read cap with query-aware trimming: 6 of 24 Exa reads had the first evidence past 6,000 chars.
  • Parallel fast + /v1/extract as optional failover.

Highlight and read-depth figures: source: INTEGRATION_RECOMMENDATIONS.md

OpenCode: Parallel fast

Parallel fast
  • Parallel fast for interactive search: 24/24 evidence@5 with ~0.5 s search p50.
  • Exa auto as the equally accurate but slower (~4.5 s) alternative.
  • Keep search an optional tool, not run on every task.

Not tested

Brave LLM Context (brave:llm-context) was blocked: OPTION_NOT_IN_PLAN on the key's Free AI plan. Listed as blocked, not as a loss. Optional reranker ablation skipped.

Read the fine print

Method & limitations

Provider fee estimate

$1.72

List-price estimate; Exa portion metered

Agent model estimate

$7.47

List prices, not a proxy bill

Judge model estimate

$8.86

Model total: $16.33

Same tool contract, frozen profiles

One tool contract for all providers: web_search returns at most 5 results; web_read reads only a returned result. Search responses have a 4,000-token cap; reads have a 6,000-token cap.

Deterministic scoring against gold fact patterns. Failures stay in denominators. Profiles were frozen from a pilot, not tuned on the held-out results.

  • exa:auto
  • parallel:fast
  • tavily:advanced
  • brave:web
  • firecrawl:search

Read the protocol

What this evaluation cannot establish

Small samples; only two Revera bugs, both version-upgrade bugs in JS libraries. Do not read the ordering as a universal provider ranking. Provider indexes change over time.

Model costs are list-price estimates, not proxy bills. Anthropic list-price estimate; proxy billing not visible.

Read all limitations

All results come from the frozen evaluation export; the two recommendation ablations are linked to their source above. Inspect the code and artifacts.