Best Web Search API for AI Agents: Six APIs Compared on Price, Page Text, MCP and Failed Calls

Six search APIs for AI agents compared: price per 1,000 calls, page text, freshness, rate limits, MCP and failed-call billing, with 20 live test calls.

Illustration of a small robot at a desk. In front of it, six glass vending machines stand in an arc. Some machines give out thin paper strips, some give out thick bound documents, and some give out cards with small tags. The middle machine has a teal outline and gives out both strips and documents. A small balance scale on the desk holds coins on one side and a paper strip on the other. The picture shows how you pick a search API by price and by what it returns.

There is no single best web search API for AI agents, but there is a clear winner for each job. For high volume with links only, Serper is the cheapest: $0.30 per 1,000 searches on its largest pack, against our $0.694. Now take an agent that searches once and then reads three pages. Per 1,000 tasks, we cost $2.78, Bright Data $6.00, Tavily $8.00, Firecrawl $19.00 and SerpAPI $100.

We ran 20 live calls against our own API on 2026-11-07. Across 8 web calls, which include one empty result, the median time was 1.74 seconds. Six calls failed or found nothing (401, 422, 429 and one empty result), and none of them cost money. One cross-check call lost one of its four methods, so it charged $0.002083 for three methods instead of the $0.002778 list price.

The other five vendors we did not call. We read their public pricing pages, docs and GitHub releases, and every number about them links to that source. Where a page did not say something, we write "not stated".

This guide is for developers who are choosing a search tool for an agent. The phrase "best web search API for AI agents" hides a simple point. A model reads the results, not a person, so you must compare the APIs on what the model needs. We compared six products.

Five are other vendors: Serper, Tavily, SerpAPI, Firecrawl and Bright Data. The sixth is our own search API, EverySearch (POST /api/v1/search). Its tool list is free to read without a key.

Illustration of three conveyor belts that lead to a rounded box on the right. The box is the model's context window and it is half full. The top belt carries many thin paper strips, which are search snippets. The middle belt carries thick stacks of pages, which are full page text. The bottom belt carries single sheets past three magnifier stations, and the sheet that all three stations checked gets a teal check clip. That sheet is a source that several search methods found.
A search API gives a model one of three things: snippets, full text, or checked sources. Pick the vendor after you know which one your agent needs.

The summary table: best web search API for AI agents across eight factors

The table puts the six APIs side by side on the eight factors that matter most to an agent. The price row is the entry paid price for 1,000 searches of 10 results each. The number in brackets is the lowest public tier. Vendor numbers come from their pricing pages and docs, which we read on 2026-11-07 between 00:22 and 00:50 Singapore time (SGT). Our numbers come from the search tool list and from billing.charged_usd in live responses.

FactorEveryInfraSerperTavilySerpAPIFirecrawlBright Data
Per 1,000 searches (entry / lowest)$0.694 / $0.694$1.00 / $0.30$8.00 / $5.00$25.00 / $9.17$7.60 / $1.50$1.50 / $1.30
Free usage$1 for starring our two GitHub repos, valid 24 hours2,500 queries, no card1,000 credits a month, plus keyless trial250 searches a month1,000 credits a month, plus keyless trial5,000 requests a month
What the model getsTitle, link, snippet of about 150 characters, dateStructured JSON of the results pageUp to 3 relevant chunks per result, full text optionalJSON or MarkdownTitle, description, highlights, full text optionalRaw HTML or parsed JSON
Full text in one calldeep with full=true, or readNot in the 10 APIs on the home pageinclude_raw_contentSeparate Extract callscrapeOptions, 1 credit per pageSeparate scrape call
Freshness controlafter / before datesPage says no cachingtime_range, max_age_hoursCached searches are freetbs time filterEngine parameters
Requests per minute, entry tier60 (account default)3,000 (50 per second)1,000 (production key)About 3.3 (200 per hour)100 (/search)Page says unlimited
Remote MCPYes, 12 tools, list tools without a keyNot listed on home pageYes, OAuth or keyYes, keyYes, keyless endpoint with 3 toolsYes, token in URL
Failures and empty resultsFailures cost $0, empty results refundedOnly successful responses chargedFailed extractions freeErrors free, empty result countsNo document, no charge; 403/404 pages cost 1 creditPay for success only

Most people misread the first row. Our $0.694 is a flat price with no tiers. Serper's $0.30 needs you to buy 12.5 million queries for $3,750 at once, and the pack expires after 6 months. Firecrawl's $1.50 comes from its Scale plan at $749 a month.

Bright Data's $1.30 is the overage price on its Scale plan. So Serper wins on the lowest unit price, and we win on the smallest first payment. Each section below takes one factor. It gives the source text first, then our live calls, and then what the factor means for an agent.

How we compared: public prices, source text and our own live calls

The six APIs bill in different units, so we set three rules. Serper and SerpAPI count searches. Tavily counts credits: 1 for a basic search and 2 for advanced (Tavily credits docs). Firecrawl charges 2 credits per 10 search results, rounded up (Firecrawl billing docs). Bright Data charges $1.5 per 1,000 SERP requests or MCP results (SERP pricing, MCP pricing). We charge $0.000694 per search call and $0.002778 per cross-check.

Rule one is a common unit: one search that returns 10 results, times 1,000. Rule two is two prices per vendor. The entry paid tier is the smallest plan you can buy without a contract. The lowest tier is the lowest public unit price on the page. We left out prices that only sales can quote, because we cannot check them.

Rule three is that our own numbers come only from live calls. On November 7, 2026, from 00:15 to 00:29 SGT, one test key made 20 calls for a total charge of $0.012495. Each time is the curl time_total from Singapore, so it includes the network round trip.

Agents need more than ten links: what the API gives the model

A person scans a results page and picks a link. A model reads every character, and every character costs tokens. So the shape of the response matters more than speed when you judge a search API for agents. The six APIs take four paths here.

The first path is the results page. Serper and SerpAPI turn a results page into structured JSON. The Serper home page example has four blocks: knowledgeGraph, organic, peopleAlsoAsk and relatedSearches. This shape is best for SEO and rank tracking, because it keeps positions and answer boxes. For an agent it is often too much, because the model needs only a few good sources.

SerpAPI saw this problem too. Its Markdown output page shows its own test. One Google search took about 150K tokens as JSON and about 39K as Markdown, a 74% cut. A Google Shopping search dropped from 361K to 38K tokens. Ask for Markdown with output=md, the /search.md path, or an Accept: text/markdown header.

The second path is chunks. Each Tavily result has a content field with up to 3 relevant chunks. Each chunk is about 500 characters, or about 800 at advanced depth (Tavily search endpoint). Tavily picks the chunks by how well they match the query. Add include_answer for a short answer, or include_raw_content for the full page. Firecrawl search also returns highlights by default, and highlights=false gives a plain description (Firecrawl search docs).

The third path is full text. Add scrapeOptions to a Firecrawl search, and it fetches each result page too, with extra credits per page. Bright Data's SERP API returns raw HTML by default, and brd_json=1 returns parsed JSON (Bright Data SERP docs). Our deep tool with full=true returns text with the results. At 00:17:02 SGT it returned 5 results about the MCP specification, with 47,596 characters of text in total, for $0.000694. The fourth path is checked sources, which our crosscheck tool gives, as a later section shows.

Our web tool takes the plainest form. Each result has four fields: title, url, snippet and published. Our 8 web calls with results, 7 over REST and 1 over MCP, returned 75 results.

Of these, 72 had a snippet and 39 had a date. Ten results add up to less than 2,000 characters, which fits in any context window. The cost is less detail, with no knowledge panel and no position field. The web/search capability of our data API adds rank and sub_links at $0.001389, twice the price.

APIMain result fieldsWhat each result gives the modelSwitches that save tokensSource
EveryInfra webtitle, url, snippet, publishedSnippet, median about 150 charactersAlready short; use read or deep for full textSearch tool list
Serperorganic[].title/link/snippet/position and moreResults page snippetNot statedSerper home page
Tavilyresults[].title/url/content/score, optional raw_contentUp to 3 relevant chunkschunks_per_source, max_resultsSearch endpoint
SerpAPIorganic_results[].position/title/link/snippetResults page snippetoutput=md, about 50% fewer tokens on averageMarkdown output
Firecrawldata.web[].title/url/description, optional markdownHighlights or descriptionhighlights=false, scrapeOptions only when neededSearch docs
Bright DataRaw HTML, or JSON with brd_json=1Results page contentbrd_json=1 for parsed fields onlySERP docs

More fields do not make an API better for an agent. An agent that puts a 10,000-character results page into context each turn soon pushes the important facts out.

Our own numbers show the gap. Ten web results added up to less than 2,000 characters, but one deep call with full text returned 47,596 characters, about 24 times more. So use short snippets to find candidates, such as our web, Tavily basic or SerpAPI Markdown. Then read full text only for the few pages you chose. The next section prices that pattern.

Page text: the cost of "search once, read three pages"

The search price alone hides what most agents really spend. Agents seldom answer from snippets. Research, support and price-check agents usually search once, pick 2 to 5 promising results, read them, and then answer. We took the middle case, one search plus three page reads, as one task. Then we priced 1,000 tasks on each vendor's entry paid tier.

Horizontal bar chart on a log scale with steps of 10 times. Title: cost of 1,000 tasks that search once and read 3 pages. From low to high: EveryInfra deep with full text $0.694, EveryInfra web plus read $2.78, Bright Data $6.00, Tavily $8.00, Firecrawl $19.00, SerpAPI $100. The two EveryInfra bars are teal and the rest are gray.
APIHow one task is builtUnit price used (entry tier)Per 1,000 tasksSource
EveryInfra1 web + 3 read$0.000694 per call$2.78Search tool list
EveryInfra1 deep with full=true, 5 results with text$0.000694 per call$0.694Our live call at 00:17:02 SGT
Bright Data1 MCP search + 3 scrapes$1.50 per 1,000 results$6.00MCP pricing
Tavily1 basic search with include_raw_content1 credit at $0.008$8.00Credits docs
FirecrawlSearch 10 results (2 credits) + 3 pages (1 credit each)Hobby $19 for 5,000 credits$19.00Billing docs
SerpAPI1 search + 3 Extract callsStarter $25 for 1,000 searches$100Extract API
SerperNo page text API among the 10 on the home page—Needs a separate scraperSerper home page
Screenshot of the Firecrawl billing docs. It shows a three-column table: Endpoint, Credit Cost and Notes. Scrape and Crawl cost 1 credit per page, and Map costs 1 credit per call. The Search row has a teal box. It reads 2 credits per 10 results. The note says the count rounds up, so 11 results cost 4 credits. Each scraped result is billed as a scrape.
Screenshot of the Firecrawl billing docs, taken at 00:33 SGT on 2026-11-07. The teal box marks the search rule "2 credits / 10 results" and the extra charge for scraped results. Firecrawl Billing

Some rows need a note. Tavily looks cheap because its credit docs bill by search depth, and include_raw_content adds no credits. One basic search brings back full text for every result. But it brings back text for all results, and max_results defaults to 10, so the context fills far faster than with three pages.

Firecrawl and SerpAPI look expensive because their entry tiers are expensive. Firecrawl Hobby works out to $3.80 per 1,000 credits, and one task is 5 credits. SerpAPI Starter is $0.025 per search, and one task is 4 searches. On their top public tiers, the gap gets much smaller. Firecrawl Scale is $749 / 1,000,000 = $0.000749 per credit, so a 5-credit task is $0.003745, or $3.75 per 1,000 tasks. SerpAPI Big Data is $275 / 30,000 = $0.00917 per search, so a 4-search task is $0.0367, or $36.67 per 1,000 tasks.

Our deep call is the cheapest row, but it pays in time. Our call took 16.1 seconds, more than 9 times the median web call. That is too slow for a chat agent that must reply in a few seconds. It is fine for a research job that runs at night, where the savings matter more.

We also tested read. At 00:16:44 SGT it read a ranking post on the Firecrawl blog and returned 30,957 characters of Markdown in 2.35 seconds. At 00:21:53 SGT we gave it a redirect link from a search result. It followed the link to serpapi.com/pricing and returned 3,450 characters in 1.05 seconds, with all six SerpAPI plans in the text.

Price per 1,000 searches: entry tier and lowest tier

If the agent only searches and never reads pages, the ranking changes. This section lists every public price we used, with its source and tier.

Horizontal bar chart on a log scale with steps of 10 times. Title: price per 1,000 searches. Each vendor has two bars, the entry paid tier and the lowest public tier. EveryInfra is $0.694 for both, in teal. Serper $1.00 and $0.300. Bright Data $1.50 and $1.30. Firecrawl $7.60 and $1.50. Tavily $8.00 and $5.00. SerpAPI $25.00 and $9.17.
APITierPrice and volumePer 1,000 searchesValidity or billing periodSource (read 2026-11-07)
EveryInfraNo tiersweb $0.000694 per call$0.694Top-up balance does not expirePricing, tool list
SerperStarter$50 for 50,000, 50 per second$1.006 monthsSerper home page
SerperStandard$375 for 500,000, 100 per second$0.756 monthsSame
SerperScale$1,250 for 2.5 million, 200 per second$0.506 monthsSame
SerperUltimate$3,750 for 12.5 million, 300 per second$0.306 monthsSame
TavilyPay as you go$0.008 per credit$8.00MonthlyTavily pricing
TavilyProject$30 a month, 4,000 credits$7.50Resets on the 1stSame
TavilyLowest monthly$0.005 per credit$5.00MonthlyCredits docs
SerpAPIStarter$25 a month, 1,000 searches$25.00MonthlySerpAPI pricing
SerpAPIDeveloper$75 a month, 5,000 searches$15.00MonthlySame
SerpAPIProduction$150 a month, 15,000 searches$10.00MonthlySame
SerpAPIBig Data$275 a month, 30,000 searches$9.17MonthlySame
FirecrawlHobby$19 a month, 5,000 credits$7.60Monthly, no rolloverFirecrawl pricing
FirecrawlStandard$99 a month, 100,000 credits$1.98Monthly, no rolloverSame
FirecrawlScale$749 a month, 1,000,000 credits$1.501 month rolloverSame
Bright DataPay as you go$1.5 per 1,000$1.50Usage basedSERP pricing
Bright DataScale380K included, $1.3 per 1,000 after$1.30 (overage)MonthlySame

Three points need a clear note. First, Serper's lowest tier really is cheaper than us: $0.30 against $0.694, or 2.3 times less. Even its Standard tier at $0.75 is only 8% above us. If you search several hundred thousand times a month and use the pack within 6 months, Serper is the cheapest row in this table. Second, you must convert Firecrawl credits before you compare.

One credit is not one search, because search costs 2 credits per 10 results. So the 5,000 Hobby credits buy 2,500 searches, and the pricing page says "2,500 searches or 5,000 pages scraped". Third, Bright Data shows the Scale monthly fee on a slider. When we read the page, it showed "380K requests included" and the $1.3 overage, but no monthly fee. So the table shows only the overage price.

Why do we have no tiers? EveryInfra takes each call from a prepaid balance. A $10 top-up and a $1,000 top-up get the same unit price, and the balance does not expire, as the pricing page says. This helps small agents and agents with uneven traffic, because you never prepay for a pack you cannot use. It hurts large, steady teams, because more volume does not lower the price unless you ask for a custom deal.

Monthly cost by volume: the lines cross near 300,000 searches

A unit price table cannot tell you which vendor to pick. Each vendor has different tier steps, expiry rules and free usage. So we priced search-only usage at four monthly volumes: 1,000, 10,000, 100,000 and 1 million searches. The rules are below the chart.

Line chart with a log scale on the vertical axis. The horizontal axis is monthly searches: 1,000, 10,000, 100,000 and 1 million. The EveryInfra teal solid line rises from $0.69 to $694. The Serper dark gray solid line rises from $8.33 to $542. Bright Data dashed line rises from 0 inside the free tier to $1,493. Firecrawl dashed line rises from $19 to $1,749. Tavily dashed line rises from 0 to $7,992. SerpAPI has only two points, $25 and $150, because plans above 30,000 searches have no public price. The EveryInfra and Serper lines come close near 100,000, and at 1 million Serper is lower.
Monthly searchesEveryInfraSerperBright DataFirecrawlTavilySerpAPI
1,000$0.69$8.33$0 (inside free 5,000)$19$0 (inside free 1,000)$25
10,000$6.94$16.67$7.50$94$72$150
100,000$69.40$79.17$142.50$349$792No public price
1,000,000$694.00$541.67$1,492.50$1,749$7,992No public price

Here is how we built each column, one row per vendor, so you can check the math. All rows count search only, with 10 results per search. Free usage comes off first where it applies.

VendorFree usagePlan choiceAbove the plan
EveryInfraNone countedNo tiers$0.000694 per call
SerperNone counted (the 2,500 queries are one-time)Take six months of usage, find the cheapest mix of the four prepaid packs, divide by 6Buy one more pack. Example: 10,000 a month is 60,000 in six months, so two Starter packs for $100, or $16.67 a month
Bright Data5,000 requests a month come off first. The MCP repo says paid usage starts after the free requests if you have fundsPay as you go$1.5 per 1,000
FirecrawlThe free plan gives 1,000 credits a month, so up to 500 searches cost $0. The free plan has no top-ups, so above that you need a paid plan, and paid plans do not add the free 1,000 (billing docs)2 credits per search. Pick the monthly plan with the lowest fee plus top-ups$5 per top-up: 1,000 credits on Hobby, 2,000 on Standard, 2,500 on Growth, 5,000 on Scale (pricing FAQ). Example: 10,000 a month is 20,000 credits, so Hobby at $19 with 5,000, plus 15 top-ups for $75, is $94
Tavily1,000 credits a month come off first. The page does not say if they stack with pay as you go; we assume they doPay as you go. Monthly plans go down to $0.005, but only the Project plan has a public volume$0.008 per credit. At 1 million, the lowest monthly price would give $5,000
SerpAPI250 a month, inside each planThe smallest plan that covers the volumeThe largest public plan is 30,000 a month, so larger volumes say "no public price"

With these rules, the lines cross near 300,000 searches a month. At that point we cost $208.20 and Serper costs $208.33. Above it, Serper's large packs win, and at 1 million it is $152.33 cheaper than us. Below 10,000 a month, the three cheapest choices are us and the free tiers of Bright Data and Tavily.

Keep one condition in mind. The Serper numbers assume you use each pack within 6 months. Unused queries expire, and that raises your real unit price. Our balance does not expire, so this risk does not apply.

Freshness: time filters, date fields and caching

A common agent error is to treat an old result as today's fact. So the third factor is whether you can ask for recent results only, and whether each result carries a date.

The six APIs differ a lot here. The Serper home page says that each call goes to the search engine for fresh results, in its words "We do not cache anything". SerpAPI does the opposite. It caches, and cached searches do not count toward your plan (SerpAPI pricing FAQ). That saves money when an agent asks the same question again, but a cached result can be older than a live one. The FAQ does not say how long the cache lasts.

Tavily added a max_age_hours parameter in its October 2026 changelog. It sets the maximum age of Tavily's cached copy of each page, from 1 to 720 hours. A value of 0 means a live fetch every time (Tavily changelog). A lower value costs no extra credits, but live fetches are slower. On the free plan, the docs say a live fetch under high load can fail with a 434 status (Tavily rate limits).

The search endpoint also has time_range, start_date and end_date filters. Results with no known date stay in by default. To drop them, add filter_by_published_date. To see dates at all, add include_published_date, and the docs call that date a best estimate.

Firecrawl filters by time with the tbs parameter. Its docs say the filter applies to web results only, not to news or image results. Our web tool uses two date parameters, after and before, and you can combine them with site and location.

We tested how after behaves. At 00:16:34 SGT we searched "MCP server web search agents" with after set to 2026-10-31. The call returned 10 results in 2.50 seconds, and 9 had a date. But most dates were relative, such as "1 day ago" or "5 days ago", so the agent must convert them.

Other calls in the same batch returned dates like "Oct 21, 2026". There is a bigger issue too. Seven of the 10 links were search engine redirect links, not the real page address.

Another search at 00:21:08 SGT, with no date filter, had 6 redirect links in 10 results. Across the 8 web calls with results, 13 of 75 results were redirect links, or 17%. The read tool follows redirects for you. At 00:21:53 SGT we gave it one redirect link, and it reached serpapi.com/pricing and returned the page text. If your agent shows links to users, read each link once or resolve the redirect first.

APITime filterDates in resultsCachingSource
EveryInfraafter, before39 of 75 had a date, mixed formatsNot statedOur live calls, tool list
SerperNot listed on home pageExample shows datePage says no cachingSerper home page
Tavilytime_range, start_date, end_dateOnly with include_published_date, an estimatemax_age_hours sets cache ageSearch endpoint
SerpAPIEngine parametersWhen the results page shows oneCaches, cache hits are freePricing FAQ
Firecrawltbs, web results onlyNot statedSearch caching not statedSearch docs
Bright DataEngine parametersWhen the results page shows oneNot statedSERP docs

On freshness, Tavily gives the finest control and Serper's "no caching" is the most direct. Our date field covers about half the results in mixed formats, and we admit that this is a weak point. For time-critical work such as news, prices or outage notices, use a news search. Our data API has a google_news/search capability for this. Then use read on the source page to confirm the date.

Rate limits and latency: is 60 requests per minute enough?

An agent can send many searches for one answer. A research agent may send 5 to 10 queries in parallel in one turn. With 20 users at once, that is several hundred requests a minute. So check the rate limit first. If you get it wrong, the agent does not just slow down. It gets 429 errors.

Horizontal bar chart on a log scale. Title: requests per minute on the entry paid tier. Serper 3,000 per minute, Tavily 1,000, Firecrawl 100, EveryInfra 60 in teal, SerpAPI 3.3. A subtitle says Bright Data states unlimited concurrency, so it has no bar.

The entry tiers differ by three orders of magnitude. Serper allows 50 queries per second on Starter and 300 on Ultimate (Serper home page). Tavily gives a development key 100 requests a minute and a production key 1,000 (Tavily rate limits). Firecrawl /search allows 10 a minute on Free, 100 on Hobby and 500 on Standard (Firecrawl rate limits). SerpAPI sets an hourly cap: 50 on Free, 200 on Starter and 6,000 on Big Data (SerpAPI pricing). Bright Data says its SERP concurrency is unlimited (SERP pricing).

Our default is low in this group. Each account gets 60 requests a minute, shared by all its keys. A 30-day boost gives 180 a minute for $10, 600 for $30 or 1,800 for $78. Total top-ups of $50, $150 or $750 unlock the same levels (pricing). Our test key had its own limit of 10 a minute, so we hit 429 at 00:17:55, 00:19:43 and 00:19:51 SGT.

Each 429 gave a retry_after value of 5 or 8 seconds and the words "Nothing was charged". All three cost $0. After a 60-second wait, the calls at 00:19:01 and 00:21:08 SGT both worked. This kind of 429 is easy for an agent to handle, but it does not raise a low limit. For several hundred searches a minute, the production tiers of Serper and Tavily start higher than ours.

Scatter chart. The horizontal axis is seconds from 0 to 18. Title: end-to-end time of our 20 live calls. Eight web calls are teal dots between 1.34 and 2.70 seconds, with a dashed line at the 1.74-second median. Two web/search calls sit at 1.59 and 6.02 seconds. Two read calls sit at 1.05 and 2.35 seconds. One deep call with full text sits at 16.11 seconds. One crosscheck call sits at 5.36 seconds. One web call through MCP sits at 3.96 seconds. Five 401, 422 and 429 calls all sit below 1.2 seconds.
Our live calls (2026-11-07, SGT)CallsEnd-to-end timeResults per callCharge per call
web81.34 to 2.70 s, median 1.74 s0 to 10$0.000694 ($0 for the 1 empty result)
web/search (data API)26.02 s and 1.59 s10$0.001389
read22.35 s and 1.05 s1 page, 30,957 and 3,450 characters$0.000694
deep with full=true116.11 s5 with full text, 47,596 characters$0.000694
crosscheck15.36 s3 methods with 5 each$0.002083
MCP tools/call to web13.96 s5$0.000694
401 / 422 / 42950.11 to 1.16 s0$0

We did not time the other vendors, so we quote their pages. Serper says 1 to 2 seconds, or 2 to 4 with retries. SerpAPI sells Ludicrous Speed at twice the price. Its page claims 2.2 times faster on average and 4.2 times faster at p99 (Ludicrous Speed). Bright Data says "under 5 sec" on its pricing FAQ and "under 1 second per request" in its docs. Tavily and Firecrawl give no seconds.

Our web median of 1.74 seconds sits inside Serper's claim and meets our target of a median under 3 seconds. The first web/search call took 6.02 seconds, but the same query on web took 1.78 seconds. So for plain web results, use the search API, which costs half as much.

The data API path took 6 seconds on one of our two calls. The deep call took 16 seconds, too slow for a path where a user waits. The MCP call took about 2 seconds more than direct HTTP. That is one sample, so test both paths if latency matters to you.

Failures, empty results and rate limits: when do you pay?

Agents fail far more often than people do. The model builds the parameters, and it often gets them wrong. It searches in loops and hits rate limits. It asks narrow questions and finds nothing. Multiply each failure by the agent's call volume, and the billing rule for failures becomes real money.

CaseEveryInfra (our live calls)SerperTavilySerpAPIFirecrawlBright Data
Bad key or bad parameter401 and 422 cost $0, 422 gives the fixOnly successful responses chargedNot statedErrored searches freeNo document, no chargePay for success only
Zero resultsRefunded, empty_result_refundedNot statedNot statedCounts as 1 searchNot statedNot stated
Target page returns 403/404Not stated for readNot applicableFailed extraction freeNot applicable1 creditNot stated
Rate limit 429$0, with retry_afterNot statedReturns Retry-AfterNot statedNot statedNot stated
Partial successcrosscheck charges per method that ranNot applicableExtract charges per successful URLCrawl charges per pageCharges per document returnedNot applicable
SourceOur live callsHome page FAQCredits docsPricing FAQBilling docsSERP docs

Three cells matter most for agent builders. First, SerpAPI counts an empty result as one search. Its FAQ says a response with 100 results and a response with 0 results each count as one search. The narrower your agent's questions, the more this adds up. Second, Firecrawl counts a 403 or 404 page as a document and charges 1 credit, about $0.0038 on Hobby ($19 for 5,000 credits). Its billing docs suggest you check metadata.statusCode and stop retrying URLs that always refuse you.

Third, our 422 does more than cost nothing. It tells the model how to fix the call. At 00:15:57 SGT we sent the query as query on purpose. The response said did_you_mean: ["q"] and listed every valid parameter: after, before, location, num, page, q, site.

The model can fix the call in one try. Our guide on self-healing API errors has more examples. For refund checks, see failure refund reconciliation.

Screenshot of real API responses in two dark code cards. The top card is the cross-check response. Of four methods, keyword, index and semantic show status ok with 5 results each and times of 1410, 5122 and 551 ms. The reasoning method shows status unavailable in a teal box. The independence line says 4 planned, 3 ran, reasoning missing, in a teal box. The usage line shows charged_usd 0.002083 in a teal box. The bottom card is a web search for a string that does not exist. Results are empty. Billing shows charged false, charged_usd 0.0 and reason empty_result_refunded, and the refund shows status succeeded. All four lines have teal boxes.
Screenshot of our real API responses with balance and other internal fields removed. Top at 00:17:25 SGT, bottom at 00:29:19 SGT on 2026-11-07. EveryInfra search API response

At 00:29:19 SGT we searched a long string that does not exist, on purpose. The response was 200 with empty results. It showed billing.charged false and reason empty_result_refunded, and the refund block showed status succeeded, so the $0.000694 for that call came back. This tells you how billing works for agents. We charge first and refund after, so the balance drops for a moment and then goes back up. Your reconciliation script should read the final charged_usd, not the balance change at the time of the request.

MCP: how each of the six connects to an agent

MCP (Model Context Protocol) is now the common way to connect tools to an agent. The latest specification version is dated 2026-07-28 (MCP versioning, specification releases). Five of the six vendors run a remote MCP server of their own. The setup differs a lot, and the biggest difference is where the key goes.

APIRemote URLAuthenticationToolsLatest updateSource
EveryInfrahttps://api.everyinfra.com/mcpAuthorization: Bearer header; no key needed to list tools12, search through everyinfra_search (17 tools)Read 2026-11-07MCP server page
Tavilyhttps://mcp.tavily.com/mcp/OAuth, key in URL, or keyless headersearch, extract, crawl, map, researchREADME updated 2026-11-05Tavily MCP docs
SerpAPIhttps://mcp.serpapi.com/mcpBearer header (recommended) or key in pathSearch, Extract and morev2.2.0 on 2026-11-04SerpAPI MCP repo
Firecrawlhttps://mcp.firecrawl.dev/v2/mcpKeyless endpoint opens 3 tools; a key opens all27 in total, 9 on the search-only endpointv3.36.1 on 2026-11-03Firecrawl MCP repo
Bright Datahttps://mcp.brightdata.com/mcp?token=…Token in the URL30v4.6.0 on 2026-11-05Bright Data MCP repo
SerperNot listed on home page—Home page lists LangChain, CrewAI, Haystack and other framework integrations—Serper home page

Where the key goes has a real effect on agents. A URL ends up in client logs, proxy logs and browser history. A model can even repeat it word for word in an answer. The SerpAPI README lists both methods and recommends the header, because "the key stays out of URLs and logs".

The Bright Data remote URL takes the token only in the URL. Tavily recommends OAuth, and clients that accept only a URL must put the key in ?tavilyApiKey=. We accept only a Bearer header.

Tavily and Firecrawl offer keyless trials, and we do not. With the X-Tavily-Access-Mode: keyless header, a Tavily request gets free, rate-limited search and extract. The response shape stays the same (keyless docs). The Firecrawl remote MCP opens three tools without a key, search, scrape and parse, with daily limits per IP (rate limits).

Our MCP lists tools without a key. At 00:17:48 SGT, tools/list returned 12 tools with full parameter docs in 0.10 seconds. A model can see every tool first, but a real call needs a key.

To add our MCP server to Claude Code, run one command. Keep the key in an environment variable, not in a config file:

bash
# Keep the key in an environment variable; run /mcp in Claude Code to see the tools
claude mcp add --transport http everyinfra https://api.everyinfra.com/mcp \
  --header "Authorization: Bearer ${EVERYINFRA_API_KEY}"

If your client reads a JSON config, you can add two search servers and let the agent pick one per task. In this example, the Tavily entry uses OAuth as its docs describe, so it has no key:

json
{
  "mcpServers": {
    "everyinfra": {
      "url": "https://api.everyinfra.com/mcp",
      "headers": { "Authorization": "Bearer ${EVERYINFRA_API_KEY}" }
    },
    "tavily": { "url": "https://mcp.tavily.com/mcp/" }
  }
}

For each client's full setup, see our MCP server page and the remote MCP setup guide. For what the 2026 specification means in production, see MCP 2026 spec production readiness.

One real round trip: what an agent gets from everyinfra_search

Here is one full MCP round trip from our test. The agent wants to know how many credits one Tavily search costs. The first call at 00:17:55 SGT hit the test key limit of 10 a minute. It returned isError: true, rate_limited, retry_after: 5 and "Nothing was charged".

We waited 60 seconds. The second call at 00:19:01 SGT returned 5 results in 3.96 seconds for $0.000694. Here is the call and the result, with balance fields removed and snippets unchanged:

json
// Request: tools/call
{"name": "everyinfra_search",
 "arguments": {"tool": "web", "params": {"q": "tavily search api credits per request", "num": 5}}}

// Response (excerpt)
{"tool": "web",
 "results": [
  {"title": "Credits & Pricing",
   "url": "https://docs.tavily.com/documentation/api-credits",
   "snippet": "You get 1,000 free API Credits every month. No credit card required. Monthly plans: $0.0075 - $0.005 per credit. Each request costs 1 API credit. Advanced ..."},
  {"title": "Tavily Search",
   "url": "https://docs.tavily.com/documentation/api-reference/endpoint/search",
   "snippet": "This uses 2 API credits per request. To avoid the extra cost, you can explicitly set search_depth to basic ..."}
 ],
 "billing": {"charged": true, "charged_usd": 0.000694}}

This round trip shows two things. The good part is that the first two results are Tavily's own docs. The snippets already hold the answer (basic is 1 credit, advanced is 2), so the model can answer without a page read, for $0.000694. The risky part is results 4 and 5, which come from review sites. One says "$0.008 a credit" and another repeats an older Tavily claim. A model that reads only snippets can take a third-party summary as the source.

Our advice is a simple rule for agents. For prices, quotas and other numbers that change, trust only results from the vendor's own domain. Or call read once on the source page to confirm. The cross-check tool in the next section exists to solve this problem.

Cross-checking: tell the agent which claims not to trust

The biggest risk in a search API for AI agents is not "no results". It is results that are wrong, old or distorted by retelling, which a model has trouble spotting on its own. Our crosscheck tool runs four search methods on one question in parallel: keyword, index, semantic and reasoning. It counts how many methods find each source and fetches the text of the top sources. It does not write a conclusion. The tool list says it gives evidence, not conclusions.

We tested a claim that this article also needs: "Serper free tier gives 2,500 queries". The call at 00:17:25 SGT took 5.36 seconds in total.

MethodStatusResultsTime
keywordok51,410 ms
indexok55,122 ms
semanticok5551 ms
reasoningunavailable0—

The verdict level was scattered. No source was found by 3 or more methods, so the agent should not trust any single source. Two third-party pages were found by both keyword and index, and the fetched text called the 2,500 queries "one-time". The Serper home page agrees: it says "Get 2,500 free queries" and does not say "per month". Our Serper free tier post checks this claim word by word.

The list price is $0.002778. The reasoning method did not run, so we charged only for the 3 methods that ran: $0.002083. Tavily's research endpoint and Firecrawl's agent endpoint also search in several rounds. But they write an answer and bill by usage (Tavily credits docs, Firecrawl billing docs). Use theirs for a finished answer with citations. Use ours when your own model must judge the sources.

When another API fits better: the best use for each

Every section above has a factor where we lose. This table pulls them together, and the notes below add the details.

Your agent needsPickMain cost of that choice
Over 300,000 searches a month, results page onlySerperPrepaid packs that expire in 6 months, no page text
Text the model can read now, a keyless trialTavily$8.00 per 1,000 on pay as you go
Many engines, full results pages, vendor carries legal riskSerpAPI$25.00 per 1,000 on Starter, empty results count
Search and scrape in one call, special indexesFirecrawlComplex credits, 403 and 404 pages cost a credit
City targeting, very high concurrencyBright DataToken in the URL, raw HTML by default
Small start, free failures, other data on one keyEveryInfra60 per minute by default, short snippets only

Serper is the cheapest search layer at high volume. At 1 million searches a month it costs $152.33 less than we do, and it does not cache. If you already have a page reader, start here. Our posts on Serper API alternatives and Serper API pricing go deeper.

Tavily picks chunks of about 500 characters by relevance. One basic search for 1 credit can bring full text and a short answer. The keyless mode lets an agent try it before you sign up. Its price is 11.5 times ours, but its production key allows 1,000 requests a minute.

SerpAPI covers more than 100 engines, and Markdown output cuts tokens by about half. Production and higher plans include the U.S. Legal Shield. The page says SerpAPI covers up to $2 million of U.S. legal liability for lawful scraping of public search data (U.S. Legal Shield).

Firecrawl fetches full text when you add scrapeOptions to a search. Its government and academic indexes are free to query, and Alexandria adds licensed sources. Its docs say the research category changes on 2026-11-16, and results move from data.web to data.research. Update any parser that reads it before that date.

Bright Data covers 195 countries with city-level targeting. The page says concurrency is unlimited, at $1.50 per 1,000 and 5,000 free requests a month. Our Bright Data alternatives page compares more.

EveryInfra costs $0.694 per 1,000 with no expiry and no monthly fee. A 401, 422, 429 or empty result costs nothing, and cross-check shows which claims not to trust. The same key calls 106 platforms and 573 capabilities in our data API, as our MCP tool description says. Examples are bing/search, duckduckgo/search and google_news/search.

What we cannot do

We want to be clear about our gaps. Each row below is a real limit of our search API today, and the last column says what to use when it matters to you.

GapDetailUse instead
No full results pageweb returns title, link, snippet and date onlySerper or SerpAPI
Country-level location onlyA city counts as its countryBright Data
10 results per pageEach extra page is one more chargeTavily returns up to 20
Dates on about half the resultsMixed formats, 17% of links are redirectsread, or a news search
60 requests a minute by defaultSecond lowest of the six entry tiersSerper or Tavily production keys, or our boost
No keyless trialSign up first; starring our two GitHub repos gives $1 for 24 hoursTavily or Firecrawl keyless modes
No written answersYour own model writes the conclusionTavily include_answer
Slow full-text searchdeep took 16 seconds; one cross-check method failedweb plus read

What changed in October and November 2026

Search APIs changed fast in the last two months. Bright Data changed its MCP tools and parameters four times in one week. If your agent prompt names tools directly, pin the version or read the tool list again at each start.

DateVendorChangeWhat it means for you
2026-11-05Bright DataMCP v4.6.0 removed scrape_as_html and extract (v4.6.0)Use scrape_as_markdown with format: 'html'
2026-11-04Bright DataMCP v4.4.0 moved all tools to one error format (v4.4.0)Update error handling in your agent
2026-11-04SerpAPIMCP v2.2.0 added an extract tool (v2.2.0)One server can search and read, at one search per page
2026-11-03FirecrawlMCP v3.36.1 released (release notes)The research category changes on 2026-11-16
2026-11-02Bright DataMCP v4.3.0 moved search to the SERP API, renamed geo_location to country (v4.3.0)Old parameter names return invalid_input
2026-11-01Bright DataMCP v4.2.0 moved scraping to an API that does not run JavaScript (v4.2.0)Pages that need JavaScript may return less text
2026-10-13Bright DataPartnership with DigitalOcean for hosted agents (announcement)Compare the total bill, not only the unit price
October 2026TavilyNew max_age_hours parameter (changelog)You can force a live fetch per request
Early November 2026FirecrawlBenzinga data in Alexandria (PR Newswire)Finance agents get licensed data

Since SerpAPI MCP v2.0.0 on 2026-09-22, a failed search returns isError: true (v2.0.0).

Two older changes still shape these choices. Google's Custom Search JSON API no longer accepts new customers, and it stops on January 1, 2027 (Google for Developers). Microsoft retired the Bing Search APIs on August 11, 2025 (Microsoft Learn). It now points users to grounding with Bing Search in Azure AI Agents. So neither Google nor Bing now offers its official general search API to new users. A new agent that needs search results must use a third-party API.

How this article differs from our other search posts

We have other articles on search, and this one does not repeat them. Agent search tool selection explains how one agent switches between search, page reads and cross-checks. It has no price comparison. The Serper free tier post covers only how long Serper's 2,500 free queries last.

Our Google search results scraper price comparison compares the cost of 1,000 Google organic result pages, for rank tracking teams. This article is the only one that compares six APIs on eight agent factors and prices the "search once, read three pages" task. For one-to-one comparisons, see our pages on SerpAPI alternatives, Firecrawl alternatives and Bright Data alternatives.

Get started: connect search to your agent in three steps

Step one is to see the output before you write code. Open the market page, find web search, and try a query. You can also read the web search product page and the search product overview, which list the 17 tools and three price levels. Step two is to follow the quickstart to sign up and get a key. A star on two GitHub repos gives $1 of trial credit for 24 hours, which is about 1,440 web calls. Step three is to put this curl call in your script, or to add the MCP server with the one-line command from the MCP section.

bash
curl -s https://api.everyinfra.com/api/v1/search \
  -H "Authorization: Bearer $EVERYINFRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"tool":"web","params":{"q":"best web search api for ai agents","location":"us","num":10}}'
# Returns results (up to 10: title, url, snippet, published), location, billing.charged_usd
# To read one result: set tool to "read" and params to {"url": "<url from results>"}

The free search tool list is the reference for parameters. A wrong name returns 422 with the list of valid ones. For the matching capabilities in our data API, see the capability catalog. Are you choosing a data layer for the whole agent, not only search? Then also look at the Bing data API, DuckDuckGo data API and Google News data API pages.

FAQ

Is there a free web search API for AI agents?

Yes, but each one has a cap. Bright Data gives 5,000 free requests a month. Tavily gives 1,000 credits a month, worth about $8 at its pay-as-you-go price, plus a keyless mode. Firecrawl gives 1,000 credits a month, about 500 searches, plus a keyless endpoint.

Serper gives 2,500 queries at sign-up, and its page does not say they reset. SerpAPI gives 250 searches a month, and we give $1 for starring our two GitHub repos, about 1,440 searches in 24 hours. (Question source: Google autocomplete for "free web search api for ai agents", read 2026-11-07.)

Is there a Google search API for AI agents from Google itself?

Not for new projects. Google's Custom Search JSON API no longer accepts new customers. Existing customers get 100 free queries a day, then $5 per 1,000 up to 10,000 a day. The API stops on January 1, 2027 (Google for Developers).

For Google-style results pages, use Serper, SerpAPI or Bright Data. For general web results, our web tool costs $0.694 per 1,000. (Question source: autocomplete for "google search api for ai agents".)

Tavily vs Serper: which should I pick?

It depends on whether your agent needs text or a results page. Tavily returns chunks chosen by relevance, with optional full text, at $8.00 per 1,000 on pay as you go. Serper returns structured JSON at $1.00 per 1,000 on entry and $0.30 at the lowest tier. But Serper packs are prepaid and expire in 6 months.

Pick Tavily for low volume where the agent must read content. Pick Serper for high volume when you already have a page reader. (Question source: autocomplete for "tavily vs serper".)

What is the difference between Firecrawl and Tavily?

Both can search and read, but they bill differently. A Tavily basic search costs 1 credit, and full text adds nothing. Firecrawl charges 2 credits per 10 results, plus 1 credit for each scraped page. On entry tiers, "search once, read three pages" costs $8.00 per 1,000 tasks on Tavily and $19.00 on Firecrawl. Firecrawl is stronger at scraping itself, with screenshots, HTML, JSON and free academic indexes. (Question source: autocomplete for "firecrawl vs tavily".)

SerpAPI vs Serper: which is cheaper?

Serper is much cheaper. SerpAPI Starter costs $25.00 per 1,000 searches, and its Big Data plan costs $9.17. Serper starts at $1.00 and goes down to $0.30. SerpAPI charges more for over 100 engines, Markdown output and the U.S.

Legal Shield. One user in r/AI_Agents called it costly at "$75/month for just 5k searches" (original thread). (Question source: autocomplete for "serpapi vs serper".)

What is Bright Data SERP API pricing?

The SERP API costs $1.5 per 1,000 on pay as you go, for successful requests only. The Scale plan includes 380,000 requests and charges $1.3 per 1,000 after that (SERP pricing). The MCP server gives 5,000 free requests a month and charges $1.50 per 1,000 results (MCP pricing). In the same Reddit thread, one user said they pay "$1.5 for 1000 searches". (Question source: autocomplete for "bright data serp api pricing".)

How do I compare the accuracy of search APIs for AI agents?

No third-party ranking is accepted by all vendors, so test with your own questions. Pick 50 to 100 questions that your agent will really ask and that have clear answers. Run them on each API and judge the answers by hand. SimpleQA is a public benchmark of short questions with one correct answer (paper), and its question design is a good model. (Question source: a r/LocalLLaMA thread titled "What is the most accurate web search API for LLM?", found with a forum search on 2026-11-07.)

References

  1. Serper home page (pricing, free tier, FAQ), Serper, read 2026-11-07.
  2. Serper Terms of Service, Serper, updated 2024-05-29, read 2026-11-07.
  3. Tavily Pricing, Tavily, read 2026-11-07.
  4. Tavily Credits & Pricing, Tavily docs, read 2026-11-07.
  5. Tavily Rate Limits, Tavily docs, read 2026-11-07.
  6. Tavily Search endpoint reference, Tavily docs, read 2026-11-07.
  7. Tavily Changelog, Tavily docs, October 2026 entries, read 2026-11-07.
  8. Tavily MCP, Tavily docs, read 2026-11-07.
  9. Try Tavily Without an API Key, Tavily docs, read 2026-11-07.
  10. tavily-ai/tavily-mcp, GitHub, latest commit 2026-11-05.
  11. SerpApi Plans and Pricing, SerpApi, read 2026-11-07.
  12. SerpApi Markdown Output for AI Agents, SerpApi, read 2026-11-07.
  13. SerpApi MCP Integration, SerpApi, read 2026-11-07.
  14. SerpApi Ludicrous Speed, SerpApi, read 2026-11-07.
  15. SerpApi Extract API, SerpApi, read 2026-11-07.
  16. SerpApi U.S. Legal Shield, SerpApi, read 2026-11-07.
  17. serpapi/serpapi-mcp, GitHub, read 2026-11-07.
  18. serpapi-mcp v2.2.0, GitHub, 2026-11-04.
  19. serpapi-mcp v2.0.0, GitHub, 2026-09-22.
  20. Firecrawl Pricing, Firecrawl, effective 2026-09-04, read 2026-11-07.
  21. Firecrawl Billing, Firecrawl docs, read 2026-11-07.
  22. Firecrawl Search, Firecrawl docs, read 2026-11-07.
  23. Firecrawl Rate Limits, Firecrawl docs, read 2026-11-07.
  24. firecrawl/firecrawl-mcp-server, GitHub, read 2026-11-07.
  25. firecrawl-mcp-server v3.36.1, GitHub, 2026-11-03.
  26. Best Search Tools for AI Agents in 2026, Firecrawl blog, 2026-09-24, read 2026-11-07.
  27. Bright Data SERP API Pricing, Bright Data, read 2026-11-07.
  28. Bright Data MCP Server Pricing, Bright Data, read 2026-11-07.
  29. Introduction to SERP API, Bright Data docs, read 2026-11-07.
  30. brightdata/brightdata-mcp, GitHub, read 2026-11-07.
  31. brightdata-mcp v4.6.0, GitHub, 2026-11-05.
  32. brightdata-mcp v4.4.0, GitHub, 2026-11-04.
  33. brightdata-mcp v4.3.0, GitHub, 2026-11-02.
  34. brightdata-mcp v4.2.0, GitHub, 2026-11-01.
  35. DigitalOcean and Bright Data Unlock the Web for AI Agents, Business Wire release on Yahoo Finance, 2026-10-13.
  36. Benzinga Collaborates With Firecrawl, PR Newswire, early November 2026.
  37. Custom Search JSON API, Google for Developers, updated 2026-02-18.
  38. Bing Search APIs Retiring on August 11, 2025, Microsoft Learn.
  39. MCP Versioning, Model Context Protocol, 2026-07-28 version.
  40. modelcontextprotocol specification releases, GitHub, 2026-07-28 release.
  41. Measuring short-form factuality in large language models (SimpleQA), arXiv, 2024-11-07.
  42. What's the best tool/API for web search in an agentic stack?, Reddit r/AI_Agents, read 2026-11-07.
  43. EveryInfra search tool list, EveryInfra, read 2026-11-07 00:15 SGT.
  44. EveryInfra capability catalog, EveryInfra, read 2026-11-07 00:15 SGT.
  45. EveryInfra OpenAPI, EveryInfra, read 2026-11-07.
  46. everyinfra/everyinfra-agent-plugins, GitHub, one of the repos for the trial star.

Reference BL-105

Our own visit stats are always on. Google Analytics loads only if you choose “Allow analytics”. Privacy policy