Which web scraping tool should your AI agent use? 7 options compared
Your agent is only as good as what it can read. Raw HTML wastes tokens, JavaScript-heavy pages come back empty, and some sites block anything that looks like a bot. The tools below fix this in different ways: some turn one URL into clean Markdown, some crawl a whole site, some search the web for you, and some drive a real browser that can click and log in. Choosing well mostly means choosing the right kind of tool. We checked every price and free tier on the vendor's own pages on October 22, 2026, and every tool here has an MCP server, so you can try it from your coding agent today.
Some links are affiliate links: if you sign up, Hotfix Daily may earn a commission at no extra cost to you. It never affects what we recommend.
Before you compare prices, decide what your agent actually needs to do:
Read one page you already have the URL for. You want a URL-to-Markdown call. Jina Reader, Firecrawl /scrape, Crawl4AI and Exa Contents all do this.
Crawl a whole site or docs section. You want a crawler that follows links, respects limits and returns every page. Firecrawl and Crawl4AI are built for this, and Tavily has a lighter crawl.
Find pages you don't know yet. That's a search API, not a scraper. Exa and Tavily lead here, and Firecrawl, Jina, Crawl4AI Cloud and Browserbase also offer search.
Act on a page (log in, click through a form, scroll an app). You need a real browser. That's Browserbase with Stagehand, or Playwright on your own machine.
Pull structured data from one popular site (Google Maps, Amazon, Instagram). A site-specific scraper usually beats a general one. That's Apify's Store.
Most agent stacks end up with two of these: a search API plus a reader or crawler. Need a model to process what you fetch? Estimate the bill with our AI API cost calculator.
Tool by tool
Prices are in US dollars, from each vendor's own page on October 22, 2026.
Type · free tier: Hosted scrape, crawl, map and search API · 1,000 credits a month, no card required
What it is: an API that turns websites into LLM-ready data. Endpoints include /scrape (one page), /crawl (a whole site), /map (list a site's URLs), /search (web search, optionally scraping each result), /interact (a browser session) and an Agent endpoint in preview. The core is open source under AGPL-3.0, with more than 190,000 stars on GitHub.
For agents: returns Markdown or JSON and renders JavaScript. The hosted MCP server offers a keyless option ("Search, Scrape, and Parse within daily limits"), browser sign-in, or an API key.
Pricing: credits are shared between scraping and search. Scrape and crawl cost 1 credit per page, and map costs 1 credit per call. Search costs 2 credits per 10 results. JSON output adds 4 credits per page.
Free: 1,000 credits a month, 2 concurrent requests, 10 requests a minute for scrape, map and search, and 2 a minute for crawl.
Pay-as-you-go (paid plans only) adds credits in $5 steps when you run out, up to a monthly limit you set. Credits only roll over on Scale and Enterprise.
Downsides: a page that returns an error such as 403 or 404 still costs a credit. JSON output costs 4 extra credits per page, five times the base price, so let your own model structure the Markdown when you can. Self-hosting is real but limited: Firecrawl's own docs say the open-source stack doesn't include its Fire-engine anti-bot service, screenshots or page actions, and some features (Agent, Browser, interact) stay cloud-only. AGPL also matters if you plan to modify it and offer it as a service.
Our take: The best all-round hosted option. One API covers reading, crawling and search, the free tier needs no card, and the MCP server works in minutes. Start here unless you need free self-hosting (Crawl4AI) or a real interactive browser (Browserbase).
Type · free tier: Open-source Python crawler (Apache-2.0) · library free; cloud 1,000 credits a month, no card
What it is: an open-source crawler and scraper that turns websites into clean Markdown for RAG and agents. You run it as a Python library or as a Docker server with a monitoring dashboard and playground. The latest release is v0.9.5 (October 14, 2026). The team also runs Crawl4AI Cloud, a hosted API with /scrape, /extract, /search, /serp and /answer.
For agents: Markdown output, structured extraction with an LLM, and browser rendering through Playwright. MCP is available on your own Docker server, and the cloud has a hosted MCP endpoint (https://api.crawl4ai.com/mcp).
Pricing: the library and self-hosted server are free; you pay for your own machines and proxies.
Cloud Free: 1,000 credits a month, 10 requests a minute, 2 at a time, no card.
Starter: $10/month for 10,000 credits and 120 requests a minute.
Builder: $99/month for 125,000 credits.
Scale: $499/month for 750,000 credits.
Cost depends on how hard the page was: a page rendered in a real browser costs twice a plain fetch, and a page that "fought back" costs four times. A page already in their archive costs half.
Downsides: self-hosting means you run browsers, handle blocking and pay for proxies yourself. It's Python-first. The cloud is labeled "soft launch", and its docs say prices and the free credit can change, so don't build a budget on today's numbers.
Our take: The pick when you want to own the crawler. Apache-2.0 is a friendlier license than Firecrawl's AGPL, and for big docs crawls on your own hardware the cost is just compute. Try the cloud for convenience, but treat it as new.
Type · free tier: URL-to-Markdown and search API · works without an API key at 20 requests a minute
What it is: put https://r.jina.ai/ in front of any URL and you get the page back as clean, LLM-friendly text. s.jina.ai searches the web and returns the top five results with their content. It also reads PDFs and can caption images. The Reader service code is on GitHub under Apache-2.0.
For agents: the simplest possible integration, a GET request. The default engine renders the page in a headless browser so JavaScript runs first, and headers let you wait for a selector, strip elements, or set a token budget. There's an MCP server at mcp.jina.ai.
Pricing: billed in tokens, counted on the output. Rate limits for r.jina.ai: 20 a minute with no key, 500 with a free or paid key, 5,000 with a premium key. Search needs a key, and each search costs a fixed number of tokens, starting at 10,000. New keys come with free tokens. Jina's page doesn't show the free-token amount or the token package prices in its text, so check your dashboard before you budget.
Downsides: it reads one URL at a time and doesn't crawl a site for you. Jina says Reader doesn't try to get around anti-bot systems, so sites that block bots stay blocked. On networks it flags as having a poor reputation (common for cloud servers), Jina asks for a key even for basic reads. The ReaderLM-v2 and jina-vlm models it uses are CC-BY-NC 4.0 (non-commercial), and commercial licenses for them are now sold by Elastic.
Our take: The fastest way to give an agent "read this link", with no signup at all. Great for prototypes and light use. Move to a crawler when you need whole sites, and check the pricing in your dashboard before you lean on it in production.
Type · free tier: Scraper marketplace (Apify Store) and cloud platform · $5 of platform usage a month, no card
What it is: a cloud platform for running scrapers called Actors. The Apify Store has ready-made Actors for specific sites and jobs, and you can write your own (Apify also maintains Crawlee, an open-source crawling library under Apache-2.0).
For agents: the hosted MCP server at mcp.apify.com lets an agent search the Store, read an Actor's input schema and pricing, and run it. By default it loads a RAG web browser Actor and a web-fetch Actor that renders JavaScript. Apify deliberately keeps full-permission Actors out of MCP, so an agent can't run those on its own.
Pricing:
Free: $0 with $5 of usage a month. When it runs out, you're blocked until next month.
Starter: $19/month ($17 billed yearly), with $19 of usage included.
Compute is $0.20 per compute unit (1 GB of RAM for one hour) on Free and Starter. Most Store Actors charge per event (for example per result), shown on each Actor's page. Unused usage doesn't roll over.
Downsides: costs are harder to predict than a flat per-page price, because every Actor sets its own pricing and proxies are billed separately (residential proxy is $8 per GB on Free and Starter). Actors are built by different developers, so quality varies.
Our take: The best choice when the job is "get this data from that specific site". A maintained Actor saves you from writing and fixing your own scraper. For general "read any page" work, Firecrawl or Crawl4AI is simpler.
Type · free tier: Hosted headless browsers + open-source AI browser SDK · 1 browser hour, 3 concurrent browsers
What it is: Browserbase runs fleets of headless browsers in the cloud for agents, and also offers Fetch and Search APIs. Stagehand is its open-source SDK (MIT) for TypeScript, Python and Go. It mixes AI steps (act, extract, observe) with normal Playwright-style page commands.
For agents: this is the tool for interacting, not bulk reading: logging in, filling forms, clicking through flows. extract returns data checked against a schema you define. Browserbase publishes an open-source MCP server (mcp-server-browserbase on GitHub).
Pricing:
Free: $0, 3 concurrent browsers, 1 browser hour, 15-minute sessions, 1,000 Search and 1,000 Fetch calls, $5 of model tokens. No captcha solving.
Developer: $20/month, 25 browsers, 100 browser hours then $0.12 an hour, captcha solving, 1 GB of proxies.
Startup: $99/month, 100 browsers, 500 browser hours then $0.10 an hour.
Scale: custom.
Stagehand itself is free and works with Chromium-based browsers, though Browserbase recommends running it on its hosted browsers.
Downsides: one free browser hour runs out fast, so you'll be paying almost as soon as you build anything real. Every AI step also costs model tokens. For pages that just need reading, a browser is slower and pricier than a scrape API.
Our take: The best pick for agents that must act on the web. Use Stagehand's plain page commands where you know the selector and save act for the parts that change, which keeps runs cheaper and easier to debug. For reading pages, use a scraper instead.
Type · free tier: AI search API · $10 of credits every month (plus a one-time $10 onboarding bonus), no card
What it is: a search engine built for AI. It finds pages by meaning and returns their contents ready for an LLM. Endpoints include Search, Deep Search (multi-step research with citations), Contents (full text for URLs you already have), Answer, Monitors and an async Agent API.
For agents: finding sources you don't know about yet, then reading them in the same call. Hosted MCP server at https://mcp.exa.ai/mcp, plus Python and JavaScript SDKs.
Pricing: pay as you go, no subscription or minimum.
Search: $4 per 1,000 requests (lite), $7 per 1,000 for the default auto type, up to 10 results each.
Deep Search: $12 to $15 per 1,000. Contents: $1 per 1,000 pages. Answer: $5 per 1,000.
Free tier: $10 of credits each month (Exa says up to 2,500 lite searches).
Downsides: it's search plus page contents, not a site crawler. Costs add up per piece: results beyond 10, AI summaries and each content type are billed separately.
Our take: The best search API when relevance matters, for example research agents, finding docs or company pages, and "find me examples of X". The monthly $10 reset makes it easy to keep testing.
Type · free tier: Search API with extract, map, crawl and research · 1,000 credits a month, no card
What it is: a search API made for agents, plus Extract (clean content from URLs), Map (a site's structure), Crawl (map plus extract) and Research (multi-source answers).
For agents: a hosted MCP server at https://mcp.tavily.com/mcp/ with OAuth sign-in, and keyless access for Search and Extract if you don't have an account yet. The MCP server is MIT on GitHub.
Pricing:
Basic search costs 1 credit, advanced 2. Extract costs 1 credit per 5 successful URLs (2 on advanced), and failed extractions are free. Crawl is the map cost plus the extract cost.
Free (Researcher): 1,000 credits a month.
Project: $30/month for 4,000 credits. Bootstrap: $100 for 15,000. Startup: $220 for 38,000. Growth: $500 for 100,000.
Pay as you go: $0.008 per credit.
Downsides: Tavily's extract and crawl are lighter than a dedicated crawler, with no browser actions. Monthly plans cost more up front than Exa's pay-as-you-go.
Our take: The easiest "one tool does search and reading" setup, especially through MCP. If your agent mostly searches and sometimes reads a page, Tavily alone may be enough. Choose Exa when search quality on harder queries matters more.
Most builders, hosted, one API:Firecrawl (affiliate). It crawls, scrapes, maps and searches, the free tier needs no card, and its MCP server works without a key.
Free and self-hosted, or big docs crawls: Crawl4AI. Apache-2.0, runs on your own machine, no per-page fee.
"Read this URL" with zero setup: Jina Reader. Put r.jina.ai/ in front of the link.
Data from a specific site (Maps, Amazon, social): Apify Store Actors.
Logging in, clicking, filling forms: Browserbase with Stagehand.
Search by meaning, research agents: Exa.
Search plus light extraction in one MCP server: Tavily.
A cheap starter stack for a research agent: Exa or Tavily to find pages, plus Jina Reader or Firecrawl to read them. All four have free tiers.
Before you scrape
Treat fetched pages as untrusted input. A page can contain instructions aimed at your agent (prompt injection). Keep approvals on for tools that can write or send things. See our MCP setup guide for safety settings.
Respect the site. Check its terms and robots.txt, keep your rate low, and don't scrape behind logins you don't own. Jina, for one, says it doesn't bypass anti-bot systems, and that you're responsible for how you use the output.
Cap your spend. Firecrawl lets you set a monthly pay-as-you-go limit, and Apify's free plan simply stops when its $5 runs out. Check each dashboard's limits before you let an agent loop.
Keep keys out of shared configs. Use OAuth sign-in where the MCP server offers it, or environment variables.
FAQ
What's the best web scraping tool for AI agents?
It depends on the job. Our take: for most builders who want a hosted API, Firecrawl covers reading, crawling and search in one place. For free self-hosting, use Crawl4AI. For agents that must click and log in, use Browserbase with Stagehand.
Firecrawl or Crawl4AI?
Firecrawl if you want it hosted with no infrastructure, and Crawl4AI if you want to run it yourself for free under Apache-2.0. Firecrawl's own self-hosted version is AGPL-3.0 and leaves out some cloud features, such as its Fire-engine anti-bot service, screenshots and page actions.
How do I turn a URL into Markdown for an LLM?
The quickest way is https://r.jina.ai/ followed by the URL, which works without an API key at up to 20 requests a minute (Jina may ask for a free key if you call it from a cloud server). Firecrawl's /scrape endpoint and Crawl4AI do the same with more options.
Is there a free web scraping API?
Yes. Firecrawl, Tavily and Crawl4AI Cloud each give 1,000 free credits a month, Exa gives $10 of credits a month, Apify gives $5 of usage a month, and Browserbase gives 1 browser hour. Crawl4AI's library is free to self-host, and Jina Reader works without a key.
Which of these tools have an MCP server?
All seven: Firecrawl, Crawl4AI, Jina (mcp.jina.ai), Apify (mcp.apify.com), Browserbase, Exa (mcp.exa.ai) and Tavily (mcp.tavily.com). Firecrawl and Tavily have keyless options you can try before you create an account, and Apify lets you search its Store without one.
Exa or Tavily?
Both are search APIs for agents with free tiers. Exa is pay-as-you-go and focuses on search quality and research. Tavily bundles search with extract, map and crawl, and has monthly plans.
Do I need a headless browser?
Only if your agent has to interact with the page (log in, click, scroll an app). Firecrawl, Jina Reader and Crawl4AI already render pages in a browser when they need to read JavaScript-heavy sites.
How we write these: every price, free tier and feature comes from the vendor's own pricing page, docs or GitHub repo (linked as Source) and was checked on the date above. Licenses come from each project's GitHub repo. "Our take" lines are our opinion. Prices change often, so check the link before you pay. The Firecrawl links are affiliate links, marked "(affiliate)", and they didn't change any verdict or order on this page. Something out of date? Email hello@hotfixdaily.com. No tool paid to be listed.
Get the next issue in your inbox
Five things worth your attention in AI, agents, vibe coding and game dev, every weekday. Plus 13 free Starter Prompts and the AI Builder Starter Stack PDF when you join.
