/search by Firecrawl

Product Introduction

1. The /search endpoint by Firecrawl is a web search API that returns full LLM-ready page content for every search result, enabling developers and AI agents to access structured data directly from web pages. It combines traditional search functionality with automated content extraction, eliminating the need for separate crawling or parsing steps.
2. This product delivers immediate access to processed web data in formats like Markdown and JSON, optimized for integration with AI models and data pipelines. Its core value lies in bridging the gap between raw web search results and actionable structured data for machine learning applications.

Main Features

1. The API returns complete page content including Markdown, JSON, and screenshots for each search result, enabling direct consumption by LLMs without additional preprocessing. This includes parsed headings, text blocks, and metadata extracted from HTML structures.
2. Built-in JavaScript rendering and smart waiting ensure accurate results for modern SPAs and dynamically loaded content, handling delays up to 30 seconds for complex page interactions. The system automatically detects when AJAX calls or React/Vue components finish loading.
3. Multi-format output supports integration with Python, Node.js, and cURL environments through REST API endpoints, with SDKs offering native typing support and error handling. Developers can implement search-driven data collection in under 10 lines of code using pre-built templates.

Problems Solved

1. Eliminates manual effort required to convert search results into usable training data for AI models, reducing preprocessing time from hours to milliseconds per query. Traditional methods require separate crawling, parsing, and formatting steps that this endpoint consolidates.
2. Serves AI developers building RAG systems, growth teams conducting competitive analysis, and researchers aggregating web data at scale. The solution particularly benefits users needing real-time web insights without infrastructure overhead.
3. Enables use cases like monitoring brand mentions across news sites, tracking product updates from competitors' blogs, and gathering training data for domain-specific language models. Supports bulk operations through concurrent processing of up to 100 search results per API call.

Unique Advantages

1. Unlike standard search APIs that return only metadata snippets, Firecrawl provides full-page semantic content including embedded tables, code blocks, and formatted text. This depth of extraction typically requires custom scraping setups that take weeks to develop.
2. Proprietary anti-blocking mechanisms combine residential proxies with Chrome headless browser emulation, achieving 98% success rates on sites protected by Cloudflare and PerimeterX. The system automatically rotates IPs and user agents between requests.
3. Integrated credit-based pricing combines search and content extraction costs into single API calls, offering 50% cost reduction compared to using separate search engines and scraping services. Rate limits scale dynamically based on account tier, supporting up to 1,000 RPM on enterprise plans.

Frequently Asked Questions (FAQ)

1. How does Firecrawl handle JavaScript-heavy websites? The system uses headless Chrome browsers with intelligent waiting mechanisms that execute client-side scripts and wait for network idle states before extracting content, supporting React, Angular, and Vue.js applications.
2. What websites can't be crawled with /search? The API respects robots.txt exclusions and cannot access password-protected sites, though it bypasses most cookie walls and age gates through automated interaction simulations.
3. How many credits does a /search request consume? Each search result page costs 1 credit, with additional charges for optional features like screenshot capture (0.5 credits) and PDF extraction (2 credits). A typical 10-result query with screenshots would consume 15 credits.
4. Can I filter search results by date or domain? Yes, the API supports advanced filters including publication date ranges, top-level domain restrictions, and content type parameters (blogs, news articles, documentation).
5. What happens when search results contain paywalled content? The system detects paywalls through DOM pattern matching and returns partial content with a paywall flag, allowing users to implement custom bypass logic through the action system if legally permitted.