Website to Markdown API

Product Introduction

1. Definition: The Exabase Website to Markdown API is a cloud-based, multi-modal data extraction service that converts the content of any publicly accessible URL into a clean, structured Markdown document. It operates as a headless browser-as-a-service, executing JavaScript and rendering modern web applications before extracting the core textual content.
2. Core Value Proposition: It exists to provide developers and AI applications with a reliable, zero-configuration method for converting web content into a token-efficient, LLM-ready format. Its primary value is eliminating the complexity of web scraping—handling anti-bot measures, JavaScript rendering, and content cleaning—through a single API call, thereby accelerating the ingestion of web data into knowledge bases, RAG pipelines, and AI agent context windows.

Main Features

1. JavaScript Rendering & Content Extraction: The API automatically loads and executes JavaScript, CSS, and fetches subsequent data requests to fully render pages built with frameworks like React, Next.js, Vue, and Angular. It then applies intelligent content extraction algorithms to strip away navigation menus, footers, cookie banners, and advertisements, returning only the primary article or page content as clean text.
2. Multi-Modal Format Support with Unified Markdown Output: The same POST /v2/extract endpoint processes not just HTML but also PDFs, DOCX, PPTX, images, audio, and video files. By appending the ?format=markdown parameter to the result request, users receive a standardized Markdown document for any supported file type—whether it's the transcribed text from a video, OCR text from an image, or the body of a web article.
3. Anti-Bot Evasion & Reliability Infrastructure: To ensure high success rates, the service employs a managed infrastructure featuring proxy rotation, browser fingerprinting, and automatic retry logic. This combats IP-based blocking and challenges from sites employing basic anti-scraping measures, providing a more reliable extraction service compared to simple HTTP GET requests.
4. Asynchronous Processing with Webhook Support: URL extraction jobs are processed asynchronously. Users can poll for results or, more efficiently, configure a webhook endpoint with webhookFormat: "markdown" to receive the completed Markdown document via a POST request automatically, enabling event-driven architectures.
5. Integrated CDN-Hosted Screenshot: Each extraction job includes the generation of a full-page screenshot, which is hosted on a CDN and provided as a URL within the API's JSON response. This provides visual context and verification without requiring additional screenshot services.

Problems Solved

1. Pain Point: Manually building and maintaining a web scraping pipeline is complex, requiring expertise in headless browsers (Puppeteer, Playwright), proxy management, HTML parsing, and anti-bot countermeasures. This distracts from core product development.
2. Target Audience: AI/ML Engineers building RAG systems; Developers creating content aggregation, competitive intelligence, or SEO auditing tools; SaaS companies needing to import user-submitted web content; Research teams automating data collection from publications and news sites.
3. Use Cases: Feeding LLM Context Windows: Directly inserting cleaned web page Markdown into an AI agent's prompt for up-to-date information. Building a Searchable Knowledge Base: Ingesting documentation, blog posts, or support articles as Markdown into a vector database for semantic search. Automating Competitive Analysis: Scheduling periodic extractions of competitor pricing or feature pages to track changes. Creating Accessible Content: Converting image-heavy PDFs or presentations into readable Markdown text.

Unique Advantages

1. Differentiation: Unlike standalone Markdown converters (e.g., Jina Reader) or open-source libraries (e.g., Trafilatura), Exabase is part of an integrated "data layer for agents." The extracted Markdown can be instantly stored as a searchable Resource within an Exabase Base and queried using its Deep Search (hybrid multi-modal search) without any additional setup. It combines extraction with storage, search, and memory.
2. Key Innovation: The "one API, all formats" approach is a significant innovation. The abstraction of diverse content types (websites, PDFs, audio) into a single extraction job endpoint and a unified output format (Markdown or JSON) drastically reduces the integration complexity for developers needing to process heterogeneous data sources.

Frequently Asked Questions (FAQ)

1. How does the Website to Markdown API handle sites that block scrapers? The API uses a managed infrastructure with automatic proxy rotation and browser fingerprinting to mimic legitimate user traffic, increasing success rates against common anti-bot measures. However, it respects robots.txt and cannot access pages behind logins.
2. What is the difference between the JSON and Markdown output formats? The JSON output provides a highly structured response with separated metadata, chunks of text, and other programmatic data. The Markdown format condenses this into a single, human-readable document optimized for direct use in LLM prompts or documentation. Both are generated from the same extraction job.
3. Can I extract content from a PDF or video to Markdown with the same API? Yes. The Exabase Extract API is multi-modal. Submitting a PDF, audio, or video file URL to the same POST /v2/extract endpoint and requesting the result with ?format=markdown will return a Markdown document containing the extracted text or transcript.
4. How long does the API retain the extracted files and data? Processed files are retained for 24 hours from job creation. It is recommended to download or transfer the output Markdown or JSON to your own storage system within this period, as files are permanently deleted afterward.
5. Is there a way to receive the Markdown result without polling the API? Yes. You can configure a webhook URL in your Extract settings and specify webhookFormat: "markdown". When the extraction job completes, Exabase will immediately HTTP POST the full Markdown document to your configured endpoint.