LightCrawl

LightCrawl is a self-hostable web scraping API and MCP server that converts web pages into Markdown for developers and AI workflows.

AI Developer Tools AI API Design AI Web Scraper

Visit Website

Overview

LightCrawl is a lightweight, single-container, self-hostable web scraping API and Model Context Protocol (MCP) server. It converts web pages into clean Markdown and is positioned in the repository as a minimal, low-cost alternative to Firecrawl for local development and low-resource environments.

The project is designed to handle both straightforward pages and more difficult targets. It starts with a static fetch, falls back to Playwright when it detects JavaScript rendering needs or bot protection, and then exposes the result through an HTTP API or MCP interface. The README also describes article-focused extraction, full-page conversion, crawling, site mapping, Redis-backed distributed queues, and optional Brave Search–based search tools.

The repository presents the service as a self-hosted tool for developers and AI-agent workflows that need sanitized web content rather than raw browser access. Its docs emphasize containerized isolation, Markdown output, and configuration through environment variables such as `PORT`, `API_KEY`, `ALLOWED_IPS`, `REDIS_URL`, and `MAX_CONCURRENCY`.

Features

Hybrid fetch pipeline

Tries a lightweight static HTTP fetch first, then falls back to a headless browser only when the page appears dynamic or protected. This keeps simple pages fast without giving up on JavaScript-heavy sites.

Two content extraction modes

Provides `article` and `full` extraction modes. `article` uses Mozilla Readability to isolate primary content, while `full` converts the entire HTML body to Markdown.

Fast mode for lighter runs

Supports `fast=true` to skip the longer browser wait path and use a lighter DOM parser. The README positions this mode for lower CPU and memory usage.

Dual API and MCP interface

Runs as both an Express HTTP API and an MCP server over stdio. The repository also references Swagger UI for endpoint documentation and testing.

Crawl and site mapping

Crawls sites recursively and can map internal URLs within a registered domain. It supports in-memory queues as well as Redis-backed distributed crawling.

Operational tooling for self-hosting

Uses Playwright with stealth tooling, native Prometheus-compatible metrics, and structured JSON logs. The Docker image is built to install only Chromium to keep the container smaller.

Use Cases

AI and automation pipelines
Run it as a local or server-side scraping layer when you want web pages normalized into Markdown for downstream processing instead of opening a browser in your own app.
Article extraction from content pages
Use the article mode when the target is a blog post, news story, or documentation page where the main content matters more than navigation or layout.
Full-page conversion for structured pages
Use the full mode when you need the entire page body preserved, such as for portals, directory listings, or search results.
Site discovery and recursive crawling
Use crawl and map endpoints when you need to discover related pages on the same registered domain and recursively process a site within depth and count limits.
Self-hosted scraping infrastructure
Deploy it in Docker with optional Redis when you need a shared crawl queue, background worker, or a self-hosted service that stays inside your own infrastructure.

Pros and Cons

Pros

Single-container self-hosted design keeps deployment simple.
Markdown output is tailored for downstream automation and LLM workflows.
Hybrid fetch logic avoids a browser when a page can be handled statically.
Supports both HTTP API and MCP access from the same service.
Includes crawling, site mapping, and optional Redis-backed distribution for larger jobs.

Cons

The repository docs are the main source of detail; some endpoint behavior and output specifics are only partially described in the available text.
Advanced capabilities such as distributed crawling and Brave Search tools depend on optional configuration or external keys, so the basic setup may not cover every workflow.

FAQ

What is LightCrawl used for?

LightCrawl is a self-hostable web scraping API and MCP server that converts web pages into Markdown. It also exposes HTTP endpoints and a Swagger UI for API documentation and endpoint testing.

How is LightCrawl deployed?

The README lists HTTP API and MCP usage, and the Dockerfile shows a containerized deployment built on Playwright with Chromium installed. The project is designed to run as a single container and can be configured with environment variables such as PORT, API_KEY, and REDIS_URL.

What output and scrape modes does it support?

The README describes two extraction modes: `article`, which uses Mozilla Readability to focus on primary content, and `full`, which converts the entire HTML body to Markdown. It also offers a `fast=true` mode that reduces browser and parser overhead.

When should someone choose LightCrawl over a simple page fetcher?

The project supports a lightweight static fetch first, then falls back to Playwright when JavaScript rendering or bot protection is detected. It also provides website mapping, recursive crawling, Redis-backed distributed crawling, and optional Brave Search–powered search tools.

Is LightCrawl a paid service?

The repository and changelog indicate the project is open source and self-hostable, with a minimal-cost hosting target called out in the README. GitHub’s pricing page also shows a free tier for repositories, while the project itself does not publish its own pricing.