crawlforge-mcp

29 web scraping, crawling, deep-research & autonomous-extraction tools for Claude, Cursor & any MCP client.

Clean Markdown & structured JSON from any site. Get started with 1,000 free credits — no credit card required.

⭐ to follow along — it genuinely helps others discover the project.

Table of Contents

• Why CrawlForge?
• CrawlForge vs. alternatives
• Quick Start (2 Minutes)
• Available Tools
• Pricing
• Advanced Configuration
• Usage Examples
• Security & Privacy
• Support
• Contributing

🎯 Why CrawlForge?

• 29 MCP-native tools — scraping, crawling, search, real Google SERP rank tracking, deep research, an autonomous `agent`, a unified multi-format `scrape`, document processing, stealth browsing, and more, callable directly from your AI assistant.
• Generous free tier — 1,000 credits to start instantly, no credit card. The grant is one-time rather than monthly, and the credits never expire.
• Local-LLM by default — `extract_with_llm` runs against a local Ollama model out of the box: no LLM API key, no per-token cost, and your data never leaves your machine. Cloud (OpenAI/Anthropic) is opt-in.
• LLM-ready output — clean Markdown, structured JSON (schema-driven), screenshots, links, and metadata from a single fetch.
• Autonomous `agent` — describe what you need in natural language; it plans, gathers, and shapes an answer under orchestrator-enforced hard stops (max steps/URLs/wall-clock) — no URLs required.
• Security-hardened — SSRF protection on every request, a fail-closed backend allow-list, a vetted action allowlist for browser automation, and per-tool credit gating.
• Works everywhere MCP does — Claude Desktop, Claude Code, Cursor, and any other MCP-enabled client, configured in one command.

📊 CrawlForge vs. alternatives

> Comparison reflects publicly documented capabilities at time of writing. CrawlForge is MIT-licensed and MCP-first — built to plug straight into AI coding assistants.

🚀 Quick Start (2 Minutes)

1. Install from NPM

npm install -g crawlforge-mcp-server

2. Setup Your API Key (required)

Every tool requires a CrawlForge API key — new accounts get 1,000 free trial credits to start:

npx crawlforge-setup

This will:

• Guide you through getting your free API key
• Configure your credentials securely
• Auto-configure Claude Code and Cursor (if installed)
• Verify your setup is working

Don't have an API key? Get one free at https://www.crawlforge.dev/signup

> One-step setup (v4.6.0+): `crawlforge init` detects your API key, installs the agent skill, and idempotently merges the MCP config stanza into Claude Code, Claude Desktop, and Cursor. Use `crawlforge init --all --yes` to configure every detected client non-interactively.

3. Configure Your IDE (if not auto-configured)

🤖 For Claude Desktop

Add to `claude_desktop_config.json`:

{
  "mcpServers": {
    "crawlforge": {
      "command": "npx",
      "args": ["-y", "crawlforge-mcp-server"]
    }
  }
}

Location:

• macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`
• Windows: `%APPDATA%/Claude/claude_desktop_config.json`
• Linux: `~/.config/Claude/claude_desktop_config.json`

Restart Claude Desktop to activate.

🖥️ For Claude Code CLI (Auto-configured)

The setup wizard automatically configures Claude Code by adding to `~/.claude.json`:

{
  "mcpServers": {
    "crawlforge": {
      "type": "stdio",
      "command": "crawlforge-mcp"
    }
  }
}

After setup, restart Claude Code to activate.

💻 For Cursor IDE (Auto-configured)

The setup wizard automatically configures Cursor by adding to `~/.cursor/mcp.json`:

{
  "mcpServers": {
    "crawlforge": {
      "type": "stdio",
      "command": "crawlforge-mcp"
    }
  }
}

Restart Cursor to activate.

🔁 For n8n (workflow automation)

n8n's built-in MCP Client Tool node connects over Streamable HTTP (works on n8n Cloud and self-hosted). Run the server in HTTP mode:

export CRAWLFORGE_API_KEY=your_api_key
npm run start:http    # Streamable HTTP endpoint at http://localhost:10000/mcp

Then point the MCP Client Tool node at `http://:10000/mcp` with transport HTTP Streamable and a Bearer credential set to the same API key. On self-hosted n8n you can instead use the community `n8n-nodes-mcp` node over STDIO (`npx -y crawlforge-mcp-server`).

Full guide: docs/n8n-integration.md

> Which launch command? `npx -y crawlforge-mcp-server` needs no global install and always runs the published version (recommended for Claude Desktop). For a global install (`npm i -g crawlforge-mcp-server`), use the dedicated `crawlforge-mcp` bin — it resolves on your `PATH`, so it survives Node/nvm version switches. The bare `crawlforge` command still launches the server when an MCP client spawns it over stdio (backward compatibility for configs created before v4.2.5); interactively it's the CLI — run `crawlforge mcp` to start the server by hand.

📊 Available Tools

CrawlForge requires a CrawlForge API key — every tool is metered and consumes credits. New accounts get 1,000 free trial credits to start. Get a key at crawlforge.dev/signup.

All Tools (API key required)

For the full canonical capabilities reference (all tools, CLI commands, stealth engines, research workflow), see SKILL.md.

💳 Pricing

Every tool is metered and requires an API key. New accounts get 1,000 free trial credits — no credit card required to start.

All plans include:

• Access to all 29 tools
• Credits never expire; paid-plan credits roll over month to month
• API access and webhook notifications

View full pricing

🔧 Advanced Configuration

Environment Variables

# Optional: Set API key via environment
export CRAWLFORGE_API_KEY="cf_live_your_api_key_here"

# Optional: Custom API endpoint (for enterprise)
export CRAWLFORGE_API_URL="https://api.crawlforge.dev"
# As of v3.0.18, this variable is validated against an allow-list of CrawlForge backend hosts.

# Optional: Local LLM (Ollama) overrides — extract_with_llm, extract_structured
# and deep_research all use Ollama when no cloud key is set
export OLLAMA_BASE_URL="http://localhost:11434"   # default; set https://ollama.com for Ollama Cloud
export OLLAMA_DEFAULT_MODEL="gemma3:4b"            # optional; unset = pick the best installed model automatically
                                                   # deep_research judges claims with gemma3:12b when it is installed (ollama pull gemma3:12b);
                                                   # conflict detection is on only with that model, or a cloud provider
export OLLAMA_EMBEDDING_MODEL="nomic-embed-text"   # default: OLLAMA_DEFAULT_MODEL; used for semantic ranking in deep_research
export OLLAMA_API_KEY="..."                        # only for authenticated endpoints (required by Ollama Cloud; a local instance needs none)
export DISABLE_OLLAMA="true"                       # skip Ollama entirely and use CSS/keyword fallbacks

# Optional: Cloud LLM keys — only needed when you pass provider: "openai" or "anthropic"
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."

# Optional: limit which tools this client sees — by name, by group, or both (comma-separated)
export CRAWLFORGE_TOOLS="scrape,search_web,extract_content"
export CRAWLFORGE_TOOL_GROUPS="basic,search,scrape"   # unset = all tools; unknown names/groups are ignored with a warning

# Optional: deep_research stealth extraction fallback (v4.6.6) — see below
export RESEARCH_STEALTH_ENGINE="auto"      # auto (default) | camoufox | chromium
export RESEARCH_STEALTH_FALLBACK="true"    # set to "false" to disable entirely
export RESEARCH_MAX_STEALTH_RETRIES="8"    # cap on stealth retries per research run

MCP Spec Features

CrawlForge tracks the current MCP spec (2025-06-18) plus select experimental extensions:

• Structured output — `scrape`, `map_site`, `serp_rank`, `reddit_search`, `search_web`, `extract_structured`, and `crawl_deep` return machine-parseable `structuredContent` alongside the usual text, validated against a published `outputSchema`; legacy clients keep working off the text.
• Self-correctable errors — invalid tool input now comes back as an `isError: true` result the calling model can read and retry from, instead of a raw JSON-RPC protocol error.
• JSON Schema 2020-12 tool schemas, deterministic `tools/list` ordering (client prompt-cache friendly), and cacheable-result hints on read-only tools.
• Icons on the server, its tools, and its prompts.
• Async tasks (experimental) on the four long-running tools — `crawl_deep`, `batch_scrape`, `deep_research`, `agent` — for clients that support polling; synchronous results are still returned for clients that don't.

See docs/mcp-spec-adoption.md for wire-level examples and client-compatibility notes.

Local-LLM quickstart (`extract_with_llm` with Ollama)

`extract_with_llm` defaults to a local Ollama model — no LLM-provider key, no per-token LLM costs, and no data leaving your machine (the CrawlForge credit cost still applies).

# 1. Install Ollama:  https://ollama.com
# 2. Pull any model from https://ollama.com/library
ollama pull llama3.2

# 3. Discover what's installed (from your MCP client)
#    list_ollama_models()

# 4. Extract — defaults to Ollama with the model from step 2
#    extract_with_llm({ url: "https://example.com", prompt: "…", model: "llama3.2" })

Stealth extraction for `deep_research` (Camoufox)

`deep_research` automatically retries sources that block the normal fetch path (Reddit, Quora, forums, and Cloudflare/DataDome-protected pages return HTTP 403) through a real fingerprinted browser, then re-extracts from the rendered HTML. It's bounded (`RESEARCH_MAX_STEALTH_RETRIES`, default 8, plus a per-page timeout) and lazy — the browser stack only loads when a source is actually blocked.

Engine selection (`RESEARCH_STEALTH_ENGINE`):

• `auto` (default) — prefer Camoufox (Firefox anti-detect), fall back to Chromium stealth, then plain fetch.
• `camoufox` — force Camoufox.
• `chromium` — force the Chromium stealth engine.

Headless Chromium cannot clear modern challenges (Cloudflare Turnstile, DataDome) — Camoufox can. In testing it recovered Quora and Trustpilot pages that were otherwise fully blocked. To enable it, install the optional dependency and run its one-time binary fetch:

# Camoufox is declared as an optional dependency, so a normal install already pulls it.
# If you installed with --no-optional, add it explicitly:
npm install camoufox

# One-time download of the Camoufox Firefox binary (~130 MB):
npx camoufox fetch

Without the Camoufox binary, `deep_research` silently falls back to Chromium stealth and then to plain fetch — no errors, just lower recovery on heavily-protected sites. Disable the whole fallback with `RESEARCH_STEALTH_FALLBACK=false`.

> Note: Hard IP-reputation blocks (e.g. Reddit's edge `403`) resist headless stealth from any IP and require residential/mobile proxies, which CrawlForge does not provide. See docs/stealth-engines.md for details.

Manual Configuration

Your configuration is stored at `~/.crawlforge/config.json`:

{
  "apiKey": "cf_live_...",
  "userId": "user_...",
  "email": "you@example.com"
}

📖 Usage Examples

Once configured, use these tools in your AI assistant:

"Search for the latest AI news"
"Extract all links from example.com"
"Crawl the documentation site and summarize it"
"Monitor this page for changes"
"Extract product prices from this e-commerce site"

🔒 Security & Privacy

• Secure Authentication: API keys required for all metered tools
• Local Storage: API keys stored securely at `~/.crawlforge/config.json`
• HTTPS Only: All connections use encrypted HTTPS
• No Data Retention: We don't store scraped data, only usage logs
• Rate Limiting: Built-in protection against abuse
• Compliance: Respects robots.txt and GDPR requirements

Security & Approvals

• SSRF enforcement: Every scraped URL is validated before the request is sent — http/https only; blocks loopback, RFC1918, IPv6 private/link-local ranges, cloud metadata endpoints (GCP, Azure), and dangerous ports (SSH, SMTP, DNS, MySQL, Postgres, Redis, MongoDB, etc.). Redirects are re-validated each hop, capped at 5.
• Backend endpoint guard (v3.0.18): The server's own calls to CrawlForge.dev use a separate fail-closed allow-list (`{crawlforge.dev, www.crawlforge.dev, api.crawlforge.dev}`, HTTPS required). Setting `CRAWLFORGE_API_URL` to an arbitrary host is blocked at parse time.
• Action allowlist: `scrape_with_actions` accepts only 7 action types (`wait`, `click`, `type`, `press`, `scroll`, `screenshot`, `executeJavaScript`). No download, file-write, or arbitrary cross-page navigation primitives exist.
• JavaScript gate: The `executeJavaScript` action throws by default. Set `ALLOW_JAVASCRIPT_EXECUTION=true` at deploy time to enable (not recommended in production).
• MCP Elicitation (v3.6.0): Four tools request user confirmation before executing expensive operations — `deep_research` (>50 URLs), `batch_scrape` (sync mode, >25 URLs), `crawl_deep` (projected >500 pages), `extract_structured` (schema has >3 required fields with no LLM configured). Credit-low situations also elicit. Confirmation is best-effort: if the MCP client does not support elicitation the tool proceeds (fail-open).
• Per-tool credit gating: Every tool is wrapped with `withAuth()` and is metered — credits are checked and deducted before execution, and a valid API key is required for every tool (fail-closed since v3.0.18).

See docs/sandboxing-and-approvals.md for the full reference.

Security Updates

v3.0.3 (2025-10-01): Removed authentication bypass vulnerability. All users must authenticate with valid API keys.

For the full security policy and how to report a vulnerability, see SECURITY.md.

🆘 Support

• Documentation: https://www.crawlforge.dev/docs
• Issues: GitHub Issues
• Email: support@crawlforge.dev
• Discord: Join our community

📄 License

MIT License - see LICENSE file for details.

🤝 Contributing

Contributions are welcome! Please read our Contributing Guide first.

Built with ❤️ by the CrawlForge team

Website | Documentation | API Reference