How do I install Polite Crawler?

Copy the stdio config for Cursor, Claude Desktop, Claude Code, or VS Code on this page. It uses the publisher's install, not a PlugMCP proxy.

Does Polite Crawler need an API key?

The publisher lists these environment variables: POLITE_CRAWLER_DIR, POLITE_CRAWLER_CONTACT, POLITE_CRAWLER_ALLOW_PRIVATE, POLITE_CRAWLER_ALLOW_USER_PROVIDED, POLITE_CRAWLER_SIGN_KEY, POLITE_CRAWLER_SIGNATURE_AGENT.

What transport does Polite Crawler use?

Polite Crawler runs locally over stdio.

When was Polite Crawler last checked?

Last checked 2026-11-09. A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

Polite Crawler MCP server

Power your agent

Add this server

The publisher's own command. Not a PlugMCP endpoint.

Add to Cursor Add to VS Code Run on PlugMCP

Host a copy with our keys and logs.

Local command

npx -y [email protected]

Credentials from the publisher

POLITE_CRAWLER_DIR — The folder of the store. The default is ~/.polite-crawler.
POLITE_CRAWLER_CONTACT — An email address or URL for the From header of each request.
POLITE_CRAWLER_ALLOW_PRIVATE — Set it to 1 to let the crawler fetch loopback, private and link-local addresses. The default refuses them.
POLITE_CRAWLER_ALLOW_USER_PROVIDED — Set it to 1 to let user_provided read a page that refuses AI input when the client cannot ask the user to confirm.
POLITE_CRAWLER_SIGN_KEY — The file of an Ed25519 private key (PKCS#8 PEM or JWK). The crawler signs each request with it (Web Bot Auth). It needs POLITE_CRAWLER_SIGNATURE_AGENT.
POLITE_CRAWLER_SIGNATURE_AGENT — Where a site finds the public key of POLITE_CRAWLER_SIGN_KEY: the https origin of the key directory, or the URL of a JSON Web Key Set.

Claude Code has no install link for a local command. Copy the command below and run it in a terminal.

shClaude Code

1POLITE_CRAWLER_DIR=<value> POLITE_CRAWLER_CONTACT=<value> POLITE_CRAWLER_ALLOW_PRIVATE=<value> POLITE_CRAWLER_ALLOW_USER_PROVIDED=<value> POLITE_CRAWLER_SIGN_KEY=<value> POLITE_CRAWLER_SIGNATURE_AGENT=<value> claude mcp add io.github.mark1russell7-polite-crawler -- npx -y polite-crawler-mcp@0.5.2

Run in a terminal.

jsonClaude Desktop

1{2  "mcpServers": {3    "io.github.mark1russell7-polite-crawler": {4      "command": "npx",5      "args": [6        "-y",7        "[email protected]"8      ],9      "env": {10        "POLITE_CRAWLER_DIR": "<value>",11        "POLITE_CRAWLER_CONTACT": "<value>",12        "POLITE_CRAWLER_ALLOW_PRIVATE": "<value>",13        "POLITE_CRAWLER_ALLOW_USER_PROVIDED": "<value>",14        "POLITE_CRAWLER_SIGN_KEY": "<value>",15        "POLITE_CRAWLER_SIGNATURE_AGENT": "<value>"16      }17    }18  }19}

claude_desktop_config.json

Local · npmPublisher installStep-by-step: add this in Claude

Remote URL or a command on your machine? How the two installs differ

Run on PlugMCP

Host a copy with our keys and logs.

A new server you create. It is not this listing's official endpoint. Use the publisher install when you want their URL and their credentials.

Launch

Server facts

Local

Transport: stdio
Version

Tools

No tool list yet. Remote servers get one after the first health check.

FAQ

Questions

crawler

This repository holds polite-crawler, a polite web crawler for research and for AI agents, and its site.

polite-crawler fetches a source one time, keeps it, and reads it many times. It obeys robots.txt, sends one request at a time to each site, and answers each later question from a local full-text index.

The package: packages/polite-crawler is on npm as polite-crawler.

The site: packages/site is the documentation: https://mark1russell7.github.io/crawler/. Its interactive figures and tools start the real code: hybrid search, the recipes, the daemon and the robots.txt tester.

The study: Who lets AI agents in? The crawler read the robots.txt files of the top 1,000 domains. The page shows the verdict for 15 agents, with the rule that decided it.

npx polite-crawler fetch https://en.wikipedia.org/wiki/Web_crawler npx polite-crawler find "robots exclusion" claude mcp add polite-crawler -- npx -y polite-crawler-mcp

What the crawler does for you and for the sites:

It stops a host after a challenge, a 401, a 402 or a 403, for 1 to 24 hours. It does not try a different identity.

By default, it fetches only public addresses. --allow-private permits loopback and private addresses.

The MCP server gives page text in an <untrusted-web-content> envelope, so an assistant can tell data from instructions. A page that refuses AI input stays held back until a person confirms.

The library is polite-crawler, and the MCP server is the package polite-crawler-mcp (its README).

The package README tells the details.

Development

The repository is a pnpm workspace. It needs Node.js 22.19 or later and pnpm 10 or later.

pnpm install pnpm browser # the headless shell for the renderer test pnpm check # typecheck, lint, test, lint:ste, build and the smoke test of the package

The prose of the repository (the README files, the doc comments and the site) follows ASD-STE100 Simplified Technical English. The linter ste-lint examines it in CI.

CONTRIBUTING.md tells how to set up, test and write a change.

SECURITY.md tells how to report a vulnerability privately.

CHANGELOG.md records the changes in each version.

License

MIT

Add this server

Host a copy with our keys and logs.

Server facts

Tools

Questions

How do I install Polite Crawler?

Does Polite Crawler need an API key?

What transport does Polite Crawler use?

When was Polite Crawler last checked?

Documentation

crawler

Development

License