Crawl Census

crawl-census-client

Ask before you fetch. A drop-in client that stops your crawler spending requests on doors that are shut, and stops it routing around content someone is trying to sell.

Reading robots.txt answers one question and hides two others. Measured across 23,482 domains by Crawl Census:

• 2,874 domains permit AI agents in robots.txt and then refuse those same agents at the network edge. A parser sees permission; the fetch returns 403. You pay for the round trip and get nothing.
• 208 domains answer an AI user agent with HTTP 402 Payment Required. That is a price, not a refusal. Treating it as a block walks away from content the operator wants to sell you. Retrying around it takes something they are charging for.

No dependencies. No key required.

MCP server

The same measurement is exposed as a remote MCP server, so an agent can ask before it fetches rather than after it fails. Listed in the official MCP registry as io.github.taylorsmithgg/crawl-census.

{ "mcpServers": { "crawl-census": { "url": "https://crawlcensus.com/mcp" } } }

No authentication for read tools. Streamable HTTP.

Install

npm i github:taylorsmithgg/crawl-census-client
pip install git+https://github.com/taylorsmithgg/crawl-census-client

Use

import { politeFetch } from "crawl-census-client";

const r = await politeFetch("https://example.com/", { agent: "gptbot" });
if (r.skipped) console.log(r.verdict, r.reason);   // disallow | refuse | pay
else           process(await r.response.text());

from crawl_census import polite_fetch

r = polite_fetch("https://example.com/", agent="gptbot")
if r.skipped:
    print(r.verdict, r.reason)
else:
    process(r.body)

Skipping is returned, not raised. It is the normal outcome for a large share of the web, and a crawl loop should be able to count skips without a try/except around every URL.

Split a queue before crawling it

One call per 1,000 domains instead of one per host:

const { crawl, skip, pay, unknown } = await partition(urls, { agent: "gptbot" });

p = partition(urls, agent="gptbot")
p.crawl, p.skip, p.pay, p.unknown

Or just take the file

For a fetcher that only needs a deny list in memory, skip the per-domain calls entirely:

curl https://crawlcensus.com/agents/gptbot/blocklist.txt   # one domain per line, commented header

const sync = await syncBlocklist("gptbot");   // full list once
if (sync.blocked.has(host)) skip();
setInterval(() => sync.refresh(), 3600_000);  // then deltas only, a few hundred bytes

sync = BlocklistSync("gptbot")
if host in sync: skip()
sync.refresh()          # {'added': 3, 'removed': 1, 'size': 3310, 'cursor': ...}

The delta feed is https://crawlcensus.com/agents/<agent>/changes.json?since=<unix> and each response carries next_since, so a long-running crawler stays current on a few hundred bytes an hour instead of re-downloading the list.

That file covers robots.txt only. Edge refusal and HTTP 402 are per-request behaviours and still need preflight or politeFetch.

What a crawl costs the census

Measured, not asserted. Twenty hosts fetched concurrently used to cost twenty preflight calls carrying one domain each; the same host requested three times at once cost three, because the cache only helps after the first lookup resolves. The anonymous allowance is 240 calls an hour, so a crawler hit its ceiling at 240 hosts when one call covers twenty-five.

politeFetch now shares work automatically: lookups issued in the same tick leave as one batched call, and concurrent lookups for the same host await a single request.

batchSize defaults to 25, the per-call cap without a key. Raise it with a Pro or Data key. batchWaitMs widens the coalescing window for concurrency that arrives in waves rather than all at once; the default of zero flushes on the next tick.

Paying, when an origin quotes a price

A pay verdict carries the amount when the origin named one:

const r = await politeFetch(url, { agent: "claudebot" });
if (r.verdict === "pay") console.log(r.price);   // "USD 0.5", or null if none was quoted

Two things worth knowing. Most origins answering HTTP 402 name no amount at all, so price is usually null and the arrangement has to be made out of band. And pricing is per crawler: across the measured corpus, 78 of 213 charging origins charge some agents and serve others free, so ask with your own token rather than assuming a domain on the list will charge you.

Two kinds of unknown

partition splits a work queue into crawl, pay, skip, unknown and undecidable.

The last two look alike and are not. unknown means the census has not measured that domain yet: submit it and the next pass gets a real verdict. undecidable means the site's robots.txt disallows CrawlCensusBot, so this census will never measure it — retrying is guaranteed waste, and a loop that resubmits its unknowns each pass would resubmit those forever. The server marks the difference with a measurable boolean; read that, never the reason prose.

const p = await partition(urls, { agent: "gptbot" });
await Promise.all(p.crawl.map(politeFetchOne));
if (p.unmeasured.length) await submitUnmeasured(p, { agent: "gptbot" });
// p.undecidable: read their robots.txt yourself. Asking us again cannot help.

Submission is a separate call on purpose. A library that quietly POSTs during what reads as a lookup is a bad citizen, and you should choose when your queue positions are spent.

Skipping everything that will not serve you

A deny list is the smaller half. Measured against the live census, a crawler that skips only robots disallows still spends around 2,900 requests a pass on domains that permit it in robots.txt and refuse it at the edge, or that answer HTTP 402 — for PerplexityBot that set is larger than its deny list. Those fetches return nothing and cost a round trip each.

const skip = await syncSkipList("gptbot");
if (skip.has(host)) continue;        // disallowed, refused at the edge, or priced
skip.why(host);                      // "disallow" | "pay" | "refuse" | null
setInterval(() => skip.refresh(), 3600_000);

The three sets are kept apart internally, so a change moves the one it belongs to. why() follows the same precedence as preflight: a disallow outranks a price, because a price is not permission.

Keeping a deny list current

syncBlocklist / BlocklistSync download the list once, then apply only what changed.

The list is served with the exact position in the change feed it was built at, in an x-cursor header and a # cursor: comment. The clients read it and resume from there, so there is no gap between the snapshot and the first poll, and no reliance on your clock being in step with the server's. Polling by second cannot express a position inside a second, and a crawl batch writes dozens of events into one, so a second-granularity resume can drop the remainder of it: measured live, resuming after the first of three same-second changes recovered both siblings by cursor and neither by second.

const sync = await syncBlocklist("gptbot");   // cursor comes from the list itself
if (sync.blocked.has(host)) skip();
setInterval(() => sync.refresh(), 3600_000);  // a few hundred bytes per poll

refresh() applies only robots transitions to the list, because that is what the list is made of. Edge refusals, new prices and llms.txt changes come back in other for you to act on separately — an earlier version deleted those domains from the deny list, so a crawler resumed fetching exactly what had just started refusing it.

Verdicts

The authoritative definition of each verdict — what it means, what it obliges a crawler to do, and whether asking again could change it — is published as data at /api/v1/verdicts. The list below is a summary; if the two ever disagree, the endpoint is right and this file is stale.

politeFetch skips disallow, refuse and pay by default, which is the endpoint's derived do_not_fetch set. The copy here is deliberate — a crawl loop should not need a network call to decide — and a test compares the two so it cannot drift unnoticed.

onPay: "fetch" (on_pay="fetch") overrides the paywall default. It is an explicit opt-in and is recorded on the result as paidRouteOverridden so it shows up in your logs.

It degrades, it does not fail

If the census is unreachable every verdict becomes unknown and your crawl proceeds as it normally would. A third-party outage must never stop your pipeline. There is a live test for exactly this.

What we publish about your agent

const p = await agentProfile("claudebot");
// robots disallow rate, edge refusal rate, operator page, correction channel

If a figure is wrong, the correction channel is in that response and on your operator page. Registry facts are corrected without argument; disputed measurements are published alongside the dispute with the underlying scan records, rather than quietly amended.

Limits

25 domains per preflight call anonymously, 200 with a Pro key, 1,000 with a Data key. Pass apiKey. Details at https://crawlcensus.com/for-crawlers.

Tests

node test.mjs runs against the live census on purpose. The value of this client is whether its verdicts match reality, and a mocked test would assert only that the mock agrees with itself.

MIT. Data is CC BY 4.0, attribute as "Source: Crawl Census (crawlcensus.com)".