What is Pay-per-Crawl?
Pay-per-Crawl adds a payment gate for AI crawlers, SEO crawlers and scripts that request pages from a website. Site owners can deploy it on WordPress, Drupal, Laravel, Next.js, Node, Cloudflare, nginx, Caddy, Traefik or Python. A crawler that pays in USDC on Base receives the requested page after settlement, while human visitors and reverse-DNS-verified search engines can read for free.
Pay-per-Crawl Features:
Stack Coverage: Deploy the gate through integrations for WordPress, Drupal, Laravel, Next.js, Node, Cloudflare, nginx, Caddy, Traefik and Python.
Crawler Payment Gate: AI crawlers, SEO crawlers and scripts receive an HTTP 402 payment challenge, while humans and reverse-DNS-verified search engines can access pages free.
Direct Settlement: Crawlers pay USDC on Base, and the page is served only after the settlement transaction hash returns.
Owner Payments: Payments go directly to the site owner's wallet, with the owner keeping the full payment and no custody, ForgeMesh account or API key required.
Crawler Catalog: Each edition uses the same catalog of 85 crawler agents to identify automated requests.
Discovery Files: The gate serves x402.json, llms.txt and agents.txt files to provide machine-readable information about the site and its payment setup.
Pay-per-Crawl Benefits:
Earn From Crawling: Automated access can become a source of direct revenue instead of leaving crawler visits entirely unpaid.
Keep Your Proceeds: Direct wallet settlement lets site owners receive crawler payments without a service taking custody of the funds.
Preserve Human Access: Visitors and verified search engines can continue reading without paying, so the gate targets automated traffic rather than every request.
Choose Your Stack: Teams can add the payment gate within an existing web stack instead of moving their site to a different platform.
Avoid Account Setup: Owners can run the gate without creating a ForgeMesh account or managing an API key.
Use Cases:
Monetize AI Crawls: A publisher can charge AI agents that request articles while leaving human readers and verified search engines free to browse.
Protect Site Content: A site owner can put a payment challenge in front of pages requested by crawlers, SEO tools and scripts.
Add to WordPress: A WordPress operator can install the plugin to offer paid access to automated visitors without changing their web stack.
Gate an API Origin: A team can place a Cloudflare Worker in front of an origin to require payment from automated clients requesting its pages.
Deploy in Middleware: A developer can add the gate to a Next.js, Node, Python, Laravel or Drupal application using its stack-specific package or middleware.

