What is Jevcrawl?
Jevcrawl is a web crawler for AI agents that searches a site for pages and passages relevant to a question, rather than returning a broad collection of pages. It is available as an MCP server and command-line tool, and can also be used by Python callers. Developers can connect it to MCP clients such as Claude Code, Claude Desktop, and Cursor, then give an agent a goal and a starting URL. A Jev API key is required.
Jevcrawl Features:
Goal-Directed Crawling: The crawl tool takes a question and starting URL, ranks links and passages for relevance, and follows promising links within configurable page, time, and decision-cost limits.
Passage Extraction: Crawl results return relevant page passages with their URLs, heading paths, character offsets, and relevance scores, alongside borderline passages marked uncertain.
URL Mapping: The map tool ranks URLs from a starting site for a goal, so an agent can choose pages to inspect without retrieving all their content.
Page Scraping: The scrape tool retrieves a known page, returning goal-relevant passages when given a goal or all page content when used without one.
JavaScript Rendering: With the optional JavaScript extra and Chromium installed, Jevcrawl can render detected JavaScript-only pages, with a limit of five rendered pages per call.
MCP and CLI: Jevcrawl exposes crawl, scrape, and map operations through an MCP server or command-line interface, and includes a doctor command to check setup.
Jevcrawl Benefits:
Less Reading: Agents can work from selected passages instead of feeding all fetched page text, including navigation and boilerplate, into an answering model.
Traceable Answers: Passages retain source URLs, headings, and character offsets, making it easier to check where a quoted answer came from.
Controlled Spending: Configurable decision-cost limits and local caching help developers bound crawl decisions and avoid paying again for repeated goals on the same pages.
Fewer Manual Steps: Agents can search across a site and return relevant evidence in one workflow, rather than requiring a person to open and skim each candidate page.
Safer Page Handling: Because page text is scored rather than followed as instructions, the crawler reduces the chance that injected page content directs the agent's browsing process.
Use Cases:
Documentation Questions: An agent can start at a software project's documentation home page and find the passage that answers a question about a feature or version.
Policy Research: A support agent can search a company's help site for the relevant section of a returns, shipping, or account policy.
Organization Research: A research assistant can traverse a university or organization's site to locate information several links away from its home page.
Known-Page Lookup: When a user already has a page URL, an agent can scrape it for a specific answer or retrieve its full content for review.
Site Page Discovery: A developer can use ranked URLs to identify which pages on a site are worth scraping for a particular research question.