What is pagesift?
PageSift is a self-hosted Rust engine for collecting and processing web pages, documents, archives, email, and source files. It turns these inputs into Markdown, schema-validated JSON, or evidence-linked data for search and AI workflows. Founders, developers, and businesses can use it through a command-line tool, HTTP APIs, an MCP server, or Python and Node.js packages, and run it on their own infrastructure.
pagesift Features:
Web Scraping: Fetch a URL and extract main content as Markdown while preserving elements such as headings, tables, code, and math, with browser rendering when it adds content.
Site Crawling: Map site URLs, crawl pages with query-focused stopping and trap handling, and run persistent batch jobs with polling, cancellation, and webhooks.
Document Parsing: Process PDFs, Office files, e-books, email, archives, data files, and source code, with optional OCR for scanned pages and images.
Structured Extraction: Return schema-based JSON using CSS selectors that can relocate when a site changes, or use a schema generated by a model and cached by PageSift.
Evidence Graph: Record citations, redirects, attachments, source versions, and evidence spans, then group claims into facts and flag conflicting source details.
Sync and Context: Maintain persistent sources for change detection and compile cited, token-budgeted ContextPacks from indexed material for downstream AI workflows.
Developer Interfaces: Use the engine through a CLI, Firecrawl-compatible and Jina Reader HTTP endpoints, MCP, Python, Node.js, Rust, or a C ABI.
pagesift Benefits:
Fewer Conversion Steps: Handling web pages and many document formats in one engine reduces the need to assemble separate parsers and converters for an ingestion pipeline.
More Auditable Answers: Source citations and evidence spans make it easier to trace extracted claims back to the material that supports them.
Earlier Conflict Detection: Fact clusters and conflict records help teams notice when sources disagree instead of silently treating differing values as consistent.
Local Data Control: Self-hosting keeps scraping and document processing on infrastructure the team operates, rather than requiring a hosted PageSift service.
Targeted AI Context: Compiling relevant material within a token budget helps teams prepare focused, cited context for model workflows.
Use Cases:
Research Agent Inputs: Collect web pages for an agent research run, then check answer fields against the pages used as evidence.
Internal Document Search: Convert reports, scanned PDFs, email, and office files into searchable material for a retrieval or question-answering pipeline.
Site Change Tracking: Monitor documentation or other web sources on a schedule and send a webhook when PageSift detects a substantive change.
Product Data Audits: Extract prices or other structured fields from several pages and inspect the graph for conflicting values and their source excerpts.
Codebase Ingestion: Parse source files across supported languages into function- and class-aware content for indexing or an AI-assisted code workflow.

