What is SiteOne Crawler?
SiteOne Crawler is a free, open-source website crawler and audit tool for people who build, maintain, or review websites. It can run as a desktop app, command-line tool, interactive wizard, or CI/CD quality gate on Windows, macOS, and Linux. Use it to inspect site quality, create offline copies, export content as Markdown, and gather reports for technical and SEO work.
SiteOne Crawler Features:
Deep Site Crawl: Crawl pages and website assets, then check SEO and OpenGraph data, accessibility, security, performance, broken links, redirects, caching, and other technical details.
Interactive Audit: Generate an HTML report with a 0–10 score across five areas and a panel that ranks issues by suggested priority.
JavaScript Rendering: Use a Chromium-based browser mode to crawl JavaScript sites and capture rendered pages, screenshots, and console, network, and JavaScript diagnostics.
Offline Website Clone: Create a browsable local mirror that includes site assets such as images, stylesheets, scripts, fonts, documents, and third-party resources.
Markdown Export: Convert website pages into browsable Markdown, combine content into one file, or create an llms.txt file for use with AI models.
CI/CD Quality Gate: Set configurable quality thresholds for pipeline checks, with JUnit XML, GitHub annotations, and a failure exit code when a check does not pass.
LLM Analysis: Connect commercial or local language models for optional SEO grading, summaries, spelling and grammar checks, and custom prompts.
SiteOne Crawler Benefits:
Prioritized Fixes: A ranked issue panel helps teams decide which site problems to address first instead of sorting through findings without guidance.
Earlier Checks: Pipeline thresholds can surface quality problems during deployment workflows, before they become part of a released site.
Private AI Review: Using a local model lets teams apply AI-assisted checks without sending site data to an external service.
Portable Archives: A complete local copy makes site content available for browsing when the original server is unavailable or offline access is needed.
Reusable Content: Markdown exports make site material easier to review, consolidate, or provide as context to an AI model.
Use Cases:
Prelaunch Site Audit: Review SEO, accessibility, security headers, performance, and broken links before publishing a new website or major redesign.
Deployment Checks: Add crawler quality thresholds to a CI/CD pipeline so a build can fail when configured checks are not met.
Documentation Context: Export a documentation site as Markdown or llms.txt to prepare its content for use with an AI model.
Offline Site Archive: Save a website as a local mirror for offline browsing, archival work, or disaster-recovery planning.
JavaScript Site Review: Crawl a React, Vue, Svelte, Angular, or Next.js site after rendering its pages in a browser and inspect captured diagnostics.