What is Vibecheck?
Vibecheck is a self-hosted merge gate for GitHub pull requests that checks whether the person who opened a change understands it, including code produced with AI. It creates a quiz from the pull request’s diff and uses an AI grader to assess the author’s answers. Teams can make the result a required branch-protection check or leave it advisory. It requires a Docker container, Postgres database, and an Anthropic API key or Amazon Bedrock in the team’s AWS account; the project is still early.
Vibecheck Features:
Diff-Based Questions: For each pull request, Vibecheck creates a high-level question and scoped questions tied to specific sections of the change.
One Question at a Time: Authors see each question alongside the relevant portion of the diff, and answer in a browser through a tokenized link without logging in.
AI Answer Grading: Claude grades answers against an interpretation of the diff, accepts concise but correct explanations, and marks answers that contradict the change as incorrect.
Configurable Verdict: Teams set how many correct answers are needed, and Vibecheck posts a green or red verdict as a GitHub commit status.
Practice Locally: The CLI runs the same questions and grading in a terminal against the current branch or a GitHub pull request.
Self-Hosted Setup: Teams run the Docker service themselves and connect it to a GitHub Actions workflow using a bearer token.
Vibecheck Benefits:
More Informed Reviews: Reviewers can spend less time establishing what a change does when its author has already demonstrated a basic understanding of it.
Human Accountability: Teams can keep a person responsible for explaining code in the delivery process, even when AI generated much of the change.
Fewer Unexplained Changes: A required passing verdict makes author understanding part of the merge decision rather than relying only on a green CI run.
Useful Author Feedback: Authors can identify gaps in their understanding before reviewers need to point them out, and retry answers while completing the quiz.
Deployment Control: Self-hosting and bring-your-own-key support let teams run the service and use model access in their own environment.
Use Cases:
AI-Assisted Pull Requests: Require developers using code-generation tools to explain the changes they plan to merge into a shared repository.
Large Pull Request Reviews: Ask authors to walk through different sections of a substantial diff before maintainers invest time reviewing it.
Open-Source Contributions: Use an understanding check to screen drive-by pull requests whose authors may not have examined generated code closely.
Team Merge Policies: Add an author-understanding requirement to branch protection for repositories where a passing Vibecheck verdict should block or permit merging.
Local Code Practice: Run the CLI against a branch or pull request to test your own understanding without affecting the repository’s merge gate.