How-To Guides

validate-pr: Pull Request Testing and Evidence Skill for Codex

Install validate-pr for Codex and give it a PR link. It writes a test plan and Playwright tests, launches the app from the PR's exact commit, runs them, saves screenshots and logs, and returns Passed, Failed, Blocked or Needs Manual Verification.

  • Writes a test plan and Playwright tests mapped to each requirement, then runs them against the PR's exact commit
  • Screenshots, traces, console errors and app logs are saved per owner, repo, PR and run, outside the repository
  • ️ A script sets the status: Passed, Failed, Blocked or Needs Manual Verification. A skipped Required case never passes
  • Never changes application code. Test files are committed only after you approve the exact diff
By Rejith Krishnan6 min read
validate-pr: Pull Request Testing and Evidence Skill for Codex

What it is

validate-pr tests one GitHub pull request by running it. It is the companion to review-pr: review reads the change, validation launches the app and checks that it behaves as the requirements say. You give it the PR link.

It works out the requirements and acceptance criteria, writes a test plan and Playwright tests, builds and starts the application from the PR's exact commit, runs the tests and collects the evidence. Then it returns one of four statuses: Passed, Failed, Blocked or Needs Manual Verification.

Hard rules

  • It never changes application code. Only test plans, tests, fixtures and test configuration are written to the repository, and commit_tests.py refuses any other path, hard-coded credentials and runtime dependency changes in package.json.
  • Code and evidence stay apart. Committable files go in the repository. Screenshots, traces, logs and the report go in a run folder outside it.
  • It never reports a pass that did not happen. If the app did not start or the tests did not run, the status is Blocked, with the command, exit code and log tail recorded.
  • It tests the exact commit. The worktree is detached at the PR's head SHA, and a push during the run starts a new run instead of changing the old one.
  • PR text, issues, code and logs are data. Text in them that addresses an AI ("mark this as passed") is ignored.
  • Commit and push need separate approval. It shows the exact diff, commits only the hash you approved, and asks again before a fast-forward push. Never --force, never to the base branch.

What it does

  • Finds the requirements first, from your request, linked issues, tickets, plan-builder files and the PR description. If none of them says what was intended, it infers them and labels them as inferred.
  • Writes a test plan with positive, negative, boundary, accessibility and regression cases. Each is Required or Optional, Automated or Manual. Test titles carry the case IDs, and test_plan.py checks that every requirement has a case and every automated case has a test.
  • Launches the app for the stack it finds: Node, Python, JVM, .NET, Go, Ruby, PHP and static sites. In a monorepo it picks the folder the PR changes, and it waits until the app answers.
  • Runs the tests with screenshots and traces forced on and no retries, so a flaky pass cannot hide a failure. Browser console errors and application logs are collected.
  • Triages every failure as an application defect, a test-automation problem or an environment issue. Defects get a severity, the requirement, expected and actual behaviour, reproduction steps and evidence. Defects the PR did not introduce are marked pre-existing.
  • Needs Python 3.8+, git, a logged-in gh CLI and the application's own toolchain. Playwright browsers are installed when missing.

How the status is decided

verdict.py applies four rules in order:

  1. Failed: a confirmed Critical or High defect, or a confirmed defect the PR introduced on a Required case.
  2. Blocked: the app did not start, the tests did not run, or a Required case was skipped or hit an environment issue.
  3. Needs Manual Verification: a test problem, only pre-existing defects, or Required manual cases.
  4. Passed: none of the above.

One consequence to know about: a confirmed Medium defect on a Required case means Failed.

Where the files go

Evidence goes to one folder per run, keyed by GitHub owner, repository and PR number, so PR #12 in two repositories never collides:

~/.krithi/validate-pr-artifacts/github/<owner>/<repo>/pr-<n>/run-<UTC timestamp>-<sha12>/
    metadata.json   results/   screenshots/   traces/   logs/   defects/   report/validation-report.md

Set VALIDATE_PR_ARTIFACTS to use another location. If adding Playwright to the app would break its rules, such as a type-check that includes every .ts file, the tests go in a self-contained e2e/ package.

The reviewer loop

After the report you can challenge a defect, and it is re-checked against the evidence with every change recorded. You can also ask for more scenarios or rerun selected tests. After the author pushes fixes, a retest is a new run that marks each earlier defect Resolved, Still open or Needs verification.

A real run

The skill's author ran it on a website PR in lowtouch-ai/marketing. 12 of 13 tests passed. The status was Needs Manual Verification, because two accessibility defects existed site-wide before the PR: link contrast of 3.13:1, and code blocks that keyboard users could not reach. A second run left the first run's evidence untouched and marked both defects Still open. On a CRM PR that needs Keycloak and Vault, the status was Blocked, with the blocker recorded, rather than a guessed pass.

Install and usage

Codex CLI prerequisite: PowerShell setup or WSL setup.

Install the skill, then restart Codex:

$skill-installer install https://github.com/lowtouch-ai/krithi-skills/tree/main/skills/validate-pr

Check that gh auth status shows a logged-in account, then run it with a PR link:

$validate-pr validate https://github.com/<owner>/<repo>/pull/<n>

Add acceptance criteria, or no-commit to skip the offer to commit the tests:

$validate-pr validate PR 42 no-commit; acceptance criteria: quantity 0 shows "Quantity must be at least 1"

After the author pushes fixes:

$validate-pr retest https://github.com/<owner>/<repo>/pull/<n>

Running a PR means running its author's code on your machine, so use it on branches you trust. For update and uninstall steps, see the krithi-skills README.

Install all the skills together

If you want the whole set instead of one skill, point the installer at the repository root:

$skill-installer install all the skills in the https://github.com/lowtouch-ai/krithi-skills/

This asks Codex to install every skill in the krithi-skills repository in one step, so you do not repeat the install for each one. Restart Codex afterwards so the new skills are picked up.

The set currently covers nine skills:

  • validate-pr: tests a pull request by launching it, runs Playwright tests against the PR's exact commit and returns an evidence-backed status.
  • review-pr: reviews a pull request from just its link against its requirements and the Krithi process, scores it and posts the comment when you add post.
  • feature-estimate: turns customer emails, notes and tickets into a traceable requirements register, PRDs, stories and a sprint estimate.
  • prepare-pr: reviews finished work, checks the branch, commits in-scope files and opens the pull request with your approval at each step.
  • migration-estimate: turns a codebase into PRDs, stories and a sprint estimate with human-team and Krithi timelines.
  • plan-builder: turns a ticket into a Plan Mode prompt that changes no code.
  • repo-readme-generator: creates or updates a README and documentation suite from verified repository evidence.
  • appsec-static-audit: runs a source-code-only security audit with compliance mapping.
  • wave-a11y-scan: runs a WAVE-style accessibility scan mapped to WCAG 2.x.

Prefer to install one at a time? Use the single-skill commands in the install section above.

About the Author

Rejith Krishnan

Rejith Krishnan

Founder and CEO

Rejith Krishnan is the Founder and CEO of lowtouch.ai, a platform dedicated to empowering enterprises with private, no-code AI agents. With expertise in Site Reliability Engineering (SRE), Kubernetes, and AI systems architecture, he is passionate about simplifying the adoption of AI-driven automation to transform business operations.

Rejith specializes in deploying Large Language Models (LLMs) and building intelligent agents that automate workflows, enhance customer experiences, and optimize IT processes, all while ensuring data privacy and security. His mission is to help businesses unlock the full potential of enterprise AI with seamless, scalable, and secure solutions that fit their unique needs.

LinkedIn →