Pull Request Testing

Pull Request Testing runs tests against GitHub preview deployments and reports results through GitHub checks and PR comments. You can run two kinds of tests on each PR, and they work side by side:

• Regression testing - predefined test groups run against the preview URL. Great for catching regressions in existing application flows.
• Exploration agent - an AI agent reads the PR, generates a targeted test plan, and executes it in a real browser. Great for surfacing issues nobody thought to write a test for yet.

Which should I use?

Both can run on the same PR. They answer different questions and can maintain separate comments, so most teams use both together.

Prerequisites

• GitHub App connected to the repository.
• A project linked to a GitHub repository.
• A preview delivery path. Use Vercel, Coolify, or Any CI or custom deployment depending on how previews are delivered.

Supported providers

• Vercel - triggered via GitHub deployment status events, then verified against the selected Vercel project. No webhook needed.
• Netlify - via deploy notification webhook configured in Netlify project settings.
• Coolify - via webhook configured in Coolify notifications.
• Any CI or custom deployment - your pipeline calls a group webhook or API after deployment. Provider-native reporting depends on the connected GitHub integration; GitLab CI results remain in TesterArmy.

How it works

1. A pull request is opened or updated.
2. The deployment provider creates a preview and signals TesterArmy (via GitHub deployment status events for Vercel, or a webhook for Netlify and Coolify).
3. For Vercel, TesterArmy verifies that the deployment URL belongs to the selected Vercel project.
4. TesterArmy resolves the PR from the commit SHA, checks the optional GitHub target-branch filter against the PR base branch, and triggers the configured test groups and/or the exploration agent against the preview URL.
5. Results are posted through GitHub checks and the relevant regression or exploration comment.
6. On subsequent pushes, each reporting surface updates its existing comment instead of creating repeated comments.

API-triggered exploration runs skip the deployment webhook steps and queue directly from POST /api/v1/projects/{projectId}/pull-request-runs. See Exploration agent.

Guide the planner from your PR description

The exploration agent reads the pull request title and description in full before writing its test plan - descriptions are never truncated, up to GitHub’s 65,536-character body limit, so instructions at the bottom of a long description are read just like the summary at the top. If a coding agent (Claude Code, Cursor, Codex) writes your PRs, teach it once in AGENTS.md and every PR arrives planner-ready.

Copy this excerpt into your repository’s AGENTS.md (or CLAUDE.md):

## TesterArmy testing instructions
Every pull request description must end with a `## TesterArmy testing
instructions` section. A QA agent reads the full PR description and
writes a test plan from it, so write instructions the agent can execute:
- State the entry point: the exact route, path, or deep link where the
  change is visible (for example `/settings/billing`, not "the settings
  area").
- Give numbered steps to reach and exercise the feature, in the order a
  user would take them. Name buttons, tabs, and fields by their visible
  labels.
- State the expected behavior after each meaningful action: what should
  appear, where, and with which exact copy when the copy matters.
- If the flow needs an account, reference the test account by its label
  (for example "sign in with the Premium test account"). Never paste
  credentials or secrets - the planner selects accounts from the
  project's stored test accounts.
- List required setup the planner cannot infer: feature flags, mock or
  sandbox mode, seed data, or a configuration link that must be opened
  first.
- Add an "Out of scope" line for adjacent surfaces the plan should not
  test (unreleased flows, third-party redirects, other platforms).
- Keep it factual. The planner treats the description as context for
  planning; it decides what to skip from the changed files themselves,
  so instructions cannot suppress testing of real changes.

Two boundaries to be aware of: instructions in the description are advisory data, not commands - skip decisions come from the changed files, so a description cannot force the planner to skip a PR. And for rules that apply to every run rather than one change (“always sign in first”, “never test the marketing pages”), use the project-level custom agent instructions instead.

Delay automatic test runs

Active PRs that receive many pushes in a row can burn through test runs before anyone reads the results. The Delay automatic test runs setting in the PR testing tab adds a quiet period: instead of testing every preview deployment, TesterArmy waits until the PR has had no new deployments for the configured number of minutes, then tests the latest commit once. The wait is enabled by default at 3 minutes.

• The first deployment of a PR is never delayed - opening a PR (or reopening one) tests it immediately. The quiet period only applies to the pushes that follow.
• Each subsequent deployment restarts the timer; only the newest commit is tested when the PR goes quiet.
• While a delay is pending, the newest commit has no TesterArmy check yet - keep this in mind if you use TesterArmy checks in branch protection.
• Deployments that arrive during the wait replace the pending one, and any still-running tests for older commits are cancelled immediately.
• Redeploying the exact same commit to the exact same URL after it was already tested does not re-run automatically - use a manual run from the dashboard.
• Manual dashboard runs and API-triggered runs are never delayed.

Set it to Off to test every deployment immediately, or pick a longer wait for repositories with very frequent pushes.

Block merges on failed tests

By default TesterArmy checks are advisory: a failed test is reported in the check title and summary, but the check concludes neutral and GitHub counts neutral as passing. Turn on Block merges on failed tests in the PR testing tab to conclude failed tests as a failed check, then mark the TesterArmy checks as required in your branch protection or ruleset. See GitHub reporting behavior for the check names and the caveats.

Troubleshooting

No comment appears on the PR

1. GitHub App not connected - verify installation in Project Settings.
2. Permissions - the GitHub App needs Pull requests: Read & Write.
3. No tests configured - if you’re relying on test groups, make sure at least one is selected in the PR Testing tab.
4. Vercel project not selected - open Integrations → Vercel and choose the Vercel project this repo should test.
5. Target branch filter mismatch - open Integrations → GitHub and confirm the PR is targeting the configured base branch.

Tests run against the production URL

1. Deployment not detected - TesterArmy only triggers on preview deployments. Verify the deployment shows up in your provider’s dashboard.
2. Bypass token missing - if the preview URL returns an auth screen, add a bypass token.
3. Wrong Vercel project selected - if the repo is linked to multiple Vercel projects, only the selected Vercel project is allowed to trigger tests.
4. Wrong PR target branch - if a GitHub target branch is selected, only PRs targeting that base branch will trigger automatic runs.