Pull Request Testing | TesterArmy: Documentation

Pull Request Testing runs tests against GitHub preview deployments and reports results through GitHub checks and PR comments. You can run two kinds of tests on each PR, and they work side by side:

Regression testing - predefined test groups run against the preview URL. Great for catching regressions in existing application flows.
Exploration agent - an AI agent reads the PR, generates a targeted test plan, and executes it in a real browser. Great for surfacing issues nobody thought to write a test for yet.

Which should I use?

Both can run on the same PR. They answer different questions and can maintain separate comments, so most teams use both together.

	Regression testing (static)	Exploration agent (dynamic)
Steps	The steps you saved	Generated per PR from the change
Best for	Guarding known, critical flows	Surfacing issues nobody wrote a test for
Coverage	Exactly what you defined, every time	Adapts to what changed in each PR
Setup	Author and maintain the tests	Zero step authoring
Can skip a PR	No, always runs the enabled tests	Yes, when there is nothing user-visible to test

Prerequisites

GitHub App connected to the repository.
A project linked to a GitHub repository.
A preview delivery path. Use Vercel, Coolify, or Any CI or custom deployment depending on how previews are delivered.

Supported providers

Vercel - triggered via GitHub deployment status events, then verified against the selected Vercel project. No webhook needed.
Netlify - via deploy notification webhook configured in Netlify project settings.
Coolify - via webhook configured in Coolify notifications.
Any CI or custom deployment - your pipeline calls a group webhook or API after deployment. Provider-native reporting depends on the connected GitHub integration; GitLab CI results remain in TesterArmy.

How it works

A pull request is opened or updated.
The deployment provider creates a preview and signals TesterArmy (via GitHub deployment status events for Vercel, or a webhook for Netlify and Coolify).
For Vercel, TesterArmy verifies that the deployment URL belongs to the selected Vercel project.
TesterArmy resolves the PR from the commit SHA, checks the optional GitHub target-branch filter against the PR base branch, and triggers the configured test groups and/or the exploration agent against the preview URL.
Results are posted through GitHub checks and the relevant regression or exploration comment.
On subsequent pushes, each reporting surface updates its existing comment instead of creating repeated comments.

API-triggered exploration runs skip the deployment webhook steps and queue directly from POST /api/v1/projects/{projectId}/pull-request-runs. See Exploration agent.

Delay automatic test runs

Active PRs that receive many pushes in a row can burn through test runs before anyone reads the results. The Delay automatic test runs setting in the PR testing tab adds a quiet period: instead of testing every preview deployment, TesterArmy waits until the PR has had no new deployments for the configured number of minutes, then tests the latest commit once. The wait is enabled by default at 3 minutes.

The first deployment of a PR is never delayed — opening a PR (or reopening one) tests it immediately. The quiet period only applies to the pushes that follow.
Each subsequent deployment restarts the timer; only the newest commit is tested when the PR goes quiet.
While a delay is pending, the newest commit has no TesterArmy check yet — keep this in mind if you use TesterArmy checks in branch protection.
Deployments that arrive during the wait replace the pending one, and any still-running tests for older commits are cancelled immediately.
Redeploying the exact same commit to the exact same URL after it was already tested does not re-run automatically — use a manual run from the dashboard.
Manual dashboard runs and API-triggered runs are never delayed.

Set it to Off to test every deployment immediately, or pick a longer wait for repositories with very frequent pushes.

Troubleshooting

No comment appears on the PR

GitHub App not connected - verify installation in Project Settings.
Permissions - the GitHub App needs Pull requests: Read & Write.
No tests configured - if you’re relying on test groups, make sure at least one is selected in the PR Testing tab.
Vercel project not selected - open Integrations → Vercel and choose the Vercel project this repo should test.
Target branch filter mismatch - open Integrations → GitHub and confirm the PR is targeting the configured base branch.

Tests run against the production URL

Deployment not detected - TesterArmy only triggers on preview deployments. Verify the deployment shows up in your provider’s dashboard.
Bypass token missing - if the preview URL returns an auth screen, add a bypass token.
Wrong Vercel project selected - if the repo is linked to multiple Vercel projects, only the selected Vercel project is allowed to trigger tests.
Wrong PR target branch - if a GitHub target branch is selected, only PRs targeting that base branch will trigger automatic runs.