RunPull Request Testing

Pull Request Testing

View as Markdown

Pull Request Testing runs tests against every preview deployment and posts results as a GitHub comment. You can run two kinds of tests on each PR, and they work side by side:

  • Regression testing - a predefined test group runs against the preview URL. Great for catching regressions in existing application flows.
  • Exploration agent - an AI agent reads the PR, generates a targeted test plan, and executes it in a real browser. Great for surfacing issues nobody thought to write a test for yet.

Which should I use?

Both run on the same PR and post to the same comment. They answer different questions, so most teams use both together.

Regression testing (static)Exploration agent (dynamic)
StepsThe steps you savedGenerated per PR from the change
Best forGuarding known, critical flowsSurfacing issues nobody wrote a test for
CoverageExactly what you defined, every timeAdapts to what changed in each PR
SetupAuthor and maintain the testsZero step authoring
Can skip a PRNo, always runs the enabled testsYes, when there is nothing user-visible to test

Prerequisites

  • GitHub App installed and connected to the repository.
  • A project linked to a GitHub repository.
  • For Vercel: connect Vercel and choose the single Vercel project this repo should test. Use Vercel, Coolify, or Custom infrastructure depending on how previews are delivered.

Supported providers

  • Vercel - triggered via GitHub deployment status events, then verified against the selected Vercel project. No webhook needed.
  • Coolify - via webhook configured in Coolify notifications.
  • Custom infrastructure - via group webhooks, triggered from your own CI (GitHub Actions, GitLab CI, Jenkins, shell scripts, etc.).

How it works

  1. A pull request is opened or updated.
  2. The deployment provider creates a preview and signals TesterArmy (via GitHub deployment status events for Vercel, or a webhook for Coolify).
  3. For Vercel, TesterArmy verifies that the deployment URL belongs to the selected Vercel project.
  4. TesterArmy resolves the PR from the commit SHA, checks the optional GitHub target-branch filter against the PR base branch, and triggers the configured test group and/or the exploration agent against the preview URL.
  5. Results are posted as a single comment on the PR.
  6. On subsequent pushes, the same comment is updated in place rather than creating a new one.

API-triggered exploration runs skip the deployment webhook steps and queue directly from POST /api/v1/projects/{projectId}/pull-request-runs. See Exploration agent.

Troubleshooting

No comment appears on the PR

  1. GitHub App not connected - verify installation in Project Settings.
  2. Permissions - the GitHub App needs Pull requests: Read & Write.
  3. No tests configured - if you’re relying on test groups, make sure one is enabled in the PR Testing tab.
  4. Vercel project not selected - open Integrations → Vercel and choose the Vercel project this repo should test.
  5. Target branch filter mismatch - open Integrations → GitHub and confirm the PR is targeting the configured base branch.

Tests run against the production URL

  1. Deployment not detected - TesterArmy only triggers on preview deployments. Verify the deployment shows up in your provider’s dashboard.
  2. Bypass token missing - if the preview URL returns an auth screen, add a bypass token.
  3. Wrong Vercel project selected - if the repo is linked to multiple Vercel projects, only the selected Vercel project is allowed to trigger tests.
  4. Wrong PR target branch - if a GitHub target branch is selected, only PRs targeting that base branch will trigger automatic runs.