CI/CD Pipelines

Introduction

What is the CI/CD Pipelines screen?

The CI/CD Pipelines screen (at /pipelines) is Testver’s CI/CD control plane. It connects your build pipelines to your test suites so that every push, pull request, or deploy can automatically run tests on your Testver server — and mark the build green or red based on the result.

The key idea is delegated execution. Your CI/CD platform does not install browsers, frameworks, or dependencies, and it does not run the tests. It makes one API call to Testver, waits for the verdict, and passes or fails the build accordingly. Testver does the actual work on its own machine, exactly the way it does when you click Run in the UI.

Tip

Analogy: think of your CI pipeline as a restaurant waiter and Testver as the kitchen. The waiter doesn’t cook — they take the order to the kitchen, wait for the dish, and bring back the result. Everything that makes the kitchen good (your browsers, your environments, your artifacts, the AI that fixes broken tests) stays in the kitchen where it belongs.

Note

In one sentence: the CI/CD Pipelines screen lets your CI/CD pipelines trigger, wait for, and gate on test runs that execute on your Testver server, with the results landing in Testver’s normal dashboards.

Why delegated execution?

Running browser tests on CI runners is the usual approach, and it has real costs: every job re-installs Node, browsers, and dependencies; runs are slow; and the rich output — screenshots, videos, traces, healing data, AI failure analysis — either doesn’t exist or gets thrown away with the runner.

Because Testver executes the tests itself, every CI-triggered run is a normal Testver run. It appears in Results with full artifacts, feeds the trend charts, can be compared against a baseline, and can be repaired with Ask AI to Fix → Apply & Self-Heal — none of which is possible when the run vanishes with an ephemeral CI container.

Who this guide is for

• QA engineers who want a pull request to fail when a regression suite fails, without maintaining CI YAML.
• Developers who want the PR to show a test verdict, and a link to the full report when it’s red.
• DevOps / build engineers who want a thin, portable CI job instead of a per-project browser-install pipeline.
• Test leads who need to see pass-rate trends per pipeline, branch, and environment — and trace a deploy back to the tests that gated it.

Where to find it

CI/CD Pipelines sits in the left navigation under Test Automation, between Manage and Results, marked with a branch icon.

The active tab is stored in the address bar, so /pipelines?tab=ci-runs opens straight to the CI Runs tab. Filters are stored there too, which makes any filtered view — for example “all failed runs of the pr-gate-main pipeline” — safe to bookmark and share.

The five tabs

Each tab has its own guide:

Tab	Guide	What you do there
Dashboard	Dashboard tab	See CI health: KPIs, a verdict trend, and insights per pipeline, branch, and environment.
Pipelines	Pipelines tab	Create pipeline mappings and generate the CI config for each one.
CI Tokens	CI Tokens tab	Create and revoke the machine tokens CI jobs authenticate with.
CI Runs	CI Runs tab	Browse and filter the full history of CI-triggered runs.
Trace	Trace tab	Find every run that gated a given commit or branch.

Key terms (glossary)

Term	What it means in Testver
Delegated execution	Tests run on the Testver server; the CI platform only triggers the run and gates the build on the result.
Pipeline mapping	A saved rule: “when this pipeline triggers, run this suite / cycle / command with these settings.” Its name is the trigger key.
Trigger key	The mapping name your CI job (or the CLI) passes to identify what to run.
CI token	A long-lived machine token (tvci_…) used by CI jobs and the CLI to authenticate to the Gate API. Shown once at creation.
Gate API	The endpoints CI calls: trigger a run, poll its status, cancel it.
Gate mode	Whether the CI job waits for the verdict (exit code gates the build) or fires and forgets (a commit status carries the verdict).
CI run	One trigger, tracked by Testver. It points at the underlying test run that actually executed.
Queue position	Where a triggered run sits in Testver’s global queue. CI runs execute one at a time so bursts queue instead of overloading the server.
Commit sync	Optionally checking out the exact commit the pipeline is building before running, then restoring the previous branch.
Status push-back	Testver posting a commit status to your git host so pull requests show the verdict.

How it works

1. Your pipeline runs a small job that calls Testver’s Gate API, passing the mapping name plus the branch, commit, and build metadata.
2. Testver records a CI run and places it in a global queue. It responds immediately with a run id, a report link, and (if the server is busy) a queue position.
3. When a slot frees up, Testver optionally checks out the triggering commit, then executes the mapped suite, cycle, or command exactly as a manual run — same environments, browsers, artifacts, and reporting.
4. The CI job polls until the run reaches a terminal state, prints the summary and report link, and exits 0 or 1 so the build is marked pass or fail. (In fire-and-forget mode the job exits immediately and a commit status delivers the verdict instead.)
5. The finished run appears in Results with a CI badge and a link back to the CI build.

Getting Started

Prerequisites

• Your Testver server must be reachable from your CI runners over HTTPS. A server on localhost works with self-hosted runners on the same network, but cloud runners (GitHub-hosted, GitLab.com shared) need a publicly reachable address.
• The CI runner needs curl and jq. The generated configs install them where needed (GitLab and Bitbucket use alpine + apk, CircleCI installs jq); on Jenkins agents you may need to install them yourself.
• Something to run: a Suite (Test Explorer → Suites), a Test Cycle with mapped automation (Test Plans & Cycles), or a run command.

Set up your first gate in four steps

1. Create a pipeline mapping — on the Pipelines tab, click New Mapping, give it a name (this is the trigger key, e.g. pr-gate-main), and pick what it should run.

2. Create a CI token — on the CI Tokens tab, name a token and click Create Token. Copy the value immediately; it is shown only once. Store it as a secret in your CI platform (conventionally TESTVER_CI_TOKEN).

3. Generate the pipeline config — back on the Pipelines tab, click the config icon on your mapping, pick your platform, and click Generate Config. Save it to your project, then commit and push it.

4. Trigger it — push a commit, or try it from any terminal first:

   testver trigger --server https://testver.your-company.com \
     --mapping "pr-gate-main" --token tvci_… --wait

Triggering From Anywhere (CLI)

The Gate API is not CI-only. The same trigger-wait-gate flow ships as a terminal command, usable from any machine that can reach your Testver server — a laptop, a cron job, a git hook, or another team’s tooling.

# Gate on a saved mapping (waits, exits 0 = passed, 1 = failed)
testver trigger --server https://testver.your-company.com --mapping "pr-gate-main" --wait

# Target a suite, cycle, or command directly
testver trigger --suite "Regression Suite" --env staging --wait
testver trigger --cycle <cycleId> --wait
testver trigger --command "npx playwright test --grep @smoke" --wait

# Fire and forget — prints the run id and report URL immediately
testver trigger --mapping "nightly" --no-wait

# Chain it: deploy only if the smoke gate passes
testver trigger --mapping "smoke" --wait && ./deploy.sh

Flag	Purpose
--server	Testver base URL (or set TESTVER_SERVER).
--token	CI token (or set TESTVER_CI_TOKEN).
--mapping / --suite / --cycle / --command	What to run.
--env, --browser	Trigger-time overrides.
--sha, --branch, --build-url, --build-number	Trigger metadata, used for commit sync, status push-back, and tracing.
--wait / --no-wait	Wait for the verdict (default) or return immediately.
--json	Machine-readable output for scripting.
--timeout	Seconds to wait before giving up and cancelling the remote run (default 3600).

Pressing Ctrl-C while waiting cancels the remote run as well, so an abandoned gate doesn’t keep occupying the queue.

Results Integration

CI-triggered runs are ordinary Testver runs. In Results they carry a CI badge from the moment they start, and the run header shows CI · <pipeline> #<build> plus a CI build link back to the pipeline. Everything else behaves exactly as for a manual run: live progress while executing, per-test detail, screenshots and videos, console output, trends, comparison and baselines, AI Failure Analysis, and Ask AI to Fix.

Common Tasks (How Do I…?)

I want to…	Do this
Gate a GitHub pull request on a suite	Create a mapping targeting the suite → create a CI token → Wire into CI → GitHub Actions → Wait & gate → save, commit, and add the token as a repo secret. See the Pipelines tab.
Run a release cycle’s automation from CI	Create a mapping with the Cycle target and wire it into your release pipeline.
Use different settings per pipeline	Add Environment / Browser overrides on the mapping instead of duplicating the suite.
Make sure the gate tests the built commit	Enable commit sync on the mapping.
Show the verdict on a PR without holding a CI runner open	Enable status push-back on the mapping, then generate the config in Fire-and-forget mode.
Trigger a run from a shell script	testver trigger --mapping "…" --wait — it exits 0 on pass, 1 on fail.
Stop a running CI run	Cancel the CI build (the job propagates it), press Ctrl-C in the CLI, or DELETE /api/ci/runs/:id.
See only failed runs of one pipeline	On the Dashboard, click that pipeline in Insights, then filter state to failed.
Find what tested a deploy	Trace tab → paste the deployed commit SHA.
Retire a pipeline safely	Disable the mapping (triggers are rejected) or revoke its token, then remove the CI config.

Tips & Best Practices

• One mapping per pipeline purpose. A pr-gate on a fast smoke suite and a nightly on the full matrix beats one mapping you keep re-configuring.
• Enable commit sync on gating pipelines. Without it the gate tests whatever the workspace currently has checked out, which may not be the commit being built.
• Use fire-and-forget for long suites. A 40-minute suite holding a hosted CI runner open costs CI minutes; a commit status delivers the same verdict for free.
• Name mappings after the pipeline, not the suite. The name is a contract with your CI config — pr-gate-main survives a suite rename; regression-suite doesn’t.
• Keep the queue in mind. CI runs execute one at a time. If several pipelines fire together, later ones queue — expected behavior, and the queue position appears in the CI log.
• Test the mapping from a terminal first. testver trigger … --wait proves the mapping, token, and connectivity before you debug it through a CI job.

Troubleshooting & FAQ

Symptom	Likely cause / fix
Invalid or revoked CI token (401)	The token is wrong, revoked, or the secret isn’t reaching the job. Re-create it on the CI Tokens tab and update the CI secret.
No pipeline mapping named "…" (404)	The mapping was renamed or deleted, or the CI config has a typo. Names must match exactly.
Pipeline mapping "…" is disabled	The mapping’s Enabled checkbox is off.
Run sits in queued for a long time	Another CI run is executing — runs are serialized. The position is shown in the log; it starts automatically when a slot frees.
Run ends in error mentioning a busy target	The suite or cycle was already running (for example started from the UI) and stayed busy past the retry window. Re-trigger once it’s free.
CI job can’t reach Testver	Cloud runners can’t see a localhost or LAN-only server. Expose Testver over HTTPS or use self-hosted runners on the same network.
Jenkins job fails at the first command	The agent is missing curl or jq. Install them, or run the stage in a container that has them.
A queued run reports “lost by a server restart”	Queued runs live in memory; a restart clears them. Trigger again.
CI runs don’t appear in Results	They do — look for the CI badge. If a run shows no badge, it may have been triggered before this feature was enabled.

Related

• Dashboard tab · Pipelines tab · CI Tokens tab · CI Runs tab · Trace tab
• Results — where every CI-triggered run lands.
• Test Explorer → Suites — build the suites pipelines target.
• Test Plans & Cycles — cycles you can gate a release on.
• Schedules — time-based runs, the complement to pipeline-triggered ones.