Auto-Heal

Introduction

What is Auto-Heal?

Auto-Heal repairs the most common cause of flaky-looking failures: a locator that no longer matches the page because the app changed — a renamed data-testid, a changed id or class, a label that moved. After a run finishes with failures, Auto-Heal inspects the evidence the run left behind, finds the broken locator in your source, proposes an exact replacement, re-runs the affected tests to prove it, and only then keeps the change (or suggests it for your review). If verification fails, every file is restored — nothing half-fixed is ever left behind.

Note

In one sentence: when the app’s UI changes and a locator breaks, Auto-Heal fixes the locator in your code from real page evidence and proves the fix by re-running the tests — or touches nothing.

Three principles shape everything below:

1. Evidence over guessing. A fix is proposed only from what the failing page actually contained — a trace, a captured DOM, or a live probe. No evidence, no proposal; the test is skipped with a plain-language reason instead.
2. Verify or revert. Every applied fix is validated by re-running all tests affected by the same broken locator. All pass → the fix stands. Any fail → every file is restored and the attempt is recorded as reverted.
3. Your framework stays yours. Fixes are exact find→replace edits to the locator strings already in your code — no wrappers, no runtime hooks, no framework changes. Every change is visible, attributable, and undoable.

Who this guide is for

• Teams whose apps evolve fast — locator churn eats maintenance time; Auto-Heal turns most of it into a one-click review.
• CI owners who want a red build caused by a rename to fix itself (optionally re-greening the build) instead of paging someone.
• Anyone evaluating healing tools who has been burned by magic that rewrites tests invisibly — Auto-Heal is deliberately the opposite.

Key terms (glossary)

Term	What it means here
Suggest mode	Auto-Heal finds and verifies a fix, then restores your files and presents the change in Results for one-click Apply. Recommended starting point.
Auto mode	Verified fixes are kept in your code automatically (still shown in Results, still undoable).
Heal signal	The failure patterns eligible for healing: locator-not-found, strict-mode violations, element-interaction failures — including Selenium explicit-wait timeouts (“waiting for element located by…”). Assertion failures, network errors, etc. are never touched.
Group heal	All tests failing on the same broken locator heal together: fix once, re-run all of them, keep only if all pass.
Evidence	What the fix is derived from: Playwright trace DOM, an ARIA snapshot, a live session capture, or a live browser probe.
Session capture	Reading the page through the test’s own browser session at failure time — logged in, exact failure state. Works for local drivers, remote grids, and cloud runs.
Live probe	An opt-in last resort: Testver opens the page fresh in a headless browser to read its current DOM. Only useful when the page is reachable and doesn’t require the test’s session state.
Attempt cap & cooldown	A permanently broken test can’t loop: attempts are spent only on real cost (AI call, capture re-run, apply+verify), capped, then cooled down for a configurable number of hours. A successful heal resets the counter.
Stamp	The record attached to healed/skipped tests in Results — what changed, from what evidence, at what confidence, with Apply/Undo.
Review window	How long Apply / Undo stay offered on a fix (Settings → Auto-Heal; 1–30 days, default 7). Past it the fix reads Expired.
Expired / Stale	A fix whose actions have retired: Expired = older than the review window; Stale = your code changed, so the recorded edit no longer matches. Both keep their history row; neither loses you anything (a fresh fix regenerates on the next failure).
CI gate mode	What a heal means for a CI-gated build: Report only (default — the build’s verdict never changes) or Update build (a fully healed run flips the gate to passed, labeled “auto-healed & verified”).

Turning it on

Settings → General → Auto-Heal:

Setting	Values	Notes
Mode	Off / Suggest / Auto	Default Off. Start with Suggest.
Max attempts	number	Attempts per test before cooldown; spent only on real cost — free skips never count.
Cooldown	hours	How long a capped test rests before Auto-Heal tries again.
CI gate	Report only / Update build	Per-workspace default; pipeline mappings can override per pipeline.
Review window	1–30 days (default 7)	How long Apply / Undo stay offered on a fix — see When fixes retire.
Live browser probe	On / Off	Opt-in. Lets non-Playwright and cloud runs heal even when no session capture exists — see evidence tiers below.
Remote hub URL	URL	Only needed for self-hosted Selenium Grid or frameworks with a hardcoded hub: point them at Testver’s relay (http://localhost:4499) and session capture works for remote browsers. Cloud providers configured in Connectors need nothing here.

Scopes can override the workspace default either way:

• Suites — each suite can set Inherit / Off / Suggest / Auto (Suites → suite settings).
• CI pipelines — each mapping can set Inherit / Off / Heal only / Heal + re-gate (the last forces Auto mode and Update-build for that pipeline).

How it works, step by step

1. Trigger — strictly after a run completes and is saved. Nothing runs mid-test; Auto-Heal never holds browsers open.
2. Eligibility — only failures with a heal signal qualify. Everything else is left for Failure Analysis.
3. Locate first — before spending anything, Auto-Heal finds the broken locator as a literal in your source (often a page object, not the test file). Dynamic or data-driven locators that don’t exist as literals are refused with a clear reason — that’s a human decision, use Ask AI to Fix.
4. Gather evidence, cheapest first:
   • Playwright: the failure’s ARIA snapshot and the run’s trace (the trace’s captured HTML holds every attribute on the page — including all data-testids that existed).
   • Selenium / WebdriverIO / Robot: the shadow session capture — from run start, Testver quietly follows the test’s own browser (local driver, remote hub via the relay, or a cloud session it observed) and keeps the last page states. At failure time it holds the page as the test saw it: logged in, mid-flow.
   • Cloud SDK runs: sessions are discovered from the vendor SDK’s own journal and captured through the hub the same way.
   • Live probe (opt-in): if nothing above produced evidence, Testver opens the page fresh — URL discovered from the trace, the code, the cloud session log, or the project’s config files — and can follow one link toward the failing page.
5. Propose — deterministic tiers first (free): exact rename detection for test ids, shape-preserving fixes for CSS id/class/attribute changes, ARIA-based re-anchoring; the variable name holding the locator is used as context (this.emailInput strongly suggests email-input). Only when deterministic tiers can’t decide does a single bounded AI call run, fed the same evidence.
6. Apply + verify — the fix is applied as an exact find→replace (exactly once per file, backed up), then every affected test re-runs headlessly through the normal runner. All pass → kept (Auto) or restored-and-suggested (Suggest). Any fail → full revert, recorded.
7. Record — the run and each test get a stamp visible in Results.

What it will and won’t touch

Heals	Never touches
getByTestId / getBy* calls (whole-call replacement)	XPath locators (different matching semantics — refused by design)
CSS selectors (ids, classes, attributes)	By.id / By.name (value-swap semantics differ)
Selenium By.cssSelector failures, incl. explicit-wait timeouts	Assertion values, test data, URLs, waits/timeouts
WebdriverIO and Robot Framework css locators	Anything without page evidence to justify it

Framework support

Framework	Failure detection	Evidence source
Playwright (JS/TS)	✅	Trace DOM + ARIA snapshot (own artifacts — richest evidence, works everywhere Playwright runs)
Selenium (Java, Python, C#, Ruby, Kotlin)	✅	Shadow session capture (local, Grid via relay, cloud) → live probe
WebdriverIO	✅	Session capture → live probe
Robot Framework (SeleniumLibrary, css)	✅	Session capture → live probe
Cypress	✅	Live probe only (Cypress is not WebDriver — no session to capture)

Cloud runs heal too: Testver observes the cloud session (through its proxies or the SDK’s journal) and captures the page through the vendor’s hub — same evidence quality as local. See Cloud Grid Execution.

What you see on a run

• Run header chip — a heal summary for the run (healed / fix ready / reverted counts).
• Per-test panel — for a healed test: the exact change ([data-testid='username'] → [data-testid='username-input']), the evidence source (Trace match / Live session capture / Live browser probe / AI), a confidence value, and Apply (Suggest mode) or Undo (kept fixes).
• Skip reasons — when Auto-Heal chose not to act, the test says why in plain language (no literal in source, no page evidence, attempt cap cooling down…), with an Analyze failure with AI button as the manual path.

Reviewing fixes — the Auto-Heal tab

Results has a dedicated Auto-Heal tab (/results?view=auto-heal) covering every repair across all runs. It is a work queue first and a log second.

• Overview tiles — total repairs, kept, fix ready, reverted, undone, and the share of repairs solved with no AI at all. Fix ready counts only the fixes you can still act on (see When fixes retire).
• Fixes ready for review — the queue: each verified fix with its test, file, evidence source, confidence, and when it was prepared.
• Locator hotspots — tests healed two or more times. Repeated healing means the app is changing faster than that test’s locator strategy; each chip opens the test in Test Explorer.
• Activity — the full history, filterable by state (All / Kept / Fix ready / Reverted / Undone / Expired / Stale), by evidence source, and by free-text search across test names, locators, and files. Ten rows per page.

Click any row to expand it. You get the whole story in one place: what the state means in plain language, the matcher’s or AI’s rationale, the exact before/after diff per file, source + confidence + timestamp, how many tests that single change healed together, and the actions — Apply fix, Undo this fix, Open test, Open run, Open verification run. The same panel appears on a test’s detail page inside a run, so a fix reads identically wherever you meet it.

Applying and undoing

• Apply writes the recorded edit into your files and marks the fix Kept. The same locator change is often suggested by several runs (a test fails repeatedly before anyone reviews it) — applying once retires every sibling suggestion of the identical change, so the queue never offers work that is already done.
• Undo swaps the healed locator back and marks the record Undone. It asks for confirmation first, because it is a one-way door: the affected tests will most likely fail again, and Testver never re-offers Apply for an undone fix — only a new failure produces a new suggestion. After that, version control is the way back.
• Both actions re-check the recorded edit against your current files first, so neither can corrupt code that has moved on.

When fixes retire

A fix is only worth offering while it can still be applied cleanly. Two rules retire the buttons:

State	Meaning	What to do
Expired	Older than the review window (Settings → Auto-Heal; 1–30 days, default 7).	Nothing — a fresh fix is suggested from current page evidence the next time the test fails.
Stale	The code changed since the fix was prepared, so the recorded edit no longer matches your files — often because a related fix already rewrote that locator.	Nothing — same as above.

Retired entries keep their row in Activity with the reason spelled out; only the actions go. Both rules are enforced on the server, so the tab, the per-test panel, and the API can never disagree about whether a fix is applicable.

Auto-Heal and CI

With a pipeline gated on Testver (delegated execution):

• Report only (default): heals happen and are reported, but the build verdict is untouched — a red run stays red until a human applies/reviews.
• Update build: the gate holds while healing is in progress; if all failed tests heal and verify, the build flips to passed with the status “auto-healed & verified” pushed to your CI provider. Anything less than fully-healed stays failed.
• CI Runs rows show the heal detail per run (”✦ N auto-healed & verified” / “N fix ready” / “N reverted”).
• Suggest mode never flips a gate, by design — only kept (Auto/Heal + re-gate) fixes can.

Tips for heal-friendly tests

• Name locator variables meaningfully. this.emailInput = page.getByTestId('…') gives the matcher real context; this.el3 gives it nothing.
• Prefer css/test-id locators for Selenium — XPath is deliberately un-healable.
• Remember: a kept heal is a permanent code edit. It’s in your working tree — commit it like any other change. (Re-testing a heal scenario requires re-breaking the locator first.)
• Keep traces on for Playwright — the trace is the best evidence Auto-Heal ever gets.
• The probe is a fallback, not the plan. Pages behind logins or bot protection show the probe the wrong content; session capture (automatic for WebDriver-family runs) doesn’t have that problem.

Troubleshooting

Symptom	Meaning	Fix
”not found as a literal in any source file”	The locator is built dynamically (template/data-driven) — healing it is a judgment call.	Use Analyze failure with AI on the test.
”no page evidence was captured”	The session ended before its DOM could be read and no other evidence existed.	Enable the live probe, or for remote frameworks point them through the relay hub URL.
”probe enabled but no page URL could be discovered”	Neither the trace, the code, the cloud session log, nor config files revealed the app URL.	Ensure the base URL appears in the project config, or rely on session capture instead.
”attempt cap reached, cooling down”	Repeated failed attempts on this test — likely not a locator problem at all.	Investigate manually; the counter resets after the cooldown or a successful heal.
Heal reverted	The proposed fix didn’t make all affected tests pass — so it was undone completely.	Check the panel’s rationale; the failure may need a human fix.
A fix vanished from the queue after I applied another one	They were the same locator change suggested by different runs — applying once retires all its siblings.	Nothing; the change is in your code. The entries stay in Activity as Kept.
A fix shows Stale and offers no buttons	Your files changed since it was prepared — often a related fix already rewrote that locator, so this edit can no longer land.	Nothing; a fresh fix is suggested the next time the test fails.
Build went green “by itself”	In Update-build mode, a fully-healed-and-verified run legitimately flips the gate — the CI status says “auto-healed & verified”.	Review the heal in Results; Undo if you disagree.

FAQ

Will it change my tests without telling me? No. Suggest mode changes nothing until you click Apply. Auto mode keeps verified fixes but every one is visible in Results with one-click Undo, and the edit exists in your working tree like any other code change.

Can I redo an undo? Not from Testver — an undone fix is never re-offered (the confirmation dialog says so before you commit to it). Auto-Heal will propose a new fix the next time the test fails, and your version control holds the change you reverted.

Does it use AI on every failure? No — deterministic matching from page evidence handles the common cases free; the AI tier runs once, only when needed, and only when evidence exists to feed it.

What does verification cost? A headless re-run of the affected tests (on cloud runs, cloud minutes). Attempts are capped and cooled down, and are only spent on real cost — skips are free.

Can it heal a test that fails behind a login? Yes, when the evidence is a session capture (the test’s own logged-in browser). That’s the default for WebDriver-family runs; only the probe fallback is limited to reachable pages.