Test Plan & Test Runner Coming Soon
Structured test management with hierarchical plans, sprint cycles, and live Playwright execution.
Two Screens, One Mission
Test Plan and Test Runner are companion screens that cover the plan → execute → verify phase of the QA cycle.
- Test Plan is where you organise what needs to be tested — hierarchical plans, suites, cases, coverage metrics, and sprint cycles.
- Test Runner is where you execute those tests — on-demand runs, scheduled runs, live log streaming, and retry configuration.
They work together: a run triggered in Test Runner is linked back to the plan, and its results update the plan's coverage metrics automatically.
Test Plan — Planned Features
Test Plan provides a structured place to define what your team needs to test, track coverage, and organise work into sprint-level test cycles.
Hierarchical Organisation
Three-level hierarchy: Plan → Suite → Case. A plan groups related suites (e.g. "Checkout Flow"). A suite groups related cases (e.g. "Payment Methods"). Cases are individual test scenarios.
Coverage Tracking
Each plan shows a live coverage breakdown: pass / fail / skip counts and percentages. Drill down to suite or case level to find gaps. Coverage targets can be set per plan.
Sprint Cycles
Group test cases into sprint-level cycles. Assign cases to a sprint, mark them as planned or in-progress, and track completion as the sprint runs. Cycle results carry over to the plan's overall coverage.
Plan Structure
Each test case contains:
- Title — Short description of what is being tested
- Preconditions — Setup state required before the test can run
- Steps — Numbered action steps with expected results
- Expected result — What a passing outcome looks like
- Priority — Critical / High / Medium / Low
- Linked test file — Optional pointer to the Playwright
.spec.tsfile (enables automated execution from Test Runner) - Status — Not Run / Pass / Fail / Blocked / Skip
Coverage Metrics
At the plan level, coverage is summarised as:
| Metric | Meaning |
|---|---|
| Pass % | Cases with status Pass divided by total cases |
| Fail % | Cases with status Fail — needs investigation |
| Skip % | Cases intentionally skipped (e.g. pending feature) |
| Blocked % | Cases that cannot run due to an upstream dependency or blocker |
| Not Run % | Cases not yet executed in the current cycle |
Test Runner — Planned Features
Test Runner executes Playwright tests from within TotalApp without requiring a terminal or CI trigger. It is designed for on-demand spot checks, pre-merge smoke runs, and scheduled regression suites.
On-Demand Execution
Select a plan, suite, or individual case and run it immediately. Choose target browsers, worker count, and retry settings from the run config panel.
Scheduled Runs
Define a cron schedule for nightly or weekly regression suites. Scheduled run history is stored and linked back to Test Plan results.
Live Log Streaming
stdout and stderr from the Playwright process stream in real time with colour highlighting. PASS lines are green, FAIL lines are red, slow steps are amber — at a glance without scrolling.
Retry on Failure
Configure the number of automatic retries (0–3). Each retry attempt is logged separately. Retry data feeds directly into Flake Tracker's retry-rate calculation.
Screenshot on Fail
Automatically capture a full-page screenshot when a test step fails. Captured screenshots are sent immediately to Artifact Vault — no manual upload needed.
Run Configuration Options
| Option | Values | Default |
|---|---|---|
| Target browsers | Chromium, Firefox, WebKit (multi-select) | Chromium |
| Workers | 1 – 8 (or auto) | Auto (CPU count / 2) |
| Retries | 0 – 3 | 0 |
| Screenshot on fail | On / Off | On |
| Trace on fail | On / Off / Always | On |
| Timeout per test | 5s – 120s | 30s |
How Test Plan and Test Runner Work Together
- Create a plan in Test Plan and populate it with suites and cases. Link each case to its Playwright spec file if it has one.
- In Test Runner, select the plan and click Run. Configure browsers, workers, and retry settings.
- Watch live output stream in the log panel. Failed tests are highlighted immediately.
- When the run completes, Test Plan's coverage metrics update automatically — no manual status entry needed for linked cases.
- Failure screenshots and traces appear in Artifact Vault within seconds of the run ending.
- Retry rate data flows to Flake Tracker — tests that passed only after retries show an increased retry rate.