A/B Testing
Run controlled experiments on landing pages, emails, and ad copy to find what converts best.
Overview
A/B Testing in TotalApp lets your marketing team run structured experiments to answer a single question at a time: which version of a headline, button, image, or email subject converts better? Instead of guessing, you split your audience between a control (A) and one or more test variants (B, C...) and let the data decide.
Each test tracks its success metric in real time, calculates statistical significance automatically, and highlights the winning variant once enough data has been collected. A two-click "Declare Winner" flow ends the test and applies the winning variant to your live content.
Creating a Test
Click New Test in the toolbar to open the test setup form. Complete the fields below before adding variants.
| Field | Description |
|---|---|
| Test Name | Required. A short identifier, e.g. "Homepage CTA — June 2026". |
| Hypothesis | Your prediction, e.g. "A blue CTA button will have a higher click rate than the current orange button." Writing the hypothesis first prevents post-hoc rationalisation of results. |
| Element Being Tested | Headline, CTA Button, Hero Image, Email Subject Line, or Custom. Determines which editor fields appear in the Variants tab. |
| Traffic Split | Percentage of your audience assigned to the test variants. The remainder sees the control. Default: 50/50 for two variants. For three variants: 33/33/34. |
| Success Metric | The primary KPI the test optimises for: Click-Through Rate, Conversion Rate, Revenue Per Visitor, or Open Rate (for email subject tests). |
| Minimum Sample Size | The number of impressions per variant before significance is calculated. Default: 500. Lower values produce unreliable results. |
Variants
After creating the test, open the Variants tab to edit each variant's content. Variant A is always the control — it contains the current live version. Variant B (and C, D...) contains the challenger content you want to test.
The existing live version. Never modify Variant A during a running test — changes invalidate the baseline.
The version you believe will perform better. Change only the element being tested — keep everything else identical to A.
Click Preview on any variant to see exactly what each audience segment will see. Confirm both previews look correct before activating the test.
Good Testing Hygiene
Change only one element per test. If you test a new headline and a new image at the same time, you will not know which change caused the result. One variable per test is the golden rule of A/B testing.
Results
Once a test is active, the Results tab shows a live table updating with each new impression and conversion recorded. Columns in the results table:
| Column | Description |
|---|---|
| Variant | A (Control), B, C, etc. |
| Impressions | Number of users who saw this variant. |
| Conversions | Number of users who completed the success metric action. |
| Conversion Rate | Conversions ÷ Impressions, as a percentage. |
| Uplift vs Control | How much better (or worse) this variant is performing vs Variant A, as a percentage. |
| Statistical Significance | Confidence that the result is real and not due to chance. Displayed as a percentage (e.g. 87%). The winning variant is highlighted once significance reaches ≥ 95%. |
Conversion Rate: 4.8% — Uplift: +24% vs Control — Significance: 97%
Declaring a Winner
When statistical significance reaches 95% or above, a Declare Winner button appears on the winning variant row. Clicking it does two things:
- Ends the test and stops assigning users to variants. All future users see the winning version.
- Applies the winning variant's content to the live element (headline, button, email subject, etc.) if the integration is connected. For email subject tests, the winning subject is applied to future sends of that campaign automatically.
You can also end a test early by clicking Stop Test without declaring a winner — this preserves the data in Test History but does not apply any variant.
Test History
All past tests — whether a winner was declared or the test was stopped early — are saved in the History tab. Each entry shows the test name, dates, success metric, total impressions, winning variant (if applicable), and the final conversion rates for each variant.
Use the History tab to build an institutional knowledge base of what your audience responds to. Before starting a new test on the same element, check the history to avoid re-testing a hypothesis you already answered.
AI Assistant
The Experimentation Station includes an embedded AI Assistant panel that reads the current experiment queue — name, target metric, status — and, for the experiment you have selected, its variant matrix (impressions, conversions, conversion rate, traffic allocation), p-value, statistical significance, confidence interval, and declared winner. It answers questions about your tests directly on the screen.
Queue Summary
Ask for a plain-language summary of what the current set of experiments says about testing velocity and coverage.
Significance Triage
Ask whether the active experiment has reached statistical significance yet, and what the current p-value and sample size imply.
Winner Recommendation
The assistant suggests which variant should be declared the winner, and flags the risk of deciding too early if the sample size is still insufficient.
Risk Flagging
Ask which experiments are unlikely to reach a conclusive result given their current traffic allocation and impression counts.
Open the panel from the vertical AI Assistant tab on the right edge of the screen, or the Sparkles button in the panel header once open. It uses the Writer Engine configured in Settings → General → Writer Engine (API, Local CLI, or in-browser engines) — no per-screen configuration is required.