HomeCasa Prompts

Analyze A/B test results and make a sound decision

Interpret an A/B test rigorously: sanity checks, conversion rates, lift, confidence interval and p-value, practical significance and a clear recommendation for your team. Designed for ChatGPT, Claude, Gemini and Microsoft Copilot. It includes 5 customizable variables and you can copy it for free in one click.

Advanced · Text · Pending editorial review

Designed for:

Prompt

Act as an experimentation analyst with a solid statistics background. You explain results so a non-technical team can decide, without overstating what the data says.

Test data:
- What was tested and the hypothesis: {{test_description}}
- Results per variant (users or sessions, and conversions): {{results}}
- Primary metric: {{metric}}
- Duration and dates: {{duration}}
- Agreed confidence level: {{confidence}}

If a variant's sample size, the randomization unit or the duration is missing, ask me before calculating.

Deliverables:
1. Sanity checks: sample ratio mismatch, full-week duration, changes made during the test, possible peeking and novelty effects.
2. Conversion rate per variant, absolute difference and relative lift.
3. A two-proportion test: z statistic, p-value and confidence interval for the difference, showing the formulas and steps.
4. Interpretation: statistical vs practical significance (is the change worth it?), and approximate power if the sample looks small.
5. Warnings about post-hoc segmentation and about secondary or guardrail metrics (for example, average order value or refund rate).
6. Recommended decision: ship, iterate or keep collecting data, with the reason and the risk of each option.
7. A 5-line summary for the team and a list of learnings for the next test.

Constraints:
- Do not invent numbers or segment results I have not given you: use [FILL IN].
- State the assumptions (independence, correct randomization) and that the calculations should be verified with a statistics calculator or code.
- If the data does not support a conclusion, say so clearly.

Format: a results table, a calculation block and a final summary.

How to use it

Export users and conversions per variant from your experimentation tool. Paste the original hypothesis too, so the interpretation sticks to what you set out to test.

Usage example

Test: green vs orange checkout button · Results: A 10,240 users and 512 purchases, B 10,180 users and 571 purchases · Metric: purchase conversion · Duration: 14 days.

Sample output

Sanity checks

  • Split 50.1% / 49.9%: no obvious sample ratio mismatch.
  • 14 days = two full weeks. ✔
  • Any changes mid-test or daily peeking? [FILL IN]

Results

VariantUsersPurchasesConversion
A (control)10,2405125.00%
B (reviews on top)10,1805715.61%

Absolute difference: +0.61 pp · relative lift: +12.2%

Two-proportion z-test

p_pooled = (512 + 571) / (10,240 + 10,180) = 0.0530
SE  = √[p(1−p)(1/nA + 1/nB)] = 0.00314
z   = 0.00609 / 0.00314 = 1.94
p   ≈ 0.052 (two-sided)
95% CI for the difference: −0.01 pp to +1.22 pp

Interpretation

At the agreed 95% confidence level, the result is not statistically significant: the interval includes zero, just barely. The signal is promising, but the data does not let you call B the winner.

Keep collecting data for one or two more full weeks with no changes, and check guardrails (average order value, refund rate). Shipping now means accepting the risk that the true lift is zero.

Assumes independent users and correct randomization. Verify with a statistics calculator or code. …

Key facts

Category
Data management · Data analysis
Output type
Text
Level
Advanced
Designed for
ChatGPT, Claude, Gemini and Microsoft Copilot
Verified in
Pending editorial verification
Variables
What was tested, Results per variant, Primary metric, Duration and Confidence level
Language
English
Price
Free, no sign-up
Updated
Nov 7, 2026

Frequently asked questions

What is sample ratio mismatch?

It is when the actual split of users between variants is far from the planned one (for example, 50/50). It usually points to a technical issue and can invalidate the result.

If the p-value is below 0.05, can I ship?

Not necessarily. The lift also has to matter for the business, the test must have run long enough and guardrail metrics should not get worse.

What is this prompt for?

Interpret an A/B test rigorously: sanity checks, conversion rates, lift, confidence interval and p-value, practical significance and a clear recommendation for your team.

Which AI tools does it work with?

It is designed for ChatGPT, Claude, Gemini and Microsoft Copilot. We have not verified it in these tools yet, so results may vary between them.

How do I customize the prompt?

Click “Customize” and fill in the variables (What was tested, Results per variant, Primary metric, Duration and Confidence level). The text updates instantly and you can then copy it in one click. Values are stored only in your browser.

Do I need experience to use it?

It is advanced: it is meant for people with experience in the topic who can evaluate and adjust the result.

Is this prompt free?

Yes. Every HomeCasa Prompts prompt is free, with no sign-up or subscription. You can also use them in professional projects.

Can I trust the AI output?

Treat it as a high-quality draft: always check data, figures and important claims before publishing or making decisions.

Related prompts