Before/after analysis: how to compare app versions fairly

Maya

v4.9.0 went out on Monday, 11 May, with the new first-expense flow. My dashboard says week-2 retention fell from 29% to 15%. Did we break something?

You

Can't we just split today's users by version? Everyone on 4.9.0 against everyone still on 4.8.0: same week, same weather.

Alex Tutor

That's the tempting one, and it's tilted before you start. Nobody drew lots for the update. People who open Halves every few days get 4.9.0 within a day or two, often automatically; people who drifted away sit on 4.8.0 for weeks. The 4.9.0 side is packed with your most loyal users.

You

So the new version would “win” even if it changed nothing.

Alex Tutor

Yes: the groups picked themselves. This is an observational comparison, not a coin flip, so fairness has to be built in. Take new users grouped by sign-up date: a newcomer installs whatever is current, so the date decides the version, not their enthusiasm. Then two more rules: the same fixed window for every cohort, and only cohorts that have lived all of it.

Alex Tutor

Question 1

Which comparison of v4.8.0 and v4.9.0 is fair enough to read?

Choose one answerThis week's users on v4.9.0 vs this week's users still on v4.8.0All active users in May vs all active users in AprilNew users grouped by sign-up date, before vs after 11 May, active in days 8–14, complete cohorts onlyNew users before vs after 11 May, “returned at any time since sign-up”

Alex Tutor

Here are daily sign-up cohorts from 13 April to 31 May, pulled on the morning of 1 June. Retention means opening the app on at least one of days 8–14 after sign-up; the sign-up day is day 0.

Alex Tutor

Dataset · 49 rows

Daily sign-up cohorts around v4.9.0

New Halves users from 13 April to 31 May 2026, one row per sign-up day. v4.9.0 was released on 11 May. Data pulled on the morning of 1 June.

Download CSV

Columns
signup_date: Sign-up day (day 0)
app_version: Version new users installed that day
days_observed: Days after the sign-up day seen by 1 June
new_users: New users that day
returned_any_time: Opened the app on any later day up to 31 May
active_days_8_14: Opened the app on at least one of days 8–14 (so far)

Data

signup_date	app_version	days_observed	new_users	returned_any_time	active_days_8_14
2026-04-13	4.8.0	48	72	42	15
2026-04-14	4.8.0	47	75	52	23
2026-04-15	4.8.0	46	70	51	25
2026-04-16	4.8.0	45	77	56	31
2026-04-17	4.8.0	44	70	50	16

Show 44 more rows

2026-04-18	4.8.0	43	105	82	26
2026-04-19	4.8.0	42	97	65	33
2026-04-20	4.8.0	41	76	41	16
2026-04-21	4.8.0	40	75	47	21
2026-04-22	4.8.0	39	66	53	21
2026-04-23	4.8.0	38	76	54	19
2026-04-24	4.8.0	37	72	44	16
2026-04-25	4.8.0	36	88	58	27
2026-04-26	4.8.0	35	94	74	31
2026-04-27	4.8.0	34	69	37	23
2026-04-28	4.8.0	33	74	51	23
2026-04-29	4.8.0	32	72	54	29
2026-04-30	4.8.0	31	69	47	24
2026-05-01	4.8.0	30	66	43	18
2026-05-02	4.8.0	29	101	59	22
2026-05-03	4.8.0	28	97	65	34
2026-05-04	4.8.0	27	69	44	22
2026-05-05	4.8.0	26	70	38	19
2026-05-06	4.8.0	25	69	52	21
2026-05-07	4.8.0	24	78	46	18
2026-05-08	4.8.0	23	68	47	14
2026-05-09	4.8.0	22	105	66	27
2026-05-10	4.8.0	21	89	58	24
2026-05-11	4.9.0	20	78	41	23
2026-05-12	4.9.0	19	70	40	23
2026-05-13	4.9.0	18	76	50	26
2026-05-14	4.9.0	17	71	46	20
2026-05-15	4.9.0	16	74	48	24
2026-05-16	4.9.0	15	94	51	27
2026-05-17	4.9.0	14	88	59	29
2026-05-18	4.9.0	13	76	51	24
2026-05-19	4.9.0	12	79	46	18
2026-05-20	4.9.0	11	76	50	14
2026-05-21	4.9.0	10	74	44	11
2026-05-22	4.9.0	9	75	39	9
2026-05-23	4.9.0	8	93	53	5
2026-05-24	4.9.0	7	87	51	0
2026-05-25	4.9.0	6	66	30	0
2026-05-26	4.9.0	5	76	24	0
2026-05-27	4.9.0	4	76	23	0
2026-05-28	4.9.0	3	73	25	0
2026-05-29	4.9.0	2	79	14	0
2026-05-30	4.9.0	1	96	0	0
2026-05-31	4.9.0	0	89	0	0

You

A cohort from 25 May has days_observed = 6. Its day 8 hasn't even happened, so its zero means nothing.

Alex Tutor

Exactly. And a cohort with 10 days observed has seen only part of its window. It needs days_observed of at least 14.

Alex Tutor

Question 2

How many v4.9.0 cohorts have lived their full days 8–14 by the morning of 1 June?

Your answer

Units: cohorts

Alex Tutor

Start with v4.8.0: all 28 of its cohorts are complete. Sum active_days_8_14 and new_users, then divide. Sum first; averaging the daily rates would give small days the same weight as big ones.

You

So a quiet Tuesday and a busy Saturday count by their users, not as equals.

Alex Tutor

Right. Pool the users, then divide once.

Alex Tutor

Question 3

What share of v4.8.0 new users were active in days 8–14? Percent, two decimals.

Your answer

Units: %

Maya

And my 15% for v4.9.0?

Alex Tutor

That's all 21 v4.9.0 cohorts, 14 of which couldn't be fully counted yet. Most of your drop is calendar, not code. Take only the seven complete cohorts.

You

11 to 17 May, then. Same sums, same division.

Alex Tutor

Question 4

Now the complete v4.9.0 cohorts only (11–17 May). What share was active in days 8–14? Percent, two decimals.

Your answer

Units: %

Theo

My report uses “returned at any time”: 66.8% for v4.8.0 against 47.1% for v4.9.0. That's a drop too.

Alex Tutor

A cohort from mid-April had six weeks to come back once; a cohort from late May had days. That column measures age, not the version.

You

So a metric without a fixed horizon always favors older cohorts.

Alex Tutor

Question 5

“Returned at any time since sign-up” is a fair metric for comparing v4.8.0 and v4.9.0 cohorts, because it uses all the data we have.

TrueFalse

Maya

So v4.9.0 is actually better? +2.33 points?

Alex Tutor

Careful twice. First, chance: seven days, 551 users. The Wilson-based interval for the difference runs from about −1.9 to +6.7 pp. It could be a small loss or a decent gain.

Second, even a narrow interval covers only chance. Anything else that changed in May, like school holidays, a campaign or the weather, sits inside the same number. Before/after can show a difference; it can't prove the release caused it.

You

Then the honest answer is: we didn't break it, and we can't claim we improved it yet.

Alex Tutor

Question 6Short answer · AI-checked task

Answer Maya's question “Did v4.9.0 break week-2 retention?” in 3–5 sentences. Use the numbers from the lesson, say why her 15% is misleading, and what the comparison can and can't support.

Write your answer and get a score with feedback from our AI reviewer.

Log in to get AI feedback

Put a date in the calendar: in two weeks, 21 v4.9.0 cohorts will be complete instead of seven. Same rules, a sharper number.

Alex Tutor

That’s the lesson. You answered every task — nicely done.

Next lesson: Traffic mix and seasons