How to choose one product success metric for a feature

I choose one product success metric by identifying the user behaviour a feature should change, then defining its population, measurement window and minimum worthwhile improvement. Before launch, I instrument that behaviour, verify the calculation and agree which guardrails would stop a rollout even if the primary number improves.

Choose a user outcome, not a feature interaction

I start with a sentence: this feature should help this group achieve this result. For a team onboarding checklist, the result might be getting a second teammate to complete a real task. Opening the checklist is not that result. Neither is ticking every item. Those interactions can explain what happened, but they cannot establish that the feature helped anyone.

Marty Cagan’s distinction between outcomes and output is useful here. Shipping the checklist is output. Changing what customers can accomplish is the outcome it needs to earn. I’d write the hypothesis as: a clearer setup sequence will help more new workspaces reach their first instance of productive collaboration. That gives the design something specific to change without making the proposed interface the definition of success.

I use Amplitude’s North Star framework to connect that local outcome to the wider product. Its distinction between a central value metric and contributing inputs helps avoid giving every feature its own disconnected score. I wouldn’t make company revenue the checklist’s primary measure. It is too distant for this decision. I’d choose a nearer behaviour, then state why I expect it to support sustained product use.

Define the metric formula and eligible users

For this example, I’d choose seven-day team activation rate: the percentage of eligible new workspaces where a second distinct member completes a non-demo task within seven days of workspace creation. Each workspace counts once, regardless of how many tasks its members complete. The denominator is all eligible workspaces assigned to that experience, not just those that opened the checklist.

That denominator is a design decision. Measuring checklist users alone selects people who were already willing to engage and can make a weak feature look useful. I’d set eligibility before exposure, using conditions available at workspace creation. Internal accounts and automated test workspaces would be excluded by a fixed rule. I wouldn’t remove customers later because they abandoned setup or never invited anyone.

I’d also settle the awkward definitions before implementation. A second member must have a different account identifier from the creator. Demo tasks must carry a reliable flag. Completion must be a stored task state, not a click on a button. I’d count the first qualifying completion and write down how deleted workspaces are handled. Otherwise, two analysts can produce different answers from the same launch.

Set a baseline, target and guardrail metrics

I need a baseline calculated with the same definition before choosing a target. Suppose historical measurement shows a 30% seven-day activation rate. Moving it to 35% is an increase of five percentage points, not five per cent. These are illustrative figures, not client results. I’d check that the baseline covers representative acquisition channels and ordinary working weeks, rather than a promotion or a holiday period.

The target should represent an improvement worth paying for. I’d weigh the build cost, ongoing maintenance and the value of another activated workspace, then agree the smallest useful gain. I wouldn’t borrow an industry benchmark to fill that gap. A product used by established teams will have different activation behaviour from one attracting people who are still deciding whether they need a team workspace.

One accountable metric does not mean one number on the dashboard. I’d keep a short list of guardrails, such as onboarding support contacts per eligible workspace and permission-related failures. Each needs a threshold and an owner. Permission exposure might justify stopping immediately; a small support increase might justify investigation. Checklist completion remains a diagnostic measure. It helps locate friction, but it cannot overrule a flat activation result.

Instrument the outcome before launching the feature

I’d turn the metric definition into a tracking contract before design sign-off. For this example, the minimum records are workspace creation, experiment assignment, membership creation and task completion. Each needs a timestamp and stable workspace identifier. Task completion also needs the completing member, task identifier and demo status. I’d record the assigned variant when assignment happens, rather than trying to reconstruct it from later screen views.

For persisted actions such as completing a task, I prefer an event generated after the server confirms the change. A browser click can happen without a successful save. I’d agree how retries are deduplicated and how late events are handled. The seven-day window uses the action timestamp. A separate reporting delay, perhaps 48 hours, allows delayed records to arrive without quietly extending the activation window.

Before release, I’d walk through test cases with an engineer and whoever owns analysis. One workspace qualifies. Another misses the deadline. A third completes only a demo task. I’d include a duplicate event and a creator completing their own task. Then I’d reconcile the calculated result against stored records. Seeing events in a debugger is not enough. I want evidence that the numerator and denominator behave as specified.

Choose a comparison that can support the decision

If traffic permits, I’d randomly assign eligible workspaces to the existing onboarding or the checklist. Workspace-level assignment keeps teammates in the same experience. I’d analyse everyone as assigned, including workspaces that never open the feature. That estimates the effect of offering the checklist, rather than the behaviour of a self-selected group. I’d also check that assignment proportions and baseline characteristics look plausible before interpreting the outcome.

I’d plan the sample size around the baseline and the smallest worthwhile improvement, with analytical help where needed. There is no sensible universal rule that an experiment needs two weeks. Recruitment volume, normal business cycles and the seven-day outcome window all matter. The final workspace recruited needs its full observation period, followed by the agreed reporting delay. Checking daily does not make an immature result ready.

With insufficient traffic, I’d be explicit about the weaker evidence. A staged release and a historical comparison can guide a decision, but acquisition changes, seasonality or a pricing change may explain the difference. I’d pair the metric with observed task completion in research to investigate the mechanism. I wouldn’t describe that combination as proof of causation, or manufacture certainty because the release calendar needs an answer.

Agree what happens when the metric moves or stays flat

Before launch, I’d write a short decision record naming the primary metric, guardrails, comparison, review date and decision owner. It should say what improvement would justify rollout and what evidence would remain inconclusive. A positive point estimate alone is not enough. If the uncertainty still allows both a useful gain and meaningful harm, I’d call the result unresolved rather than choose the interpretation I prefer.

At review, I’d separate three failure modes. A measurement failure means the records or calculation cannot be trusted. An exposure failure means eligible people rarely encountered the intended experience. An outcome failure means the feature reached people but did not change the accountable behaviour. Each calls for different work. I’d repair tracking, inspect delivery or revisit the product hypothesis, rather than treating every flat result as a request for better button copy.

I’d retain the original metric definition alongside any later revision. Changing activation from seven days to fourteen after seeing disappointing results creates a new question, not a rescued experiment. If checklist completion rises but team activation stays flat, I’d investigate where collaboration still breaks down. I wouldn’t add more checklist items simply because that is the part of the product already open in the design file.

Questions people ask

How many success metrics should a feature have?

I use one primary outcome metric, a few guardrails and diagnostic measures only where they help explain behaviour. The primary metric makes the decision legible; the guardrails prevent a local improvement from hiding damage elsewhere.

What is the difference between feature adoption and success?

Adoption tells me people started using something. Success tells me it improved the outcome it was built for. I treat adoption as diagnostic unless using the feature is itself a meaningful customer outcome.

How do you measure a feature with no baseline?

I instrument the target behaviour in the existing experience first, where possible. If that behaviour is entirely new, I look for a comparison based on the underlying customer task rather than inventing a historical baseline.

When should you measure feature success after launch?

I wait until the planned sample has completed the outcome window and the data has arrived. I monitor safety guardrails immediately, but I don't confuse an early operational check with a finished outcome evaluation.

Where I checked my thinking

• Amplitude: North Star Playbook
• Marty Cagan: Outcomes vs. Output