Most bad analysis starts with a vague question. “Is the new website working?” cannot be answered with data until you decide what “working” means, for whom and compared with what.
Four parts of a measurable question
- Decision: what will you do differently depending on the answer? If nothing, the analysis is not needed.
- Metric: the specific number that represents success, such as conversion rate, average order value or days to resolve a ticket.
- Population and period: whose behavior, over which dates.
- Comparison: against what, such as last year, a control group or a target.
Vague: Is the new website working?
Measurable: Did the share of mobile visitors who complete checkout
change between the 8 weeks before and the 8 weeks after
the redesign, and is any change larger than normal
week-to-week variation?
Decision: keep the new checkout, or roll back the mobile flow.Watch out for proxy metrics
Page views, clicks and time on page are easy to measure but may not reflect what you care about. Check that the metric you choose actually moves with the outcome that matters, such as revenue, retention or cost.
Try this
Write down a question you were recently asked at work. Rewrite it with a decision, a metric, a population and period, and a comparison.