Quality & Research
Research rigor: interrogate claims before acting on them
'We tested it' deserves three follow-ups: tested what, on which devices, with whom. Rigor is a leadership posture, not a research task.
Research rigor, applied as an advisory posture, means interrogating claims instead of accepting them. When a team says 'we tested it', the follow-up is: what, on which devices, with whom. That discipline catches invalid research and revenue leaks before they compound.
Research rigor is not just a method for researchers; it is a questioning posture for anyone who makes decisions on evidence. In practice it sounds like this. A team says 'we tested it', and the follow-up is: tested what exactly, on which devices, with whom. Someone proposes an A/B test before anyone understands the problem, and the answer is not yet. The posture assumes every confident claim has an evidence trail, and asks to see it before money moves.
Why it matters to the business
Unexamined claims are how companies fool themselves at scale. Bain found 80% of companies believe they deliver a superior experience while only 8% of their customers agree. The evidence under most dashboards is thinner than it looks: McKinsey found the typical CX survey samples only about 7% of a company's customers, and while 93% of firms measure by survey, just 6% are confident the results inform decisions. Forrester reports only 12% of CX professionals rate their voice-of-customer program's maturity as high. A leader who asks three sharp questions per claim is often the only quality control the evidence ever gets.
How to use it
- Attach three standing questions to every 'we tested it': what did you test, on which devices and conditions, with which participants?
- Ask for the sample size and sampling method behind every score presented in an executive review.
- Block A/B tests proposed before the problem is defined. Optimization cannot rescue a misunderstood problem.
- Require the research method to match the question. Behavior questions need observation, not opinion surveys.
- Before and after major launches, require observed task success, not just self-reported satisfaction.
Where teams get it wrong
Rigor gets mistaken for obstruction, so leaders soften it into politeness and accept confident summaries at face value. Or it curdles into volume worship: demanding more surveys, bigger samples, and longer reports when the real problem is method fit. Ten thousand survey responses cannot answer a question that only observation can.
Ask your team
- What evidence sits behind our claim that customers like this, and can I see it?
- Which decisions this quarter rest on research that was designed to confirm the idea?
- When we say we tested it: what, on which devices, with whom?
The follow-up to 'we tested it' is always: what, on which devices, with whom.
Apply this
Reading about research rigor: interrogate claims before acting on them is one thing. Seeing where it applies in your journey is the useful part.