CX Methods
Why satisfaction and usability scores mislead
Customers say a task was easy while quietly failing it. Self-report measures how people want to appear; behavior shows what happened.
Satisfaction and usability instruments like SUS, SUPR-Q, and UMUX ask people to rate abstractions, predict their own behavior, and be polite. The result is a number that cannot tell you what to fix. Observed behavior catches what self-report hides.
Ask a customer whether a task was easy and you get politeness. Ask what they will do next and you get a guess. Ask how confident they are and you learn nothing about whether they actually succeeded, because people can feel certain they finished a task that quietly failed. Standard instruments such as SUS, SUPR-Q, and UMUX inherit all of this. They are abstract, generic, and detached from the real tasks people came to do, so they produce a number that cannot point to a fix. The number moves, the team debates it, and the product stays exactly as it was.
Why it matters to the business
The gap between saying and doing is not small. When the analytics firm 84.51 checked survey answers against loyalty-card records, 75 percent of respondents had misstated their own purchase behavior. Bain research found 60 to 80 percent of lost customers called themselves satisfied or very satisfied shortly before defecting. A dashboard built on self-report can glow green while revenue walks out the door.
Coverage is thin too. McKinsey's CX research found the typical survey reaches only about 7 percent of a company's customers, and just 6 percent of firms are confident their survey-based measurement informs decisions.
How to use it
- Watch five customers attempt your top tasks each quarter; Nielsen Norman Group research shows five users surface roughly 85 percent of usability problems.
- Measure task success rate and time-on-task, not reported ease.
- Before funding anything based on stated intent, check the claim against behavioral data.
- Reconcile satisfaction scores with actual repurchase and churn every quarter.
- Write survey questions about what just happened, in the customer's own words, not abstract constructs.
Where teams get it wrong
The usual fix is swapping one questionnaire for another and calling it rigor. The problem is not the wording; it is the method. No instrument built on self-report can catch a silent failure. Only observation can, and observation is far cheaper than most teams assume.
Ask your team
- When did anyone on this team last watch a real customer attempt our most important task?
- Where do our scores and our behavioral data disagree, and which one are we trusting?
- What decision actually changed because of last quarter's satisfaction survey?
People tell you how they want to appear. Behavior tells you what happened.
Apply this
Reading about why satisfaction and usability scores mislead is one thing. Seeing where it applies in your journey is the useful part.