Drunk agile: moving fast in random directions

Drunk agile is moving fast in random directions and calling it progress. The symptoms are easy to spot: a heavy A/B testing calendar, little or no foundational research, dashboards full of inconclusive results, and a product still searching for its market. Each test feels like rigor. But testing button colors on a product whose core value is unproven optimizes nothing; you are measuring noise with precision. The failure mode repeats everywhere: months of split tests, budget gone, core fit never validated.

Why it matters to the business

A/B testing is premature under knowable conditions: too little traffic to reach statistical significance, a weak baseline experience, an unclear value proposition, or unvalidated customer tasks. Under any of them, tests either never conclude or conclude about the wrong thing, while the burn rate continues. The alternative is dramatically cheap. Nielsen Norman Group's research found that about five users uncover roughly 85% of usability problems in a qualitative test, and that three rounds of five-user tests beat one large study. Forrester's mapping discipline makes the same point at strategy level: draft your hypothesis from internal insight, then validate it with real customer research before treating it as truth. Direction costs days. Drift costs quarters.

How to use it

• Before approving any test, require a written hypothesis naming the customer task it serves and the research that motivated it.
• Check statistical preconditions first: estimate the traffic and effect size needed, and if significance is out of reach within weeks, do not run the test.
• Run five-user qualitative rounds to find and fix gross problems before spending traffic on split tests.
• Validate the value proposition and top customer tasks with primary research at least once a year, not with brainstorms.
• Kill zombie tests: anything running past its predeclared window without significance ends, and the learning gets written down.

Where teams get it wrong

The trap is emotional, not technical. Testing feels like action, and research feels like delay. So teams skip the two-week study and spend two quarters testing variations of a product customers never wanted. Velocity becomes the metric, direction goes unmeasured, and the burn gets booked as learning.

Ask your team

• Of the tests we ran last quarter, how many reached significance, and how many changed a decision?
• What foundational research validates the tasks and value proposition our current tests assume?
• If we paused all A/B tests for a month and did customer research instead, what would we finally learn?

Velocity without direction is waste.