The Horizon Commerce team recently spent a full day soaking in ideas at Opticon25, Optimizely’s annual event dedicated to the future of AI and digital experience optimization. It was a packed agenda full of sharp perspectives and fresh thinking, but one session in particular stuck with us.
Optimizely’s Britt Hall sat down with Riskesh Shah, Senior Manager of Test & Learn and CX VoC at United Airlines, to share how United scaled experimentation across the customer journey. Their story wasn’t just about experiments—it was about building a culture of curiosity that drives better customer experiences.
The big takeaway?
Success in experimentation isn’t just about winning tests—it’s about how well your team can embrace failure. And not all failures are created equal. That led us to ask: what actually separates good failure from bad failure?
Too often, we chase visible wins while ignoring “shadow data”—the negative or inconclusive results that rarely get airtime in a recap. But those losses can be just as valuable, if we know how to define and use them.
For us, good failure has three traits:
- It exposes something hidden. You learn something you didn’t know before.
- It forces a decision. You can’t go back to business as usual—you have to act.
- It justifies itself. The effort creates learning that moves the team forward.
That framing sparked a few questions we’ll be carrying forward:
- If we looked only at the losing variants of an experiment, what would we test next?
- How do we share failure in a way that spreads the learning across teams—so it’s not just a line buried in a deck?
- What insights did the losing variants reveal that the winning one can’t—and how do we make sure those insights shape our roadmap?
Opticon25 left us with plenty of new perspectives, but this conversation was a highlight. As AI ramps up the ability to scale experience testing, the advantage belongs to those choosing to fail smarter. Grateful to the Optimizely community for spurring valuable conversations and inspired thought leadership.