Like Amy, I didn’t find as many practical tools or insights that I could take back to my teams as I would have liked, but one talk in particular resonated the most with my current work challenges and goals; ‘Implementing a Culture of Experimentation’ by Colin McFarland of Skyscanner.
The key phrase that Colin returned to throughout the talk was ‘design like you’re right, test like you’re wrong’, which has already helped to sharpen up our test hypotheses.
Design like you’re right:
Based on [quantitative/qualitative insight]. We predict that [product change] will cause [impact].
Test like you’re wrong:
We will test this by assuming the prediction has no effect (the null hypothesis) and running an experiment for [X week(s)].
If we can measure a [Y%] statistically significant change in [metric(s)] then we reject the null hypothesis and conclude there is an effect.
For me, the ‘test like you’re wrong’ part has the biggest impact – pre-defining what success must look like in order to be adopted, as opposed to tracking multiple variables and trying to mould these into a story that proves your original idea was a good one.
The second influence the talk has had came from hearing about the high volumes of experiments that Skyscanner run at any one time, and over the course of a year. Until this point, I’ve always erred on the side of caution when ensuring tests don’t overlap and, as a result, tended to only have a handful of concurrent tests. Colin was much more relaxed about this happening, and as a result has been able to run a lot of tests and draw a lot of conclusions.
The conference was also great for general inspiration and confirmation that lean is the right approach for us and that we’re approaching it right.