Research and reports
Build-Measure-Learn (BML) can often go wrong for businesses when they first begin to adopt rapid experimentation for product delivery.
At Simply Business, we recognised some common maladies of BML early on in our journey. You can hear more about our early experiences of adopting BML in this video of a talk I gave at the Lean Startup Conference - San Francisco (2019). In the video, I explore how we identified and remedied the eight deadly maladies of rapid experimentation.
Duration: 40 minutes (approx)
Below is a summary of the key points in the video from the talk on the Eight Maladies of Rapid Experimentation, presented at the Lean Startup Conference (2019).
At Simply Business, we successfully integrated rapid experimentation, continuous integration and continuous delivery into our product development cycle several years ago. We're now able to release our main service over a hundred times a week, but in the early days, it wasn't always so.
The following section explores eight cases of what can happen when introducing BML based on our early experiences at Simply Business. It explains what can go wrong, how to recognise the symptoms and how to get back on track.
As Aristotle famously said:
"It is possible to fail in many ways, while to succeed is possible only in one way."
In one of our early BML experiments, we were looking to test whether a potential payment provider (PayPal), would get click-throughs from our website. We ran the experiment and concluded that it did. But PayPal was more expensive than our current provider. We found that we would never switch simply because our customers would use it, so we didn't have a case for moving to this new place.
Some of the flags for recognising that a testing experiment may be going off course are:
Any one of these symptoms can highlight that you might not be able to adopt what you've discovered in a test. So what exactly is this malady? It's what I call A/B Testostrophy.
A/B Testostrophy is about time wasted on experiments that don't matter. Often people experiment because they think they should, or the CTO told them to! This can result in a lack of accountability. When you don't do something important, you don't have much accountability for it. While A/B tests are critical for incremental changes, some things can be tested more quickly than by using an A/B test.
We looked at some of the causes and found it was often due to:
So it's good to reflect on what you're doing and asking whether it's valuable, and if you're moving fast enough. You may be doing A/B tests but not making decisions quickly. How will you know if what you've done is a success?
In finding ways of overcoming these problems:
1 - Decide what you need to learn, then design the experiments. Start doing experiments, then make teams accountable for the results.
2 - Customer behaviour before treatment. Understand why your customers are doing something before you build a solution to treat the symptom.
3 - Don't test whether the cost of the test is more than the test is likely to be worth.
We made many small changes to our back office system to fix small niggles for our call centre, at the expense of delivering real value. The team wanted to prove value through small changes, but the real progress on the goal would be coming in the next quarter. In the end, we didn't achieve the result we wanted.
Things that can alert you to this situation are where there are:
These symptoms are what I call Low-hanging-fruit-arrhea.
The consequences of this are low results, caused by:
The remedy is to start with a valuable goal and don't get distracted. This is the critical part.
On another project, we wanted to test the results of making improvements to our website quote comparison page (what we term toggling). For example, "how much public liability insurance do I want?"
Our tests showed that it would work, but we didn't act on the positive results because people questioned whether we trusted our findings.
Some symptoms to recognise in this situation are where there are:
This is what I term Caged-in-Syndrome.
One of the consequences of Caged-in-Syndrome are that business questions either do not move forward or move at a slow pace. But the most significant one is business failure. If you can't find your product market fit, you're unlikely to have a business. If you can't move on from the way the world is today, you've got a real problem.
The cause can be due to a number of factors:
1 - People with a lot of experience in the industry may not want to look bad in their field of expertise. If you're not doing something different, you can't look bad!
2 - Lack of understanding of what change is needed to reach your goal. If you don't know how to make the change, how can you really drive it forward?
3 - Listening to others but not having a point of view. You can end up listening to conflicting ideas and change your mind constantly.
4 - External pressure.
5 - Being anchored in the current world rather than in the desired future.
If you're familiar with the Lean Startup, you'll recognise the need to know what your riskiest assumptions are and to have a plan if you're wrong about how things work.
To overcome some of these issues:
1 - Decide on the burden of proof and negotiate ahead of time. The burden of proof may reside outside of the team, with management or external stakeholders. Knowing what the burden of proof is, is critical for verifying that an experiment does prove what you are trying to prove.
2 - Test the riskiest assumptions regardless of sensitivities. It can be tempting to avoid the most difficult assumption, but that only delays the pain.
3 - Don't permit proven questions and learnings to be reopened unless new information is discovered. Trust in your process.
The drive to find the perfect answer can be compelling. We wanted to improve our product for high value customers and were experimenting around different segments - landlords, shopkeepers, tradespeople. After testing opportunity after opportunity, we couldn't decide which to choose.
This is what I term The Best Pox - a malady that can be difficult to cure.
In the hunt for the 'best thing', you arrive at a state of paralysis in wanting the perfect answer. This state can come about where you:
In this situation, it's worth acknowledging that:
1 - Finding the best answer is difficult and expensive. Choose good and go for it.
2 - Choose an opportunity; that will likely be heuristic or a gut feel.
3 - Make the most of that good opportunity.
4 - Optimise that opportunity over time. If, after further testing, the opportunity turns out not to be one, you can put it aside and move on to the next.
On one of our early projects, we executed a series of experiments and were comfortable that those experiments had a clear path forward. There were still some tricky problems to tackle, but before these had been resolved, the team had slipped into delivery mode.
In this situation, there are no emotional highs and lows as difficult truths are learned or overcome.
This is symptomatic of what I call It's-so-easy Diseasy.
Building something truly new is never easy. Startups are hard; you're changing the world. Most business people describe running a startup like riding a roller coaster. Some reasons for being in a state of It's-so-easy Diseasy are:
In managing this situation:
During some of our early experiments, we wanted to prove a gap in the market. The learnings were not diligently documented and there was speculation over whether there were already good enough products in the market.
In this situation, the debate is never closed. There may be weak documentation about learnings, there may be lots of opinions, tests may be poorly designed, and a lack of governance. All these are strong signs that something is missing. It's what I refer to as No Rigor Mortis.
This can be the result of a lack of ownership, a lack of leadership and little or no accountability. In this case, it helps to:
After a year of researching how to serve high-value customers, we found there were always more tests to run, and we had not reached the killer conclusion. This stemmed from assuming we knew nothing when we in fact knew more, from running many tests with few results, having a few sharp puzzle pieces but no clear overall picture.
This is what I term Boil-the-Oceanoma.
Similar to the condition No Rigor Mortis, a few ways to progress from a state of Boil-the-Oceanoma are to look at what you would build right now if you had to, what progress has been made in the last two weeks and whether all the risky assumptions have been identified.
Things to consider:
On launching a new product for commercial landlords, we pulled funding in favour of higher short-term value opportunities. As a consequence, the product never reached our target performance metrics and underperformed.
This is what I call Investaphobia.
There's only one cure for Investaphobia, and that is to ensure that commitment to funding is obtained up-front and is ringfenced as soon as the opportunity is proven.
Good and bad experimentation can be difficult to differentiate. Over the course of our journey in rapid experimentation, we made some mistakes, learned from them and improved from having made them. Our culture at Simply Business is all about learning, about great people and being a great place to work.
Based on our experience, here are eight signals of bad experimentation to watch out for.
1 - A/B Testostrophy. Many similar tests, little learning. Instead: Do tests that matter
2 - Low-Hanging-Fruit-arrhea. Constant quick wins to prove worth. Instead: Stop trying to prove you are worth it. Do the important
3 - Caged-in syndrome. Nothing can be changed, so don’t try. Instead: You have to change the world to succeed
4 - The-Best-Pox. Always asking “Is this the best opportunity?” Instead: Pick a good opportunity. Finding the best takes too long.
5 - It's-so-easy Diseasy. “We got this, don’t worry.” Instead: Innovation is a roller-coaster, expect that.
6 - No-Rigor-Mortis. No innovation accounting or regular progress on risky assumptions. Instead: Make sure you measure progress on a bi-weekly basis.
7 - Boil-the-Oceanoma. Need to understand it all before any action. Instead: Small iterations towards your big goal.
8 - Investaphobia. No funds available even if the test succeeds. Instead: Without funding, don’t bother with a test.
You can download a Summary of the Eight Maladies of Rapid Experimentation, view the slides and watch the video on YouTube.
If you see a position that suits, why not apply today?Find out more
We create this content for general information purposes and it should not be taken as advice. Always take professional advice. Read our full disclaimer
6th Floor99 Gresham StreetLondonEC2V 7NG
Sol House29 St Katherine's StreetNorthamptonNN1 2QZ
© Copyright 2023 Simply Business. All Rights Reserved. Simply Business is a trading name of Xbridge Limited which is authorised and regulated by the Financial Conduct Authority (Financial Services Registration No: 313348). Xbridge Limited (No: 3967717) has its registered office at 6th Floor, 99 Gresham Street, London, EC2V 7NG.