The Eight Maladies of Rapid Experimentation

Build-Measure-Learn (BML) can often go wrong for businesses when they first begin to adopt rapid experimentation for product delivery.

At Simply Business, we recognised some common maladies of BML early on in our journey. You can hear more about our early experiences of adopting BML in this video of a talk I gave at the Lean Startup Conference - San Francisco (2019). In the video, I explore how we identified and remedied the eight deadly maladies of rapid experimentation.

Duration: 40 minutes (approx)

Further resources:

• Summary Sheet
• Slide deck: Eight Maladies of Rapid Experimentation
• More about Lukas Oberhuber - Group CTO at Simply Business, at Simply Business Tech.

Summary: eight maladies

Below is a summary of the key points in the video from the talk on the Eight Maladies of Rapid Experimentation, presented at the Lean Startup Conference (2019).

Introduction

At Simply Business, we successfully integrated rapid experimentation, continuous integration and continuous delivery into our product development cycle several years ago. We're now able to release our main service over a hundred times a week, but in the early days, it wasn't always so.

The following section explores eight cases of what can happen when introducing BML based on our early experiences at Simply Business. It explains what can go wrong, how to recognise the symptoms and how to get back on track.

As Aristotle famously said:

"It is possible to fail in many ways, while to succeed is possible only in one way."

1: Do tests that matter

In one of our early BML experiments, we were looking to test whether a potential payment provider (PayPal), would get click-throughs from our website. We ran the experiment and concluded that it did. But PayPal was more expensive than our current provider. We found that we would never switch simply because our customers would use it, so we didn't have a case for moving to this new place.

Some of the flags for recognising that a testing experiment may be going off course are:

• Performing an experiment without success criteria or a way forward
• Everything needing to be statistically significant
• Choosing a test based on familiarity with the technique

Any one of these symptoms can highlight that you might not be able to adopt what you've discovered in a test. So what exactly is this malady? It's what I call A/B Testostrophy.

A/B Testostrophy is about time wasted on experiments that don't matter. Often people experiment because they think they should, or the CTO told them to! This can result in a lack of accountability. When you don't do something important, you don't have much accountability for it. While A/B tests are critical for incremental changes, some things can be tested more quickly than by using an A/B test.

We looked at some of the causes and found it was often due to:

• Fear of breaking the system, getting something wrong or a lack of courage.
• Doing what has been asked without thinking and switching off your brain.
• Not understanding the goals.
• An intolerance of failure, which may be a cultural problem within the company.

So it's good to reflect on what you're doing and asking whether it's valuable, and if you're moving fast enough. You may be doing A/B tests but not making decisions quickly. How will you know if what you've done is a success?

In finding ways of overcoming these problems:

1 - Decide what you need to learn, then design the experiments. Start doing experiments, then make teams accountable for the results.

2 - Customer behaviour before treatment. Understand why your customers are doing something before you build a solution to treat the symptom.

3 - Don't test whether the cost of the test is more than the test is likely to be worth.

2: Do what's important

We made many small changes to our back office system to fix small niggles for our call centre, at the expense of delivering real value. The team wanted to prove value through small changes, but the real progress on the goal would be coming in the next quarter. In the end, we didn't achieve the result we wanted.

Things that can alert you to this situation are where there are:

• Continuous quick wins to prove value
• Marginal metric improvements - the big change is coming, but not yet.

These symptoms are what I call Low-hanging-fruit-arrhea.

The consequences of this are low results, caused by:

• A lack of vision
• Fear of not proving value
• Expecting immediate results. The pressure to deliver may be coming from your company; if there isn't a way of demonstrating that, it's tempting to go for low-hanging fruit.

The remedy is to start with a valuable goal and don't get distracted. This is the critical part.

3: Change the world to succeed

On another project, we wanted to test the results of making improvements to our website quote comparison page (what we term toggling). For example, "how much public liability insurance do I want?"

Our tests showed that it would work, but we didn't act on the positive results because people questioned whether we trusted our findings.

Some symptoms to recognise in this situation are where there are:

• Few ideas on how to move forward
• Doubts as to whether a change is possible
• Demands to prove every step
• Major unresolved questions despite much activity

This is what I term Caged-in-Syndrome.

One of the consequences of Caged-in-Syndrome are that business questions either do not move forward or move at a slow pace. But the most significant one is business failure. If you can't find your product market fit, you're unlikely to have a business. If you can't move on from the way the world is today, you've got a real problem.

The cause can be due to a number of factors:

1 - People with a lot of experience in the industry may not want to look bad in their field of expertise. If you're not doing something different, you can't look bad!

2 - Lack of understanding of what change is needed to reach your goal. If you don't know how to make the change, how can you really drive it forward?

3 - Listening to others but not having a point of view. You can end up listening to conflicting ideas and change your mind constantly.

4 - External pressure.

5 - Being anchored in the current world rather than in the desired future.

If you're familiar with the Lean Startup, you'll recognise the need to know what your riskiest assumptions are and to have a plan if you're wrong about how things work.

To overcome some of these issues:

1 - Decide on the burden of proof and negotiate ahead of time. The burden of proof may reside outside of the team, with management or external stakeholders. Knowing what the burden of proof is, is critical for verifying that an experiment does prove what you are trying to prove.

2 - Test the riskiest assumptions regardless of sensitivities. It can be tempting to avoid the most difficult assumption, but that only delays the pain.

3 - Don't permit proven questions and learnings to be reopened unless new information is discovered. Trust in your process.

4: Pick a good opportunity. Finding the best takes too long

The drive to find the perfect answer can be compelling. We wanted to improve our product for high value customers and were experimenting around different segments - landlords, shopkeepers, tradespeople. After testing opportunity after opportunity, we couldn't decide which to choose.

This is what I term The Best Pox - a malady that can be difficult to cure.

In the hunt for the 'best thing', you arrive at a state of paralysis in wanting the perfect answer. This state can come about where you:

• Dip your toes in the water but never jump in. Without commitment, it's hard to move forward. You can be testing, testing, testing, but without real progress.
• Fear making mistakes.
• Question the value of what you're doing against an extremely well-oiled machine in another area.

In this situation, it's worth acknowledging that:

1 - Finding the best answer is difficult and expensive. Choose good and go for it.

2 - Choose an opportunity; that will likely be heuristic or a gut feel.

3 - Make the most of that good opportunity.

4 - Optimise that opportunity over time. If, after further testing, the opportunity turns out not to be one, you can put it aside and move on to the next.

5: Innovation is a roller coaster

On one of our early projects, we executed a series of experiments and were comfortable that those experiments had a clear path forward. There were still some tricky problems to tackle, but before these had been resolved, the team had slipped into delivery mode.

In this situation, there are no emotional highs and lows as difficult truths are learned or overcome.

This is symptomatic of what I call It's-so-easy Diseasy.

Building something truly new is never easy. Startups are hard; you're changing the world. Most business people describe running a startup like riding a roller coaster. Some reasons for being in a state of It's-so-easy Diseasy are:

• Going for comfort is easy. When things get uncomfortable, it's tempting to do the thing we're already good at.
• Lack of time pressure.
• Positivity culture. If bad news is punishable, there's a tendency to be positive about everything, which can lead to glossing over problems and only talking about the good. Instead, actively look for problems that could cause issues.

In managing this situation:

• Ensure that actual progress is being made on the riskiest assumptions, rather than focussing on excessive positivity.
• Run regular Pivot Persevere Kill (PPK) meetings to know what's working.
• Insist on strong and improving evidence.

6: Measure progress bi-weekly

During some of our early experiments, we wanted to prove a gap in the market. The learnings were not diligently documented and there was speculation over whether there were already good enough products in the market.

In this situation, the debate is never closed. There may be weak documentation about learnings, there may be lots of opinions, tests may be poorly designed, and a lack of governance. All these are strong signs that something is missing. It's what I refer to as No Rigor Mortis.

This can be the result of a lack of ownership, a lack of leadership and little or no accountability. In this case, it helps to:

• Run bi-weekly cadence, where the riskiest assumptions are reduced. If you don't make progress, you may need to pivot.
• Track governance carefully to learn whether progress is being made.
• Look at team capability. The riskiest assumption may be a team's ability to experiment or learn. Focus on learning rather than experimenting.

7: Small iterations towards big goals

After a year of researching how to serve high-value customers, we found there were always more tests to run, and we had not reached the killer conclusion. This stemmed from assuming we knew nothing when we in fact knew more, from running many tests with few results, having a few sharp puzzle pieces but no clear overall picture.

This is what I term Boil-the-Oceanoma.

Similar to the condition No Rigor Mortis, a few ways to progress from a state of Boil-the-Oceanoma are to look at what you would build right now if you had to, what progress has been made in the last two weeks and whether all the risky assumptions have been identified.

Things to consider:

• Make small iterative steps in the right direction rather than giant leaps.
• Track change in riskiest assumptions carefully. No Rigor Mortis is also where tracking assumptions is critical.
• Expect change to happen in weeks, rather than months or years.

8: Without funding, don't bother with tests

On launching a new product for commercial landlords, we pulled funding in favour of higher short-term value opportunities. As a consequence, the product never reached our target performance metrics and underperformed.

This is what I call Investaphobia.

There's only one cure for Investaphobia, and that is to ensure that commitment to funding is obtained up-front and is ringfenced as soon as the opportunity is proven.

Conclusion

Good and bad experimentation can be difficult to differentiate. Over the course of our journey in rapid experimentation, we made some mistakes, learned from them and improved from having made them. Our culture at Simply Business is all about learning, about great people and being a great place to work.

Based on our experience, here are eight signals of bad experimentation to watch out for.

1 - A/B Testostrophy. Many similar tests, little learning. Instead: Do tests that matter

2 - Low-Hanging-Fruit-arrhea. Constant quick wins to prove worth. Instead: Stop trying to prove you are worth it. Do the important

3 - Caged-in syndrome. Nothing can be changed, so don’t try. Instead: You have to change the world to succeed

4 - The-Best-Pox. Always asking “Is this the best opportunity?” Instead: Pick a good opportunity. Finding the best takes too long.

5 - It's-so-easy Diseasy. “We got this, don’t worry.” Instead: Innovation is a roller-coaster, expect that.

6 - No-Rigor-Mortis. No innovation accounting or regular progress on risky assumptions. Instead: Make sure you measure progress on a bi-weekly basis.

7 - Boil-the-Oceanoma. Need to understand it all before any action. Instead: Small iterations towards your big goal.

8 - Investaphobia. No funds available even if the test succeeds. Instead: Without funding, don’t bother with a test.

You can download a Summary of the Eight Maladies of Rapid Experimentation, view the slides and watch the video on YouTube.