How many experiments do you run? · Beacon Fellowship School

How many experiments do you run?

I’ve written and talked about this many times. At one time I was a working scientist, doing real science every single day.

But what does that have to do with you?

A scientist does experiments professionally

That means:

Having a theory you want to test

Setting out the process you will use to find out if you were right

Running the experiment, making sure you know what you will measure and how you will know if you have proved your theory

Using this knowledge to take your next action

Why am I telling you ths? Business is not like science at all!

Really??

Nothing sucks the energy out of a brainstorm faster than “we already tried that.”

Why this phrase is problematic

I don’t believe in failure. Every opportunity where it doesn’t go as we expected is one to learn something new, but only if you know what you were trying to do and try to understand what caused the unexpected result…..

Often when an experiment doesn’t go the way we expected, we wall it off in our minds, and rule out entire approaches based on a single experiment. We don’t do the learning and thinking which should follow each experiment.

My favourite? “We tried Facebook ads, they don’t work.”

Really? Then why is Mark Zuckerberg a trillionaire?

Meta ads, like most good ideas, only work after dozens of attempts and learning what worked and what needed further improvement.

But how do you know if you should try again or if it's genuinely a bad idea? I’m not comfortable making huge decisions based on a single experiment whose specific details nobody quite remembers.

Try, Try, Try Again? (or not)

That’s why, in my teams, we have a rule:

If somebody says, “We already tried that,” they must show you the experiment plan. This is a plan just like I used as a working scientist to document my experiments, (back in the day, when I made discoveries that allowed me to go all over the world telling people what I had found…)

An experiment plan contains, at a minimum, 5 things:

The original hypothesis and experiment design
The success metric and everybody’s predictions
Descriptions of all variants
Screenshots of the results metrics
Notes from the debrief on learnings and next steps

Here’s a link to the plan we use (https://systemise.me/free-stuff)

How to decide if you should re-try the idea

Read through the plan and ask yourself “Does this really disprove my hypothesis?” “How does this change my thinking?” “Are these same conditions still true in our business, or has something changed?”

If you’re not convinced, run the experiment again. The best ideas in history seldom worked the first time.

What if you have no experiment plan?

In that case, definitely run the experiment again, and document it.

I know, documentation takes time.

You know what really takes time? Debating theories in the absence of data.

And ignoring your best ideas because somebody said “we already tried that.”

I hope this helps.

3 comments