New concepts have a startlingly low chance of improving the bottom line for either corporates or start-ups. Beyond headline-grabbing failures - the recent demise of Karhoo despite reportedly raising $250m; Walgreens’ closure of 2011 $429m acquisition Drugstore.com - as many as 90 per cent of all start-ups fail.

But innovation rarely fails because of a lack of ideas. Covering a wall with ideas scribbled on brightly coloured Post-it notes can be exciting, but the real hard work begins later.

A single factor, core to en vogue methodologies such as Lean Start-up, Agile Software Development, and Design Thinking, dictates more than any other the chances of success, but is rarely well understood or executed: experimentation.

Culture deters experimentation

Steve Blank wrote in 2010 that “no business plan survives first contact with a customer”. But despite so much being written since about the importance of customer-centric approaches and ‘failing fast’, environmental influences mean that the experimental mindset remains rare.

At school, we learn to obey rules, believe experts, and follow established processes. Business structures, processes, and governance are designed to reduce risk. Add natural human aversion to change, and it’s a recipe for not experimenting or taking any risk at all.

The effort required to overcome entrenched cultural opposition also explains how organisations can spend hundreds of thousands – sometimes millions – of pounds on a concept or programme, yet fail to answer fundamental questions that will define the ultimate success of their innovation.

Will customers use it, not just now but also in two to five years’ time? Can money be made from it? Can it be economically built?

Larger organisations are waking up to the need to invest in this change: the UK’s NHS recently launched a Clinical Entrepreneur training programme; and Cisco has developed an Entrepreneur in Residence programme.

But experimentation is still often conflated with ‘piloting’, ‘agile’, or ‘innovation’ more broadly. This lack of understanding of what experiments are, and how they should be run, often contributes to failure.

Getting experimentation right

In our experience, the following methodology is the best recipe for successful experimentation:

1) Become learning-centric, not product-centric. People want to build things to show progress, and agencies, developers, and product teams all have vested interests in moving quickly to a build phase.

Think less ‘how quickly can we build a prototype’, more ‘how can we learn as much as possible as quickly as possible’; reflecting the UX and product community’s shift from Minimal Viable Product (MVP) to Minimal Viable Experiment (MVE).

Uber, for example, gave itself a trial run at being a delivery company, delivering Christmas trees in 14 UK cities during December 2015. That small-scale experiment was a proof of concept that paved the way for Uber Eats, helping the company ensure it had all the relevant infrastructure and service elements in place before entering the massively competitive food delivery market.

2) Ask the right questions to the right people. As Jason Cohen (founder of several successful startups, including Wordpress Engine) has said: “When ten people say they’ll give you money if you build this thing, that’s the only validation that counts.” A paying customer base hugely de-risks the chance of failure; what you, your friends, relatives, colleagues, or ‘experts’ think doesn’t matter.

Failures often highlight the impact of not asking the right questions, or asking them to the wrong people. Microsoft’s 2010 Kin mobile phone saw its European launch scrapped just two months into the product’s life.

Aimed at a young audience, with social media core to the experience, the Kin did not support apps or games – demonstrating a fundamental lack of understanding of how dramatically consumer behaviours and expectations had changed following the early success of Apple’s App Store.

3) Be curious. Often innovations are unearthed during experiments that were targeted at a different problem: Twitter started out as Odeo, a platform to help people find and subscribe to podcasts; PayPal started off as PDA security software called Confinity; Instagram started out as a gaming-orientated app called Burbn.

Staying alert to, and knowing how to capitalise on these possibilities could reveal new opportunities and revenue streams.

Experimenting is key to allowing unexpected opportunities to emerge, but businesses also need to maintain clarity and focus to keep driving the core business forward. This can be a tricky balance, and calls for an entrepreneurial mindset.

It is also important to understand why an experiment has failed – it is one thing to recognise the point of failure and stop pursuing an idea, but another entirely to learn and apply the lessons of that failure.

4) Start small. Most concepts can be quickly and cheaply tested without building anything. Tools such as Paper prototypes, Strikingly, Launch Rock, Google AdWords, and Facebook ads are your friends.

Testing the viability of an idea is all about establishing demand. It can be as simple as using Strikingly to build a basic website and product mock-up, and Google AdWords and Facebook ads to drive traffic.

Different combinations can help you understand who your customers are and refine your targeting and messaging before building anything for real. Go where your customers are: there can also be value in old-fashioned non-digital approaches such as talking to potential customers at an event.

5) Make it repeatable. One experiment is good; many experiments are great. A series of closely-linked experiments allows learning to be validated and refined at each stage, making the experimentation process more valuable.

To make any process repeatable it is important to codify the process so that anyone can follow it. Google Ventures has succeeded at this with its Design Sprint process, run more than 100 times with companies such as Slack.

Condensed into a five day window, The Design Sprint helps business test prototypes and ask the critical questions about their business model i.e. what problems are my potential customers facing, how can I help them solve those problems, what messages resonate most with them, where’s the best place to speak to them etc.

Slack successfully used the process to overhaul its marketing strategy and reach new customers. This one-week cycle is likely to be unsustainable for most companies over a long period, but it’s not the only way to do it.

The key is to find a process that works, allowing sufficient time for feedback to be gathered, processed, and incorporated into the next wave of experimentation.

Benefiting from experimentation

Recent corporate history is littered with companies failing to experiment and being left behind. Just 10 years ago Blackberry led the smartphone market.

But it didn’t experiment early enough with full-screen devices and its once hugely popular Blackberry Messenger Service, leaving it looking longingly at the success of Apple, Samsung, Google, WhatsApp and Facebook Messenger. Its phone design and build business recently shuttered for good.

Meanwhile, companies making experimentation core to their culture can rapidly grow and transform. 3M made this article’s title possible.

Its best-known product, the Post-it note - estimated to generate $1 billion in annual revenue - was the result of a failed experiment to produce a super-strong, permanent adhesive. The strong demand for the product was only proved, and initial poor sales reversed, following experimentation with free sample distribution.

For every headline name at the extremes, hundreds of companies fall somewhere in between. Turning Post-it note ideas into real-world impact can be challenging, but is worth the investment in overcoming barriers to experimentation. The investment doesn’t have to be big: the best way to improve experimentation is – you guessed it – to start experimenting.

Jon Stephens is managing consultant at EY-Seren.