Progress is often driven not by the accumulation of small steps, but by dramatic leaps. The television wasn’t an iteration of a previous device, it was a new technology altogether. Einstein’s General Theory of Relativity didn’t tinker with Newton’s Law of Universal Gravitation, it replaced it in almost every detail. Likewise Dyson’s dual-cyclone vacuum cleaner was not a marginal improvement on the conventional Hoover that existed at the time, it represented a shift that altered the way insiders think about the very problem of removing dust and hair from household floors.

James Dyson is an evangelist for the creative process of change, not least because he believes it is fundamentally misconceived in the world today. As we talk in his office, he darts around picking up papers, patents, textbooks, and his own designs to illustrate his argument. He says:

Dyson’s journey into the nature of creativity started while vacuuming his own home, a small farmhouse in the west of England, on a Saturday morning in his mid-twenties. Like everyone else he was struck by just how quickly his cleaner lost suction.

Dyson strode into his garden and opened up the device. Inside he could see the basic engineering proposition of the conventional vacuum cleaner: a motor, a bag (which also doubled as a filter), and a tube. The logic was simple: dust and air is sucked into the bag, the air escapes through the small holes in the lining of the bag and into the motor, and the dust (thicker than the air) stays in the bag.

He says:

This realization triggered a new thought: what if there were no bag?

This idea percolated in Dyson’s mind for the next three years. A graduate of the Royal College of Art, he was already a qualified engineer and was helping to run a local company in Bath. He enjoyed pulling things apart and seeing how they worked. He was curious, inquisitive, and willing to engage with a difficulty rather than just accepting it. But now he had a live problem, one that intrigued him.

It wasn’t until he went to a lumberyard that the solution powered into his mind like a thunderbolt.

 Dyson rushed home. This was his moment of insight. “I vaguely knew about cyclones, but not really the detail. But I was fascinated to see if it would work in miniature form. I got an old cardboard box and made a replica of what I had seen with gaffer tape and cardboard. I then connected it via a bit of hose to an upright vacuum cleaner. And I had my cardboard cyclone.”

His heart was beating fast as he pushed it around the house. Would it work? “It seemed absolutely fine,” he says. “It seemed to be picking up dust, but the dust didn’t seem to be coming out of the chimney. I went to my boss and said: ‘I think I have an interesting idea.’ ”

This simple idea, this moment of insight, would ultimately make Dyson a personal fortune in excess of ?3 billion.

A number of things jump out about the Dyson story. The first is that the solution seems rather obvious in hindsight. This is often the case with innovation, and it’s something we will come back to.

But now consider a couple of other aspects of the story. The first is that the creative process started with a problem, what you might even call a failure, in the existing technology. The vacuum cleaner kept blocking. It let out a screaming noise. Dyson had to keep bending down to pick up bits of trash by hand.

Had everything been going smoothly Dyson would have had no motivation to change things. Moreover, he would have had no intellectual challenge to sink his teeth into. It was the very nature of the engineering problem that sparked a possible solution (a bag less vacuum cleaner).

And this turns out to be an almost perfect metaphor for the creative process, whether it involves vacuum cleaners, a quest for a new brand name, or a new scientific theory. Creativity is, in many respects, a response.

Relativity was a response to the failure of Newtonian mechanics to make accurate predictions when objects were moving at fast speeds.

Masking tape was a response to the failure of existing adhesive tape, which would rip the paint off when it was removed from cars and walls.

Dropbox, as we have seen, was a response to the problem of forgetting your flash drive and thus not having access to important files.

This aspect of the creative process, the fact that it emerges in response to a particular difficulty, has spawned its own terminology. It is called the “problem phase” of innovation. “The damn thing had been bugging me for years,” Dyson says of the conventional vacuum cleaner. “I couldn’t bear the inefficiency of the technology. It wasn’t so much a ‘problem phase’ as a ‘hatred phase.’ ”

Creativity is, in many respects, a response.

We often leave this aspect of the creative process out of the picture. We focus on the moment of epiphany, the detonation of insight that happened when Newton was hit by the apple or Archimedes was taking a bath. That is perhaps why creativity seems so ethereal. The idea is that such insights could happen anytime, anywhere. It is just a matter of sitting back and letting them flow.

But this leaves out an indispensable feature of creativity. Without a problem, without a failure, without a flaw, without a frustration, innovation has nothing to latch on to. It loses its pivot. As Dyson puts it: “Creativity should be thought of as a dialogue. You have to have a problem before you can have the game-changing riposte.”

Perhaps the most graphic way to glimpse the responsive nature of creativity is to consider an experiment by Charlan Nemeth, a psychologist at the University of California, Berkeley, and her colleagues. She took 265 female undergraduates and randomly divided them into five-person teams. Each team was given the same task: to come up with ideas about how to reduce traffic congestion in the San Francisco Bay Area. These five-person teams were then assigned to one of three ways of working.

The first group were given the instruction to brainstorm. This is one of the most influential creativity techniques in history, and it is based on the mystical conception of how creativity happens: through contemplation and the free flow of ideas. In brainstorming the entire approach is to remove obstacles. It is to minimize challenges. People are warned not to criticize each other, or point out the difficulties in each other’s suggestions. Blockages are bad. Negative feedback is a sin.

The second group were given no guidelines at all: they were allowed to come up with ideas in any way they thought best.

But the third group were actively encouraged to point out the flaws in each other’s ideas. Their instructions read: “Most research and advice suggests that the best way to come up with good solutions is to come up with many solutions. Free-wheeling is welcome; don’t be afraid to say anything that comes to mind. However, in addition, most studies suggest that you should debate and even criticize each other’s ideas [my italics].”

The results were remarkable. The groups with the dissent and criticize guidelines generated 25 percent more ideas than those who were brainstorming (or who had no instructions). Just as striking, when individuals were later asked to come up with more solutions for the traffic problem, those with the dissent guidelines generated twice as many new ideas as the brainstormers.

Further studies have shown that those who dissent rather than brainstorm produce not just more ideas, but more productive and imaginative ideas. As Nemeth put it: “The basic finding is that the encouragement of debate— and even criticism if warranted— appears to stimulate more creative ideas. And cultures that permit and even encourage such expression of differing viewpoints may stimulate the most innovation.”

The reason is not difficult to identify. The problem with brainstorming is not its insistence on free-wheeling or quick association. Rather, it is that when these ideas are not checked by the feedback of criticism, they have nothing to respond to. Criticism surfaces problems. It brings difficulties to light. This forces us to think afresh. When our assumptions are violated we are nudged into a new relationship with reality. Removing failure from innovation is like removing oxygen from a fire.

Think back to Dyson and his Hoover. It was the flaw in the existing technology that forced Dyson to think about cleaning in a new way. The blockage in the filter wasn’t something to hide away from or pretend wasn’t there. Rather, the blockage, the failure, was a gilt-edged invitation to reimagine vacuum-cleaning.

Imagination is not fragile. It feeds off flaws, difficulties, and problems. Insulating ourselves from failuresis to rob one of our most valuable mental faculties of fuel.

“It always starts with a problem,” Dyson says. “I hated vacuum cleaners for twenty years, but I hated hand dryers for even longer. If they had worked perfectly, I would have had no motivation to come up with a new solution. But more important, I would not have had the context to offer a creative solution. Failures feed the imagination. You cannot have the one without the other.”