|Book Review: Experimentation Matters|
Experimentation Matters: Unlocking the Potential of New Technologies for Innovation
Written by: Stefan H. Thomke. Boston, MA: Harvard Business School Press, 2003. 307+xi pages. US$35.00.
Experimentation, prototyping, and testing clearly have been an essential part of product development forever, but only recently has this activity come under scrutiny. One reason for this is that we have not recognized before that experimentation has been practiced differently in different companies and that this has resulted in differences in development performance. More immediately—and the theme of this book—is that certain new computer technologies allow experimentation to be done in ways that have the potential to improve development speed, cost, and the value of the resulting product dramatically.
Consequently, this book, which is among the first available on the subject and also the most helpful to date, has great potential to influence how product development is conducted. The title word “experimentation,” by itself, suggests too narrow a view of the book's scope, as it also covers modeling, testing, prototyping, and simulation. More simply, this is a book about trying things out to see what happens, which is an essential activity in product innovation. Consequently, Stefan Thomke's book applies to all types of product innovation and more strongly to highly innovative products, although each specific experimentation tool or technique applies more narrowly to certain types of products. Fortunately, the book interprets the subject broadly and provides examples from a wide selection of industries: automotive, banking, semiconductors, food, pharmaceuticals, and aerospace. Although most of the companies cited are huge ones, a case study from IDEO illustrates that the techniques apply equally to start-up environments and in fact probably are more natural there.
Experimentation Matters is written by an academic, but it is aimed solidly at the product development manager and student. Thomke's basis is a broad variety of well-crafted research techniques—interviews, surveys, case studies, “natural” experiments, and literature reviews—that are documented in endnotes but are not intrusive to the nonacademic reader.
Thomke's thesis is that by explicitly applying experimentation early and heavily in the product development cycle, an organization can enhance beneficial ideas and approaches and can discard weak ones early on ultimately to improve both the cost-effectiveness of the process and the market appeal of the resulting product. Moreover, recent advances in computer technologies make possible such radical changes in the use of experimentation that they have the power to change fundamentally how products are developed.
However, this is not a book about computer technology. Experimentation Matters makes it abundantly clear that simply installing computerized tools and training people to use them will not lead to substantial improvement. Thomke states, “… It is not necessarily whatthe technology is that matters but how it is used” (p. 145).
Experimentation is at the very heart of the development process and thus is connected to corporate values, habits, strategies, and organizational structures. These must change along with the technological changes to reap significant process improvement. For example, most organizations naturally focus on success, but efficient experimentation requires that the outcome of an experiment be viewed as learning whether the experiment is a success or a failure. Aligning an organization to learn from failure is more difficult than simply installing a technology.
Based on years of research in several industries, Thomke summarizes effective experimentation through six principles described in chapters 5 and 6:
At the risk of narrowing the view of the scope of this book, I provide one example from automotive research and development (R&D); others could be given from pharmaceuticals, yacht building, food flavors, or electronics development. A complex and time-consuming part of developing a modern automobile is establishing its crashworthiness. Traditionally, this has been done by building a prototype, by instrumenting it and its occupant dummies, and by crash-testing it. Such a prototype requires several months and up to US$500,000 to build and to test. Alternatively, modern computational tools, called finite element modeling, can simulate the crash in a computer using a model that takes a few weeks to build and run and costs less than US$5,000. Thus, the modern approach is 10 to 100 times faster and cheaper than the traditional one. It also has other advantages; for example, it can be “slowed down” and can be magnified to resolve detail that simply cannot be seen in a crash test. Much more importantly, results are available soon enough to influence the next round of design decisions, whereas with a crash test, results appear so late that they are useful mainly to confirm decisions that had to be made months ago.
However, a computer model is not a real car. Government regulators want to see the results of an actual test, and so do executives with deep roots in the auto industry. Because they have been fooled by computer models before, engineers want to back up each simulation with a test so that testing budgets can skyrocket beyond what they used to be. The argument of this book is: The new approach has huge advantages, but it also carries with it a large prove-it-to-me burden and requires great changes in values, mindsets, and organizational structure to view experiments as learning and to reengineer development processes to exploit this learning effectively.
Furthermore, it is not simply a matter of changing from the traditional to the new. Typically, the traditional approach will have clear advantages for some time while the new one improves, so the successful organization must integrate the two, not simply replace one with the other. And by the time that the new technology is completely superior, an even newer one is likely to appear that must be integrated.
Other aspects of experimentation strategy also will have to be worked out. For example, should several experiments be run in parallel to save time or should they be strung out sequentially to maximize learning? Thomke addresses this trade-off, which involves the cost of an experiment, the value of time, and the extent to which learning from one experiment is valuable to the next one.
There are further details to be considered. IDEO observes that a prototype should be rough, rapid, and right. “Rough” means that the prototype should be as simple and crude as possible to answer only the question at hand (many organizational forces encourage developers to build more elaborate prototypes than necessary). “Rapid” obviously is aimed at providing feedback when it is most useful. And “right” means that the prototype is focused specifically to answer the immediate question definitively.
Contemporary experimentation technologies allow decisions to be made faster, which can be exploited to make decisions later in the development process. Along with enabling organizational and cultural changes, this allows an organization to be more agile, purposely letting some decisions ride until late in the development process. For instance, rather than attempting to “freeze” product specifications before starting design, design can be started while continuing to firm up the specification through continuing technical, market, and customer experimentation—true concurrent development. Not only does this allow for the release of a more up-to-date product, but it also fits with reality better: Thomke cites research from Thomke and Reinertsen (1998) that fewer than 5 percent of developers have complete product specifications before beginning design anyway. Ultimately, this new view of specifications enables a shift from specification-driven development to experimentation-driven development.
Chapter 7 extends the experimentation theme from internal experimentation to involving customers in the experimentation by providing them with a toolkit to discover which products the supplier should manufacture for them. For example, for renovating a house, one can go to the website of a window manufacturer and can use their computer-aided design tools to experiment with window designs until the desired look is obtained. Then they will make it for you. This customer-based experimentation introduces additional organizational and cultural challenges beyond those of internal experimentation. For example perhaps the window salesperson formerly was supposed to be “pushing” newer, high-margin designs, but now customers are selling themselves.
I have mentioned the breadth of industries covered in Experimentation Matters. The author also describes, often in considerable detail, many experimentation technologies: combinational chemistry and high-throughput screening methods in drug development; field-programmable gate arrays (FPGAs) in electronics design; finite-element methods, interference checking and computer-aided styling in automotive and aircraft design; rapid prototyping for user interface development; computational fluid dynamics for yacht design; and formulation toolkits for food flavor design.
This book has a few flaws. The table of contents is sketchy, and I could find fewer than half of the items I attempted to locate through the index. The book seemed somewhat loosely organized and repetitious: Figures 3-2 and 2-7 are repeated as Figures 6-1 and 7-11, respectively, for example, and the ability of a finite-element model to zoom in on details (aforementioned) is described repeatedly on pages, 33, 104, and 205. Most importantly, in view of the author's strong—and wise—emphasis on making enabling organizational and cultural changes, he provides little guidance for doing this, although his Harvard Business School colleague John Kotter (2002) offers an excellent book on this that Thomke does not mention.
However, Experimentation Matters provides a wealth of information unavailable elsewhere on this timely topic. It explains how an emphasis on experimentation is changing the foundation of product development and what one can do to catch this new wave. It presents a much different view of the essence of product development than what can be found in the maturity model or product lifecycle management literature. I found it much more directly useful that Michael Schrage's (2000) more philosophical rendition in Serious Play. If one desires the benefits that can stem from contemporary experimentation, this book cannot afford to be missed.