This website uses cookies to store information on your computer. Some of these cookies are used for visitor analysis, others are essential to making our site function properly and improve the user experience. By using this site, you consent to the placement of these cookies. Click Accept to consent and dismiss this message or Deny to leave this website. Read our Privacy Statement for more.
Book Review: Experimentation Matters
Share |
Book Review: Experimentation Matters by Stefan H. Thomke

Experimentation Matters: Unlocking the Potential of New Technologies for Innovation

Written by: Stefan H. Thomke. Boston, MA: Harvard Business School Press, 2003. 307+xi pages. US$35.00.
Reviewed by: Preston G. Smith, CMCNew Product Dynamics

Experimentation Matters

Experimentation, prototyping, and testing clearly have been an essential part of product development forever, but only recently has this activity come under scrutiny. One reason for this is that we have not recognized before that experimentation has been practiced differently in different companies and that this has resulted in differences in development performance. More immediately—and the theme of this book—is that certain new computer technologies allow experimentation to be done in ways that have the potential to improve development speed, cost, and the value of the resulting product dramatically.

Consequently, this book, which is among the first available on the subject and also the most helpful to date, has great potential to influence how product development is conducted. The title word “experimentation,” by itself, suggests too narrow a view of the book's scope, as it also covers modeling, testing, prototyping, and simulation. More simply, this is a book about trying things out to see what happens, which is an essential activity in product innovation. Consequently, Stefan Thomke's book applies to all types of product innovation and more strongly to highly innovative products, although each specific experimentation tool or technique applies more narrowly to certain types of products. Fortunately, the book interprets the subject broadly and provides examples from a wide selection of industries: automotive, banking, semiconductors, food, pharmaceuticals, and aerospace. Although most of the companies cited are huge ones, a case study from IDEO illustrates that the techniques apply equally to start-up environments and in fact probably are more natural there.

Experimentation Matters is written by an academic, but it is aimed solidly at the product development manager and student. Thomke's basis is a broad variety of well-crafted research techniques—interviews, surveys, case studies, “natural” experiments, and literature reviews—that are documented in endnotes but are not intrusive to the nonacademic reader.

Thomke's thesis is that by explicitly applying experimentation early and heavily in the product development cycle, an organization can enhance beneficial ideas and approaches and can discard weak ones early on ultimately to improve both the cost-effectiveness of the process and the market appeal of the resulting product. Moreover, recent advances in computer technologies make possible such radical changes in the use of experimentation that they have the power to change fundamentally how products are developed.

However, this is not a book about computer technology. Experimentation Matters makes it abundantly clear that simply installing computerized tools and training people to use them will not lead to substantial improvement. Thomke states, “… It is not necessarily whatthe technology is that matters but how it is used” (p. 145).

Experimentation is at the very heart of the development process and thus is connected to corporate values, habits, strategies, and organizational structures. These must change along with the technological changes to reap significant process improvement. For example, most organizations naturally focus on success, but efficient experimentation requires that the outcome of an experiment be viewed as learning whether the experiment is a success or a failure. Aligning an organization to learn from failure is more difficult than simply installing a technology.

Based on years of research in several industries, Thomke summarizes effective experimentation through six principles described in chapters 5 and 6:

  1. Anticipate and exploit early information through “front-loaded” innovation processes. The idea here is that it is more efficient to get your learning behind you early in a project by experimenting heavily initially to discover what works and what does not. These experiments often are relatively crude, but they establish and clarify the path being followed. IDEO's motto is “Fail often to succeed sooner.
  2. Experiment frequently but do not overload your organization. Besides being front-loaded, experimentation should be done through many simple experiments along the way, not through big “killer” tests at the end of the project to confirm decisions already made. However, the typical organization that moves to the frequent-experimentation mode can overload itself, thus slowing down decision making and defeating the value of the experimentation. You must provide plenty of capacity to build, to run, and to assess the increased volume of experiments, because queues here will undermine the fast feedback being sought.
  3. Integrate new and traditional technologies to unlock performance. The new computerized technologies, as wonderful as they are in many respects, usually have critical shortcomings that keep them from completely replacing traditional ones. Usually, the new technologies are 10 or more times faster and cheaper, but they provide lower fidelity results. Thus, it is not simply a matter of replacing the traditional with the new but of combining the two cleverly. This is not easy, because the new and traditional approaches often are allied with different corresponding cultures that thus also must be integrated.
  4. Organize for rapid experimentation. Because the principle of contemporary experimentation is to provide fast feedback from experiments that will influence current decisions, the organization must be capable of responding quickly. Many are not, so they must be made flatter, must be provided with greater authority at low levels, and must be supported with rapid communication tools. This is an area where the new technology must be augmented by organizational changes.
  5. Fail early and often, but avoid “ mistakes.” Thomke makes a clear distinction between failures and mistakes. Failures are food for learning, but mistakes are not, because the experiments from which they stem were not well planned, clearly focused, or conducted to control extraneous variables. Mistakes are “dumb” experiments from which nothing can be learned.
  6. Manage projects as experiments. This ratchets the whole process up one level by viewing each development project as an experiment itself so that it can be learned from and so that the next project will be conducted more effectively. Managing projects as experiments provides a methodology for continuous improvement of the development process for the “learning organization.”

At the risk of narrowing the view of the scope of this book, I provide one example from automotive research and development (R&D); others could be given from pharmaceuticals, yacht building, food flavors, or electronics development. A complex and time-consuming part of developing a modern automobile is establishing its crashworthiness. Traditionally, this has been done by building a prototype, by instrumenting it and its occupant dummies, and by crash-testing it. Such a prototype requires several months and up to US$500,000 to build and to test. Alternatively, modern computational tools, called finite element modeling, can simulate the crash in a computer using a model that takes a few weeks to build and run and costs less than US$5,000. Thus, the modern approach is 10 to 100 times faster and cheaper than the traditional one. It also has other advantages; for example, it can be “slowed down” and can be magnified to resolve detail that simply cannot be seen in a crash test. Much more importantly, results are available soon enough to influence the next round of design decisions, whereas with a crash test, results appear so late that they are useful mainly to confirm decisions that had to be made months ago.

However, a computer model is not a real car. Government regulators want to see the results of an actual test, and so do executives with deep roots in the auto industry. Because they have been fooled by computer models before, engineers want to back up each simulation with a test so that testing budgets can skyrocket beyond what they used to be. The argument of this book is: The new approach has huge advantages, but it also carries with it a large prove-it-to-me burden and requires great changes in values, mindsets, and organizational structure to view experiments as learning and to reengineer development processes to exploit this learning effectively.

Furthermore, it is not simply a matter of changing from the traditional to the new. Typically, the traditional approach will have clear advantages for some time while the new one improves, so the successful organization must integrate the two, not simply replace one with the other. And by the time that the new technology is completely superior, an even newer one is likely to appear that must be integrated.

Other aspects of experimentation strategy also will have to be worked out. For example, should several experiments be run in parallel to save time or should they be strung out sequentially to maximize learning? Thomke addresses this trade-off, which involves the cost of an experiment, the value of time, and the extent to which learning from one experiment is valuable to the next one.

There are further details to be considered. IDEO observes that a prototype should be rough, rapid, and right. “Rough” means that the prototype should be as simple and crude as possible to answer only the question at hand (many organizational forces encourage developers to build more elaborate prototypes than necessary). “Rapid” obviously is aimed at providing feedback when it is most useful. And “right” means that the prototype is focused specifically to answer the immediate question definitively.

Contemporary experimentation technologies allow decisions to be made faster, which can be exploited to make decisions later in the development process. Along with enabling organizational and cultural changes, this allows an organization to be more agile, purposely letting some decisions ride until late in the development process. For instance, rather than attempting to “freeze” product specifications before starting design, design can be started while continuing to firm up the specification through continuing technical, market, and customer experimentation—true concurrent development. Not only does this allow for the release of a more up-to-date product, but it also fits with reality better: Thomke cites research from Thomke and Reinertsen (1998) that fewer than 5 percent of developers have complete product specifications before beginning design anyway. Ultimately, this new view of specifications enables a shift from specification-driven development to experimentation-driven development.

Chapter 7 extends the experimentation theme from internal experimentation to involving customers in the experimentation by providing them with a toolkit to discover which products the supplier should manufacture for them. For example, for renovating a house, one can go to the website of a window manufacturer and can use their computer-aided design tools to experiment with window designs until the desired look is obtained. Then they will make it for you. This customer-based experimentation introduces additional organizational and cultural challenges beyond those of internal experimentation. For example perhaps the window salesperson formerly was supposed to be “pushing” newer, high-margin designs, but now customers are selling themselves.

I have mentioned the breadth of industries covered in Experimentation Matters. The author also describes, often in considerable detail, many experimentation technologies: combinational chemistry and high-throughput screening methods in drug development; field-programmable gate arrays (FPGAs) in electronics design; finite-element methods, interference checking and computer-aided styling in automotive and aircraft design; rapid prototyping for user interface development; computational fluid dynamics for yacht design; and formulation toolkits for food flavor design.

This book has a few flaws. The table of contents is sketchy, and I could find fewer than half of the items I attempted to locate through the index. The book seemed somewhat loosely organized and repetitious: Figures 3-2 and 2-7 are repeated as Figures 6-1 and 7-11, respectively, for example, and the ability of a finite-element model to zoom in on details (aforementioned) is described repeatedly on pages, 33, 104, and 205. Most importantly, in view of the author's strong—and wise—emphasis on making enabling organizational and cultural changes, he provides little guidance for doing this, although his Harvard Business School colleague John Kotter (2002) offers an excellent book on this that Thomke does not mention.

However, Experimentation Matters provides a wealth of information unavailable elsewhere on this timely topic. It explains how an emphasis on experimentation is changing the foundation of product development and what one can do to catch this new wave. It presents a much different view of the essence of product development than what can be found in the maturity model or product lifecycle management literature. I found it much more directly useful that Michael Schrage's (2000) more philosophical rendition in Serious Play. If one desires the benefits that can stem from contemporary experimentation, this book cannot afford to be missed.

1000 Westgate Drive, Suite 252
St. Paul, Minnesota 55114 USA
tel: 1-651-290-6280
fax: 1-651-290-2266

Sitemap | Email Deliverability

Privacy Policy

Copyright © 2020 PDMA. All rights reserved.