Brief
As a 21st-century Mark Twain might observe, everybody talks about innovation, but nobody does anything about it. Of course, that’s not quite right. Several path-breaking thinkers have come up with better ways to design new products. The problem is that anywhere from 70% to 90%1 of those new products continue to fail. Lacking a systematic, repeatable and fast-moving method for designing and developing innovations, companies find themselves struggling to keep up with market changes.
But there is hope. Agile methodologies have transformed the software industry over the past 25 or 30 years. Software development is an especially challenging form of innovation since technologies and customer demands seem to change at the speed of Moore’s law. Software is also playing an increasingly important role in nearly every element of business. (As they say, every company is now a software company, whether its executives know it or not.) So it’s worth looking into how the industry has changed and what lessons it might hold for innovation in the rest of the organization.
Doing Agile Right
Our latest book is a must-have guide to transform your organization through the power of agile...the right way.
When we do, we find some remarkable facts. Agile isn’t just one more approach to creative thinking or iterative prototyping. Rather, it’s a well-developed holistic system engineered to overcome more than a dozen common barriers to successful innovation. It’s also highly effective. In tens of thousands of software development projects, Agile methods have boosted average success rates to 39% from 11%, a more than threefold improvement. In large, complex projects Agile’s success rate jumps to six times that of conventional methods.2
Agile Insights
Agile Innovation techniques are spreading beyond software development to a wide range of industries and functions. Bain's Agile Insights explore how companies can adopt scrum thinking.
We also find that more and more companies are now adopting Agile methodologies in other parts of the organization. Functions as diverse as R&D, marketing, operations and corporate strategy are tapping Agile’s value. Companies in industries as varied as broadcasting, farm equipment manufacturing and winemaking are importing the techniques. Early data from these experiments suggest an equally high success rate, reflecting Agile’s systematic and repeatable characteristics. Agile thus offers the prospect of a holistic innovation methodology that many companies can learn and put to work—and that will enable those companies to develop and implement new ideas, not just come up with them.
To be sure, many questions remain. In which contexts is Agile most useful, and in which is it less relevant? What does Agile look like and feel like when it ventures outside of its software-based home? What role should senior executives, who were mostly spectators during IT’s transformation, play in such an ambitious undertaking? What are the keys to success? Let’s turn to those questions.
 |
INTERACTIVE DIAGNOSTIC
How Agile Is Your Organization?Take our 12-question diagnostic to discover your organization’s agility quotient. |
How Agile operates
The fundamentals of Agile are simple. To tackle an opportunity, the organization forms and empowers a small, focused, cross-functional, self-managing team. The team’s initiative owner, who typically comes from a business function and divides his or her time between the Agile team and key stakeholders, uses techniques such as design thinking to build a catalog of promising ideas or features. The initiative owner continuously (and ruthlessly) ranks that list based on the latest estimates of value to customers, financial results and other innovation initiatives. A process facilitator protects the team from distractions and puts its collective intelligence to work.
The team then breaks top-priority tasks into small modules, decides how much work to take on and how to get it done, and starts building working versions in short cycles known as sprints. The process is transparent to everyone. Team members hold brief daily stand-up meetings to review progress and identify impediments. They resolve disagreements with experimental feedback loops rather than through endless debates or appeals to authority. They test small working increments with groups of potential customers. If customers get excited, the increment may be released immediately, even if the boss isn’t a fan or others think it needs more bells and whistles. The team then brainstorms ways to improve future cycles and prepares to attack the new top priority.
This approach systematically targets common impediments to software projects and other forms of innovation. It frees senior managers from micromanaging, enabling them to spend more time strategizing, removing impediments and increasing cross-functional collaboration. It increases customer engagement and satisfaction by improving visibility and adapting to the customer’s changing priorities. It brings the most valuable products and features to market faster. It minimizes the waste inherent in meetings, repetitive planning, formal documentation, quality defects and low-value product features. The process also aims to create team members who are happier, more creative, more committed to success and better trained for advancement, thus reducing employee turnover.
Because Agile relies on empirical feedback loops and full transparency, performance metrics are integral to the process. Practitioners regularly monitor changes in metrics such as customer satisfaction, quality, speed and employee engagement, and they have shared data on tens of thousands of projects with third-party researchers. According to these studies, more than 90% of IT organizations report using Agile methodologies in at least some of their software development, and 45% of individual respondents work in organizations that use Agile for the majority of their development teams.3 Agile users describe an array of specific improvements when they switch to the new techniques (see Figure 1). Overall, ratings of return-on-investment effectiveness are more than five times higher for Agile.4 Customer satisfaction is higher as well5—in fact, many customers, including some departments of the US government, now require outside contractors to use Agile methods.
The spread of Agile innovation
Over time, Agile techniques have migrated to other parts of the business. Perhaps an executive was impressed with the improvements in IT and wanted to find out what was behind them. Perhaps an innovation team learned about the new methods and decided to apply them in its own work. Whatever its genesis, this migration is more widespread than many business leaders realize. In Scrum Alliance’s recent survey of 4,452 Scrum users—Scrum is the most popular form of Agile—more than half of the respondents reported that their organization is using Scrum beyond IT, and nearly a quarter work in those non-IT departments. The list includes product development (11%), operations (3%), sales and marketing (2%), and even C-level executives (1%).6 Success rates outside of IT are comparable with those within IT. IT respondents in this study reported a success rate of 63%; non-IT respondents put it at 59%. Eighty-seven percent of IT respondents said that Scrum had improved their teams’ quality of work life; 84% of non-IT respondents agreed. Eighty percent of IT respondents said that “delivering business value to the customer” was one of Scrum’s benefits most valued by executives; 77% of non-IT respondents agreed. Nearly everyone on both sides said they would continue using Scrum.
These findings are consistent with our own experience. Having worked with hundreds of Agile teams in a broad range of industries and business functions—and examining more than 30 in significant detail—we find similar patterns of success. Innovation, which is best defined as the profitable application of creativity, always aims to do two things: design breakthrough solutions to important customer problems and develop those solutions economically. It’s about design and development, and it must be tightly integrated and rapidly adapted to the direction and pace of market changes. That’s precisely what Agile does.
Three brief examples help to illustrate the typical pathways and benefits of Agile’s migration beyond IT.
John Deere. A recognized leader in advanced technology in agricultural and construction machines, John Deere created a global technology network of research and development centers staffed with innovators. The centers’ job: Continue to build on the company’s legacy of revolutionizing the markets it serves.
George Tome came to John Deere in 1998 as project manager within the company’s corporate IT group. Tome brought with him a strong background in software development. Later, he became manager of Program/Project Management in Enterprise Advanced Marketing within Deere’s Global Technology Innovation Network, which has responsibility for discovering technologies that could revolutionize the company over the next 5 to 10 years. But he was growing increasingly frustrated with traditional project management approaches, so he began experimenting with Agile principles “under the covers,” as he put it to us. From 2010 to 2012, as Deere’s IT department moved aggressively to shift more than 800 developers into Agile teams, the culture of the organization opened up to broader expansion of the concepts.
With support from Jason Brantley, director of Enterprise Advanced Marketing, Tome and two colleagues attended Agile training classes. The only problem was that all of the terminology and examples came from software, and they were complete gibberish to Tome’s colleagues. So Tome worked with a coach named Joe Justice, whom he describes as a master in teaching Agile “with meaningful real-world non-software examples.” Soon he had a custom-designed, Agile-based set of tools which he dubbed “XI,” for eXtreme Innovation. The goal was to “think unreasonably big, work as iteratively and as small as practical, deliver faster than what’s been possible, adjust and adapt constantly.” He trained teams in all five John Deere Global Technology Innovation Technical Centers around the world and began publishing weekly one-page articles on Agile principles inside the company. John Deere’s innovators target long-term disruptions that may require 5 to 10 years to fully develop and bring to market. They typically take about nine months to identify a new market opportunity, develop the basics of a solution that meets customer needs and test the solution. Using Agile techniques and a team of individuals who were already familiar with the principles, the company was able to compress this time frame by more than 75%. A more radical example was the development of a new “machine form”—a different kind of equipment, the specifics of which remain confidential. In this case Deere went from having an idea about a customer problem that it wanted to solve to a working prototype of the new machine in about eight months. “If everything went perfectly in a traditional Waterfall process, it would be a year and a half at best,” Jason Brantley says, “and it could be as much as two-and-a-half or three years.”
According to Tome, “Almost every area [at John Deere] has someone starting to think about Agile and at least dabble with it.” One reason may be the data that supports this shift. When Tome introduced the concept, he immediately set out to measure changes in team happiness, work quality and velocity. Innovation team happiness scores shot from the bottom third of the company to the top third. Quality, as measured by innovative product projects, also improved. Velocity, as measured by the amount of work accomplished each week, increased on all teams by more than 200%, with some exceeding 400% and one exceeding 700%.
An Executive's Guide to Agile Innovation
A look at the basics of Agile and how it differs from traditional methods.
NPR. For more than 40 years, NPR (formally, National Public Radio) developed its programs the old-fashioned way. Producers came up with ideas and pitched them to NPR executives. Those who got the green light hired the staff they needed, created extensive programming and geared up for a big launch, all in strict secrecy. Meanwhile, NPR’s reps tried to sell the show to local stations, hoping that enough stations would buy in to cover the show’s costs. The process was slow, expensive and risky.
Facing an acute budget shortfall a few years ago, NPR’s board asked the programming department to come up with a new approach to developing shows. Eric Nuzum, former vice president for programming, was already intrigued by striking improvements in NPR’s Digital Media department, which had completely revamped the organization’s website in a relatively short time. So he turned to colleagues there for inspiration, and they described for him the Agile approach to software development. Nuzum and several others began thinking about how to apply Agile principles to programming and other projects at the network.
Today, Agile has become a kind of watchword at NPR, turning up in unlikely places. One team adopted Agile practices to manage the creation of a new digital archive of more than 750,000 records. Another team applied the methods to optimize the breakdown of a typical broadcast hour between national news and local information like news, traffic and weather. In programming, the network now creates a small number of pilots with a minimal staff and then begins iterating. Show developers gather feedback from local program directors. They ask listeners for critiques and suggestions, often through social media. The process is quick, public and dramatically reduces risk. In the end, this saves money. NPR developed programs such as the TED Radio Hour and How to Do Everything (a podcast) at one-third previous cost levels. Because it has spent less money on development, NPR can feed the program to local outlets free of charge for a while so that they can build an audience.
Mission Bell Winery. Erik Martella, vice president and general manager of Mission Bell Winery (a unit of Constellation Brands), introduced Agile to the organization from the top. Martella wanted Mission Bell to qualify for Safe Quality Food (SQF) Level 2 certification. It was a daunting process, and Martella wondered how quickly a 100-year-old organization could embrace so much change. While reading a book on Scrum, he decided to test the approach on the SQF certification process.
Martella described the concept to his management team, suggested that they read the book and brought in Agile trainers, who conducted a daylong workshop to introduce the tools. During that workshop, he began to see opportunities for Agile innovation in many parts of the organization. The training was quickly broadened to include more people, and was followed up with a twoday session for 45 employees and two days of coaching. Martella encouraged everyone to post sticky notes with ideas to make the winery a happier and more productive workplace. He grouped all of these ideas, added a few of his own, created a single prioritized list and established Agile pilots in each department—winemaking and specialty products, cellar operations, distribution, quality, bottling, even plant maintenance.
Mission Bell’s distribution unit, whose 45 employees ship 12 million cases of wine a year, offers an example of Agile’s effects. The unit’s first sprint focused on ways to improve innovation in special projects such as how to increase product flows through the warehouse. The main impediment team members identified was “normal daily work getting in the way,” so they began by finding better ways to accomplish daily activities and free up more time for special projects. While improving those processes, they identified interruptions from other Constellation team members as another major impediment. They formally allocated 10% of their daily time to handling such interruptions and created more effective ways for dealing with them.
Within three months, the distribution team’s ability to solve such problems accelerated more than tenfold. Staging and loading productivity improved from 411 cases per hour to 425. The ISO 9001 recertification audit was completed with zero exception findings. The accuracy of the annual finished goods inventory increased by 90%. The SQF certification is still in process, but Martella believes the new tools are definitely helping. “We’re trying to put all the components of SQF in place using employees who also have full-time day jobs,” he told us. “I’m not sure how we’d have tackled such a major project without Scrum.”
Distribution director Mark Nichols finds the effect on his team members equally important. The sprint planning process, he says, gives the team control over the amount of work they will pull into a sprint and thus encourages people to take responsibility. Members feel a sense of excitement as they watch their velocity accelerate. The process empowers the team to resolve most issues and immediately escalates issues beyond their control to the senior Executive Action Team. “Switching from command and control (‘you will get this done’) to team empowerment (‘we can get this done’) has a very positive effect on people,” Nichols told us in an e-mail. “We would not continue to show acceleration without this mindset and buy-in from our team.”
Darrell Rigby, who leads Bain's Global Innovation practice, discusses how Agile innovation teams can help companies quickly pivot to take advantage of market opportunities.
Keys to success
Despite the progress of Agile methodologies, many companies are reluctant to adopt them—and not every company that does is succeeding. Data on failures suggest that the major impediments fall into three key categories:
- Inability or unwillingness to apply the methodology. Forty-four percent of survey respondents blame failure on lack of familiarity with Agile methods. About 35% say that there are not enough personnel with the necessary experience while 33% cite the unwillingness of the team to follow Agile practices.7 About 80% agree that training enhances the adoption of Scrum.8
- Lack of management support. Thirty-eight percent blame failures on lack of management support; 22% cite management’s concerns about a loss of control.9 Support from senior management outweighs other success factors by at least fivefold when organizations successfully adopt Scrum.10
- Agile principles at odds with the company’s operating model. More than 40% blame company philosophy or say the culture is at odds with core Agile values;11 71% see tension between Scrum teams and the rest of the organization.12
When companies or teams focus on overcoming these three impediments, they improve the odds of success.
1. Adhere to principles; evolve the practices
A major advantage of Agile methodologies is that practitioners have had more than 25 years to test and codify what works and what doesn’t. Both data and experience suggest that the closer a company adheres to Agile principles, the more likely it is to achieve the hoped-for benefits. Skilled Agile teams record success rates twice those of unskilled teams.13 Scrum projects deployed through an experienced program management office report success rates of 93%.14 Teams that are at least 95% dedicated to their projects are nearly twice as productive as those that are less than 50% dedicated. Stable teams deliver 60% better productivity and 60% better responsiveness.15
But there’s a difference between principles and practices. Agile principles are enduring and widely applicable. Agile practices, in contrast, may evolve as teams gain experience and technologies advance.
Spotify, the popular music-streaming company, has geared its entire business model, including everything from product development to marketing and general management, to support Agile innovation. But managers do not dictate specific practices; on the contrary, they encourage experimentation and flexibility so long as the changes are consistent with Agile principles and can be shown to improve outcomes. As a result, practices vary considerably across the company’s 70-plus development squads and its functional chapters. For example, nearly every squad uses some form of visual progress tracking, small and crossfunctional teams, time-boxed work iterations, ranked priorities, adaptive planning, continuous improvement retrospectives and a focus on creating value over keeping people busy. But many squads do not use the burndown charts that show remaining work, a common feature of Agile teams. And few measure velocity, keep time reports or utilize the same work estimation techniques.
OpenView Venture Partners illustrates a different kind of adaptation. OpenView is a venture fund that has invested in close to 30 software companies. When founder Scott Maxwell learned about Agile, he concluded that “it isn’t a software development technology, it’s a management system.” He decided to try it out first at OpenView and then at as many of its portfolio companies as possible.
At OpenView itself, the prioritized backlog was an immediate and easy success. The team quickly cut 30% of its projects, actively identifying things not to do. Retrospectives, held every Monday morning to identify opportunities for continuous improvement, were also a hit. So was focusing team members on fewer projects—most had been overwhelmed by working on 10 or 20 projects at a time.
Within months, team productivity improved between 80% and 100%, allowing Maxwell to foster a culture of working fewer hours and freeing up weekends. Other practices, however, required testing and tuning. Team members experimented with various techniques for sizing the amount of work required from each item. They varied the number of members per team to see what worked best, and they changed the time devoted to daily Scrum stand-up meetings from 45 minutes to 15 minutes. They initially set sprint cycles at one week, tried two weeks and then returned to one week. The organization eventually landed on practices that increased productivity by at least 150%, according to Maxwell.
As OpenView tried rolling out Agile practices to its portfolio companies, Maxwell discovered that some were interested while others were not, and that Agile was hard to deploy in some functions while others were easy. Applying Agile to sales functions, for example, proved difficult—the backlog needed too much updating too frequently—and Maxwell’s team deprioritized it. Agile product development proved valuable and relatively easy. So did Agile marketing. For example, at portfolio company Intronis, a leading provider of cloud backup services for IT departments, marketing needed to rethink its tradeshow-anchored calendar and improve its combative relationships with the sales department. The company hired Richard Delahaye, a web developer turned marketer from the software industry, to shake up the unit.
Delahaye immediately implemented Agile, with positive results. In early 2014, for instance, a junior technologist learned of the CryptoLocker malware, which enabled hackers to remotely hijack computers and demand ransoms. Many believed the topic warranted an immediate campaign, perhaps starting with a webinar, even though there was no plan for it on the calendar.
Previously, a webinar would have required four to six weeks to organize. The team would have had to debate whether security was a worthwhile topic, prepare a senior Intronis executive to address it, find an external expert for validation, line up a customer who could describe the problem and solution, build a web page and promote the campaign for weeks to enroll enough attendees. This time, Delahaye opted for a “minimum viable campaign” that would reach customers before competitors could.
Within a week, he had signed up 600 registrants (a company record) and put the terrified junior technologist in the presenter’s seat. The feedback was so positive that Intronis rapidly increased marketing programs on security topics and soon created a new business segment. Today, Intronis still creates calendars and budgets for the digital marketing unit, but with far less line-item detail and with greater flexibility for serendipitous developments.
2. Beware of Big Bang Agile deployments
Large companies typically go about transforming themselves in waterfall fashion. They expect everyone to “go big or go home.” They hold conferences, launch training programs and create a blizzard of memos, Gantt charts and templates. They dot the hallways with posters and the desks with slogan-carrying Lucite blocks.
The most successful Agile implementations, however, start small. They often begin in IT, where software developers are likely to be familiar with the principles. And then maybe Agile spreads to another function, as it did at NPR, with the original practitioners acting as coaches. Each success seems to create a group of passionate evangelists, who can hardly wait to tell others in the organization how well Agile works. A leader’s job is to help this critical mass of Agile teams develop. Executives need to get people focused on the right problems, provide strategic direction, eliminate impediments and enable cross-functional collaboration.
They also need to lead by example, which itself is likely to produce significant benefits. Systematic, a global software company that provides IT solutions for healthcare, defense and many other applications, illustrates the possibilities. Its developers began adopting Agile methodologies several years ago. Founder and CEO Michael Holm, though initially skeptical, eventually came to appreciate the benefits. But what was he doing to help?
“I had this feeling that I was saying, ‘Follow me, I’m just behind you,’” he told us. “The development teams were using Scrum and were doing things differently while the management team was stuck doing things the same old-fashioned way.” So Holm decided to run his executive group as an Agile team. Senior managers were pulled out of their offices and colocated with their teams. Their office space was repurposed as a “situational awareness room,” where anyone could go to view and discuss the real-time status of projects. Systematic’s nine-member leadership team had been meeting every Monday for an hour or two. Now the team meets daily at 8:40 AM for a 20-minute stand-up discussing what they did yesterday, what they will do today and where they need help. Other functions, including HR, legal, finance and sales, operate in much the same way.
At Mission Bell Winery, the leaders of each department initially served as product owners on the various Agile teams within their departments. But their time was being spread too thin across Agile teams and daily operating activities, and it was hard to keep department priorities aligned with enterprise priorities. Now, Erik Martella is pulling department leaders into an Agile executive leadership team focused on enterprise initiatives that hold the greatest value and the greatest opportunity for cross-functional collaboration.
The team is responsible for building and continuously refining the backlog of enterprise priorities, ensuring that Agile teams are working on the right problems and have sufficient resources. They also protect the organization from pet projects that don’t deserve high priority. Shortly after Martella started implementing Agile, he received an email from a superior suggesting that the winery explore a personal passion of that executive. Normally Martella would have responded, “OK, we’ll jump right on it.” Instead, he replied that the winery was following Agile principles. The idea would be added to the list of potential opportunities, objectively sized and prioritized. As it happened, the executive liked the approach, and when the prioritization was low, he reacted well to the outcome.
3. Build an Agile architecture
In IT, architecture refers to the standards and technologies that enable collaboration across the enterprise. Agile software development is often inhibited by a company’s legacy architecture, because the new software can’t easily plug into the rest of the system. It’s much the same with Agile teams.
For example, a large financial services company we examined wanted to improve its agility, and it launched a pilot to build its next mobile app using Agile methodologies. Of course, the first step was to assemble a team. That required a budget request to authorize the project and fund the people. The request was batched with other initiatives waiting for the annual planning process. After months of reviews, the company finally approved funding. The app development pilot actually went quite well, and the team was proud of its work. Before being released, however, it needed to pass vulnerability testing, which would be conducted in a Waterfall process that already had a lengthy waiting list. Then it needed to be integrated into the core IT systems—another Waterfall process with a six-to-nine-month logjam. In the end, the total time to release improved very little.
These productivity gains illustrate the problem of different parts of an organization moving at significantly different speeds—a phenomenon that has led George Tome, of John Deere, to formulate a rule that someone else dubbed Tome’s Law. The law holds that commercialization velocity, or time to market, equals team velocity (the speed at which a team produces new value) plus organizational velocity (the speed at which the larger organization accepts and integrates this new value). If a team takes four weeks to create a new product and the organization 16 weeks to integrate it, the commercialization velocity is 20 weeks. If the team doubles its productivity to two weeks but the organizational integration doesn’t improve at all, the commercialization velocity will improve by only 10%.
In our experience, implementing Agile at the team level is relatively easy. Results improve quickly. So does team happiness. The bigger challenge is how to handle Agile’s rapid growth as it catches hold and rolls out to other functions. That’s where the idea of an Agile architecture comes in, and the leading companies build it on five pillars:
- Everyone on the same page. Individual teams focusing on small parts of large, complex problems need to see and work from the same ranked list of enterprise priorities. If a new mobile app is top priority for software development, it must also be top priority for marketing, budgeting, vulnerability testing and software integration.
- Change in roles before change in structures. Many executives assume that more cross-functional teams will require organizational restructuring. Seldom is that true, and usually it is detrimental. New structures can shatter trust and impede creativity for years, especially if they involve downsizing. Crossfunctional teams do, by definition, require some form of matrix management. But the most important changes are clear decision rights and team self-governance. The roles change in beneficial ways even though the lines on organization charts look the same.
- Only one boss for decisions. In an Agile architecture, it must be clear who commissions cross-functional teams, who selects and replaces team members, who appoints the team leader and who approves a team’s decisions. People can have multiple bosses, but decisions cannot. Senior leaders need to avoid second-guessing and overturning product owners’ decisions. It’s fine to provide guidance and assistance, but if you don’t like the results, change the product owners rather than incapacitating them. If you’re going to hire extraordinary people, give them extraordinary roles and treat them with extraordinary respect.
- A focus on teams, not individuals. While research shows that the general intelligence of individuals can affect team performance, the collective intelligence of a team is even more important. It’s also far easier to change. Agile teams use process facilitators (usually called Scrum Masters) to continuously improve their collective intelligence. Metrics evolve from tracking outputs and utilization rates (how busy people are) to business outcomes and team happiness (how valuable and engaged people are). Recognition and reward systems shift to weight team results more than individual efforts.
- Questions, not orders. General George S. Patton Jr. famously advised leaders never to tell people how to do things. “Tell them what to do, and they will surprise you with their ingenuity.” Rather than responding to arguments with orders—“because I’m the boss, and I said so!”—Agile leaders learn to guide with questions: “What do you recommend?” and “how could we test that?” Senior leaders grow from functional experts to general managers and enterprise strategists. Cultures move from siloed battles for power and resources to collaborative cross-functional teams striving to achieve shared goals.
• • •
Agile practitioners like to develop minimum viable products, early iterations that can be put out into the marketplace and tested with customers so that they can be improved. This article has been purposely designed as a minimum viable product. Our hope is that it will help executives who are unfamiliar or unsuccessful with Agile methodologies learn what’s involved and test them out.
Does Agile raise the cultural speed limit? Does it lead to improved value for customers? Does it engage employees and raise their overall level of happiness, as John Deere and others have discovered? We think that Agile is a big new idea that will have widespread application in the future of business. But we look forward to the continued testing of that proposition, and to continued improvement in the methodologies involved.
The birth of Agile
Traditionally, the so-called Waterfall process dominated software development. Work trickled down through organizational silos in sequential fashion. For example, marketing might identify a customer need. The resulting project flowed first to design, then to development, then to testing and integration, and finally to deployment. The Waterfall approach relied heavily on predictive planning, extensive documentation, tight controls and eventual delivery of a product that conformed strictly to original specifications. The process was slow, fraught with waste and demoralizing to developers (see figure below).
In the early 1990s, a developer named Jeff Sutherland began collaborating with software expert Ken Schwaber and others to create a better system. One major inspiration was a 1986 Harvard Business Review article called “The New New Product Development Game,” written by Hirotaka Takeuchi and Ikujiro Nonaka. Examining innovation in large multinational manufacturers, the authors contrasted conventional “relay race” methods of product development (which sounded remarkably like Waterfall processes in software) with a holistic or “rugby” approach, “where a team tries to go the distance as a unit, passing the ball back and forth.” Paying tribute to the rugby metaphor, Sutherland and Schwaber dubbed their newly created system Scrum.
While Sutherland and Schwaber were developing Scrum, others built like-minded alternatives to Waterfall, with names like Extreme Programming and Adaptive Software Development. In 2001, Sutherland, Schwaber and 15 other software revolutionaries (several of whom were spirited rivals) gathered in Snowbird, Utah, to share their insights and discuss common characteristics of success. Although they disagreed on much, they eventually forged a consensus on a name for their approach, Agile, and a call to arms that spelled out essential values and operating principles. Henceforth, development frameworks that aligned with these values and principles would be known as Agile techniques. Scrum and its derivatives are used at least 10 times as often as any other variety of Agile, so we sometimes default to its methodologies when illustrating Agile practices.
When and where does Agile work?
Experienced Agile practitioners find that its methodologies are proving superior in a growing range of applications, not only because the conditions favoring Agile are spreading so rapidly but also because it can be adapted to fit so many situations (see figure below). For example, Agile seeks to avoid excessive documentation, but if documentation is required, Agile can increase documentation as easily as Waterfall can, albeit with fewer initial benefits.
How Agile triples the rate and value of innovation
A common question is how Agile methodologies could possibly triple the productivity of an innovation project. To illustrate, we’ll describe how a leading apparel retailer might redesign its dressing room experience.
The company’s innovation group identifies this experience as a vital opportunity to improve customer satisfaction and increase sales. A series of workshops envisions a new concept with three key features:
- augmented reality “magic mirrors” that show customers how apparel will look on them without trying it on;
- a delivery system that lets customers select apparel electronically and have the items placed inside their personalized dressing rooms immediately or at a prearranged time; and
- videoconferences with specialized “remote stylists” who offer personalized fashion advice and can send recommended items to customers’ dressing rooms.
The innovation group believes that all three features are critical to the redesign and asks the six-person engineering team how long it will take to develop them concurrently. Engineering’s answer is 18 months. But the engineering team has a better idea.
The engineers explain to the innovators that fragmented multitasking kills productivity, squandering time and increasing errors. Grappling with three complex tasks simultaneously would waste 40% of the team’s time on “context switching”—that is, interrupting workflows and forcing team members to pause, restart, reload and refocus their problem-solving processes (see figure below, “Fragmented multitasking kills productivity”). Instead, the engineers propose to work on only one major feature at a time.
But which should they start on? How can the company rank development activities when all the features are essential? The engineering team suggests a methodology based on revenue potential, required investments and contributions to other initiatives. For example, the delivery system (feature No. 2) offers the highest revenue and profit potential along with the lowest and most predictable development costs. It will also significantly enhance feature No. 3 by enabling remote stylists to send recommended items to customers’ dressing rooms almost instantaneously. Working together, the innovation and engineering teams assign the delivery system a value of 10, remote stylists a value of 7 and the technologically riskier magic mirrors a value of 5.
Taking a two-year perspective, the team quantifies the comparative value of an Agile approach. If the company tries to develop all features concurrently, the entire dressing room redesign will take 18 months to complete, and each feature will then create value for 6 months. Multiplying the value of each feature by six and then adding them together generates a value index of 132 (see figure below, “24-month scenarios”).
But suppose the team sequences its activities to complete the most valuable features earlier? In that case, the delivery system’s benefits will start flowing in 6 months, and remote stylists will begin generating benefits in 12 months. The value index doubles.
That isn’t all. By focusing on one feature at a time, the apparel company’s engineers minimize context switching costs and reduce development times by 40%. They complete the entire project in 10.8 months, nearly tripling the value index. Moreover, this valuation excludes other Agile benefits such as minimizing the waste inherent in unproductive meetings, repetitive planning, excessive documentation and quality defects. It also enables the team to develop features iteratively with active customer participation, adapt rapidly to changing priorities, delay the risky bet on nascent magic mirror technologies and increase cross-functional collaboration. The result: an innovation that delights customers long before competitors can respond.
Darrell K. Rigby is a partner in the Boston office of Bain & Company. He leads the firm’s Global Retail and Global Innovation practices and is an expert in Bain’s Digital practice. Steve Berez is a partner in Bain & Company’s Boston office. He is a cofounder of Bain’s Information Technology practice, which he currently leads in the Americas, and an expert in Bain’s Digital practice. Greg Caimi is a partner in Bain & Company’s San Francisco office. He is an expert in Bain’s Digital practice and a member of the firm’s Technology, Media and Telecommunications, Organization, and Results Delivery® practices. Andrew Noble is a partner in Bain & Company’s Boston office. He is a member of Bain’s Retail, Digital, Customer Strategy and Marketing, and Organization practices.
1 Beth Altringer, “A New Model for Innovation in Big Companies,” Harvard Business Review, November 19, 2013, https://hbr.org/2013/11/a-new-model-for-innovation-in-big-companies
2 The Standish Group, CHAOS Report 2015
3 VersionOne, 9th Annual State of Agile Survey, 2014, https://www.versionone.com/pdf/state-of-agile-development-survey-ninth.pdf
4 Scott W. Ambler, “2013 IT Project Success Rates Survey Results,” http://www.ambysoft.com/surveys/success2013.html
5 Tom Grant with Dave West and Alissa Anderson, “Increase Agile Efficacy to Improve Customer Value,” Forrester, January 10, 2012, https://www.forrester.com/Increase+Agile+Efficac y+To+Improve+Customer+Value/fulltext/-/E-RES61254
6 Scrum Alliance, The 2015 State of Scrum Report, https://www.scrumalliance.org/scrum/media/scrumalliancemedia/files%20and%20pdfs/state%20of%20scrum/scrum-alliance-state-of-scrum-2015.pdf
7 VersionOne, 9th Annual State of Agile Survey, 2014
8 Scrum Alliance, The 2015 State of Scrum Report
9 VersionOne, 9th Annual State of Agile Survey, 2014
10 Scrum Alliance, The 2015 State of Scrum Report
11 VersionOne, 9th Annual State of Agile Survey, 2014
12 Scrum Alliance, The 2015 State of Scrum Report
13 The Standish Group, CHAOS Report 2015
14 Scrum Alliance, The 2015 State of Scrum Report
15 CA Technologies, The Impact of Agile. Quantifi ed, 2015, https://www.rallydev.com/sites/default/files/ImpactofAgileQuantified2015.pdf
First published in April 2016