Subscribe to News

    Popular Stories

    7+1 Principles And Five Frameworks for Agile Portfolio Prioritization
    Weave Toolkit: SAFe PI Planning with Weave
    Giving Thanks for Portfolio Management
    Agile 2015 Keynote: Awesome Superproblems
    Agile Portfolio Management: Five Core Frameworks for Product Portfolios
    Written by Luke Hohmann
    on October 17, 2017

    Few years ago, Luke Hohmann, Conteneo's CEO, was helping with a large scale enterprise Agile Transformation project lead by
     Applied Frameworks with several hundred engineers spread across multiple locations. The client was a cloud based infrastructure company who had grown very rapidly both organically and through multiple acquisitions. A key part of Applied Framework’s transformation services is starting every transformation with retrospectives so that we can better understand what all members of the product development function – from developers and quality assurance, to product management and customer care – think about what is going well and what needs to be improved.

    Speed Boat
     is our game of choice. But the size of the organization involved meant that we would be conducting multiple moderately large retrospectives of 30 to 40 people. This was costly, of course, and it took a lot of time, but the enlightened leadership of this company was committed to understanding the perspective of their employees before cramming Agile down their throats (sadly, too many companies are cramming Agile – a topic for another post)

    Following our own recommendations on game design, we had several teams playing Speed Boat in the same room. I’ll never forget when one of the developers in the game I was facilitating stated: “You know, Luke, this game is fine – we’ve played Speed Boat before with good results. And if you ask us about our anchors and propellers, we’ll tell you, but honestly, it won’t make much a difference. You see, my company was acquired because we’re a really good Agile shop with a great product. But now that we’re here, we’ve found multiple source code control systems, multiple requirements management systems, at least two corporate content repositories, and different testing frameworks. The difference is that when we were small we could change these things. Now we’re just a few teams out roughly 60 teams. We can’t change key things on our own. If you really want to help us, focus on the enterprise, and map out projects that can affect everyone. And sure, we’ll complain a lot as you help us standardize, but the reality is, we’ve hit a wall on what we can improve as individual teams”.

    This developer was right. I called an audible with my co-facilitators, grouped everyone into one big Speed Boat game and focused on inter-team and enterprise impediments. We then repeated this at three other development locations. It took time, and a considerable investment in logistics, but we identified and implemented some key projects such as standardizing on an ALM vendor and a source code management systems. These projects, which did indeed affect the enterprise, took months to implement, with high-impact results.

    This was my first Large, Distributed Team Retrospective (LDT/LDTR). We produced it using traditional techniques with direct support from the highest leaders in the company. But it was too costly and took too long. Since conducting this retrospective, I’ve been asking other organizations with LDTs (20, 50, 100, and even 250+ Scrum teams inevitably distributed across multiple time zones) about their experience with retrospectives. The results are not promising: most LDTs are not doing consistently conducting substantive retrospectives (I’ll expand on this later).

    If Retrospectives Are So Great, Why Do So Many LDTs Stop Doing Them?


    paper retro

    The simple answer is that traditional approaches to retrospectives – assembling a group people in a room with one or more facilitators – are too costly, don’t scale, take far too long and fail to produce high-impact results. As a consequence, large organizations either skip retrospectives entirely or they relegate retrospectives to individual teams, which tragically limit their effectiveness in identifying and implementing enterprise changes that can profoundly improve performance. Over time, because individual teams are not obtaining material benefits from retrospectives, they stop doing them at all.

    Speedboat CardSince then, we’ve changed our process to use Innovation Games® Online for LDTRs. Switching to online games enables organizations to conduct scalable, efficient, cost-effective, and high-impact retrospectives. Our game of choice is still Speed Boat. The core process is that each individual team plays their own online game at a time convenient to them (usually one or two one hours games is all that is needed). Multiple facilitators reduces biases. The producers of the event then download the results and analyze them to identify patterns of issues that affect the enterprise using advanced analytic techniques. These are shaped into projects and one or more are implemented. The process is faster and considerably lower cost than traditional in-person techniques.

    We’ve conducted LDTRs for Agile teams and even for other parts of the organization, such as sales. Given how much Agile has scaled over the past few years, I thought it was time to share our experiences and provide a playbook for others who want to use this process in their organization. In this post I’ll briefly review the motivations for retrospectives, review the challenges of existing techniques, and then present our proven process for conducting LDTRs. I’ll draw examples from several LDTRs we’ve conducted for our clients in sales and software development and present a framework that you can leverage in your organization.

    Oh – one final thing. This post is designed for people working in moderately large (10 teams/60+ people) to extremely large development organizations (250 or more teams with thousands of people).

    Retrospectives Really Are Great!

    Early in my career I had the good fortune to take Gerry Weinberg’s Problem Solving Leadership class. Norm Kerth, author of Project Retrospectives: A Handbook for Team Reviews, was my instructor, and he instilled in us the power of retrospectives.

    The same year Norm’s book was published we saw the publication of the Agile Manifesto, which lists retrospectives as one of the core principles of Agile Development:

    At regular intervals, the team reflects on how to become more effective,
    then tunes and adjusts its behavior accordingly.

    Since then, we’ve seen increasing wisdom in conducting retrospectives. For example, Certified Collaboration Architect Diana Larsen and her colleague Esther Derby wrote Agile Retrospectives: Making Good Teams Great, a book I find quite useful. And Diana continues to innovate in this space, sharing new retrospective techniques such as Circles and Soup and Anonymous Cards.

    I could go on – there are a tremendous number of Agile-related or Agile-inspired retrospective techniques, blogs and books – too many to enumerate there. It suffices to say the Agile community has embraced retrospectives.

    But Agile is not the only community to embrace retrospectives. Sales and Marketing Teams use retrospectives to improve their performance in two common ways. First, they conduct internal process retrospectives to identify opportunities for improvement. Second, they regularly conduct Win/Loss analysis to understand how to improve the entire marketing and sales process. You can find a practical guide to Win/Loss Analysis by Steve Johnson and Certified Collaboration Architect Jim Holland here.)

    The Retrospective Performance Curve

    If retrospectives are so great, why do large organizations stop doing them? To help answer this question, we’ve been asking hundreds of agilists to draw a retrospective impact curve. A composite of many individual curves is presented below. As you can see, there are four distinct phases of retrospectives for individual teams within large distributed teams.

    retro performance curve.png

    Why Bother?The last stage of retrospectives is when teams stop doing them. Oh sure, they might conduct a retrospective as a token ritual – a means to share beer at the end of the Sprint – but no one takes it seriously. And while we’re all for sharing some Tequila (or beer, as you wish) at the end of a Sprint, we think it might be better to change this curve.

    Phase Description
    Early Adoption In the early adoption phase, teams experience rapid improvements because they are often new to Agile and retrospectives help them fine tune their processes, learn new processes, understand new roles, and identify opportunities for improvement. This is often a time of substantial team-based productivity improvements, as teams focus primarily on their own team dynamics.
    Team Maturition As teams mature, the impact of their retrospectives starts to wane: The problems they identify are bigger and harder to fix, often involving coordination with other teams. We start to see a shift in focus from intra-team issues to inter-team issues.
    Organizational Limits To address these issues, many teams spontaneously choose to hold bigger, more comprehensive retrospectives, often with collocated teams with whom they collaborate. These retrospectives typically identify inter-team improvements and often some process and technical improvements. As these improvements are made, teams start to lose interest in retrospectives because they are no longer providing material value.


    Summarizing a bit, the root cause of large distributed teams stopping retrospectives is that they bump into the limits of what they can improve. And if they can’t improve, why bother with the retrospective?

    As a corollary, note that another root cause of large teams stopping retrospectives is that they conduct them too frequently and therefore too trivially. The Agile Manifesto never said to conduct a retrospective every Sprint!

    Challenges With Scaling Traditional Retrospectives

    Traditional retrospectives assemble participants in a room for a structured meeting that can last anywhere from 1 hour to as long as 1 day (you can find LOTS of very good advice on how to conduct traditional retrospectives; I won’t repeat that advice here). While getting together a single team for an in-person retrospective is often not more complex than booking a conference room, as the number of participants/teams increases, costs and complexity increase dramatically. We’ve produced in-person retrospectives for 30 people; and other Certified Collaboration Architects such as Henrik Kniberg have produced retrospectives for 65 people (see here).

    And while 65 people might seem large, we’ve produced dozens of in-person events that are much larger. We regularly produce Innovation Games® sessions with more than 100 participants; our record is a 500 person event for Intuit, and we’ve explored producing a 2,000 person event for the City of San Jose, CA as part of our Budget Games initiative through our non-profit partner, Every Voice Engaged Foundation.

    The two limiting factors for in-person games are costs and logistics. You can play around with a cost estimator later in this post. I’ll focus on various logistics challenges.

    Let us know what you think. 

    Add your comment below.

    You may also like:

    Agile Scrum PI Planning scaled agile framework SAFe product management

    Weave Toolkit: SAFe PI Planning with Weave

    A challenge faced by many SAFe organizations is creating an efficient PI planning process: one in which the in-person me...

    Agile Innovation Strategy safety diversity and inclusion

    Anonymous Participation: Collaborating Safely on Sensitive Topics

    We have released fully anonymous participation for  organizations that need to increase safety when tackling sensitive t...

    Agile Buy a Feature Decision Engine geoffrey Moore Horizon thinking McKinsey Participatory Budgeting

    How to Use Online Forums to Fix Your In-Person Meetings

    A few blog posts ago I wrote about how much I love to read. Along with books, some of my favorite magazines include Forb...