You Can’t Have Your Agile Cake and Eat it Too!
It’s probably safe to say that many organizations that perform software development are either using some form of Agile methodology or are in the process of implementing Agile. Management in these organizations very likely believes that Agile will deliver numerous benefits to the organization thereby allowing them to receive numerous accolades, applause, and recognition for their incredible foresight. Unfortunately, these lofty expectations are often wildly out of line and potentially detrimental to their organization. Frustration and disappointment are the typical reality in organizations adopting Agile for the first time. Understanding the root causes of this frustration will help avoid disappointment and the disruption to productivity in your project teams.
In my experience, the number one cause of team frustration and disappointment experienced by large development groups moving from a command and control Waterfall structure to Agile has been executive management’s need to feel as if they are contributing to a project by deciding priorities and dictating deadlines.
The Agile Manifesto simply states that software development teams should place more value on individuals, team interactions, working software, customer collaboration, and responding to change. In contrast, Waterfall places value on heavyweight processes, tools, plans, and documentation. Organizations looking to move to a more Agile methodology have typically been managed as command and control Waterfall shops. The command and control management style consists of executive management deciding on project priorities and then dictating deadlines. Project Management then creates a work breakdown structure or some other tool to identify all the tasks required for project completion and the Waterfall begins! Management’s comfort level with this command and control style is a major source of friction within an Agile methodology. Let’s face it, executives don’t complete any of the direct work related to a project. They don’t write requirements, write code or test the product. Therefore, the only way they feel they can contribute to a project is to use the big stick associated with the command and control style of dictating a deadline and forcing the team to perform heroics to meet it.
Software development teams are well aware of the Agile Manifesto and what it promises. They are generally excited about the opportunity to work in a more Agile environment and be freed from the confining command and control mindset. When our department embarked on the journey to convert from a very strict Waterfall methodology to Agile, there was tremendous hope, excitement, and enthusiasm. Teams were very anxious to move away from our current process which literally dictated that development could not begin until every requirement was officially approved by management. Obviously this was an ineffective and unfulfilling way to develop software since the inevitable requirement changes that were discovered could not be implemented without going through the entire requirement review and approval process. Projects were routinely late, over budget and failed to fully satisfy the needs of the business. After a few years of this inept performance there was a complete reorganization of the top levels of management and a mandate that our department look to Agile to save the day. Once the new management was in place, the Agile consultants began to appear. These consultants began coaching the organization on the techniques associated with Agile. Terms such as “Standup”, “Iterations”, “Retrospectives”, “Backlog Grooming” and “Iteration Planning” spread through the organization like wildfire. In hindsight, I could have made quite a fortune running a buzzword bingo game back then! Optimism was high and new objectives for learning and implementing Agile littered everyone’s yearly performance objectives.
The initial rollout of Agile went relatively well on a small proof of concept project. Based on these results the management team approved the use of the new Agile methodology on the complete redevelopment of our flagship product. This was to be an eighteen-month effort. The first misstep in our rollout of Agile was the demand that the Business Analysis group identify all of the Use Cases associated with this effort up front so we can estimate the entire project and calculate a deadline. Old habits die hard and it’s tough to teach an old dog new tricks would be appropriate cliché’s to describe this situation. The BA group pushed back against the idea that all Use Cases could be identified up front and stated that we thought we were going to be Agile. Nonetheless, we were forced to provide our best guess and the Project Management group came up with a hard deadline 18 months away. The team was now committed to delivering a complete rewrite of a product that was in production and being modified for seven years in a span of eighteen months! It’s ironic that our first step in the new Agile world was to complete a typical command and control task of identifying all work up front and spitting out a deadline. The team expressed the discomfort with this deadline but was assured by management that everyone understood this was a target and not a commitment. In reality, whenever a date is communicated to upper management, it becomes the expectation for delivery no matter how many times it is said to be only an estimate.
The team began development utilizing all of the shiny new Agile tools the consultants taught the group. Standups were attended and the textbook questions of “What did you do yesterday?”, “What will you do today?” and “Do you have any roadblocks?” were asked and answered. It became apparent pretty quickly that these questions were not very helpful. The team asked if we could abandon that script and simply use the time to communicate directly without the constraints of the three textbook questions. Project Management’s initial response to this request was negative. We have to follow the Agile methodology which says we must ask these three questions during a standup!
This is the second major cause of friction and frustration with an Agile rollout. Project Management begins emphasizing the Agile tools and processes over team collaboration and interpersonal interaction.
Ironically this is a completely anti-Agile approach. Over time, we were successful in convincing Project Management that the team should be empowered to decide on the structure and frequency of standup meetings. Unfortunately, this was quite a battle and caused a fair amount of friction between the development group and the PMs. It’s fair to say that the bloom was off the rose and the initial excitement and optimism associated with the Agile rollout was starting to wane.
This was the first of many battles between the development team and Project Management over the use of the Agile tools. The development team was forced to provide initial point estimates for each Use Case and Change Request that was accepted into each iteration. The Project Management team then used these point totals to calculate team velocity. Team velocity was then used to compare the supposed productivity of each team. Management was informed each week how many points each team was able to accomplish, “Team 1 completed 80 points this iteration while Team 2 completed 40 points!” This inevitably led to the question from management as to why Team 2 was half as efficient as Team 1. This is yet another source of friction and frustration with an Agile rollout. Point estimations used in calculating team velocity is simply an Agile tool which may help predict approximately when a feature may be completed by the development team. However, since old command and control habits are hard to break this tool is easy to use as a way to question the effort of a development team. Questioning effort based on an initial estimate that was provided without knowing all of the complexities related to the work leads to low morale and further disillusionment with the Agile method.
The team dutifully soldiered on with development and attended the structured Agile meetings. Standups, Iteration Planning, Backlog Grooming, Retrospectives, and Showcases were all scheduled to occur each iteration and were well attended. However, as time went on the usefulness of holding a retrospective at the end of each iteration began to wane and was questioned. Over many iterations continually answering “What worked?”, “What did not work?” and “What should we improve?” devolved into repeating the same things over and over. Additionally, holding mandatory showcases at the end of each iteration was also proved to be more of a negative than a positive experience. Each iteration did not necessarily deliver a full set of functionality for a feature, so showcasing the team’s work like clockwork caused confusion with the stakeholders. In light of these discoveries, the team pushed back against Project Management to see if the frequency of these mandatory Agile meetings could be adjusted. Similar to previous requests this was met with resistance. The PM’s argued “Agile says we must have a retrospective and showcase at the end of each iteration!” Once again the command and control mindset of dictating heavy process to the development team was rearing its ugly head. Morale on the development team sank once again and further disillusionment with the idea that we were Agile continued.
As we marched towards the non-negotiable deadline set many months ago, it became apparent that we would struggle to complete all of the work included within the Minimally Viable Solution (MVS). Business stakeholders began to realize (or started paying attention to) what was actually included in the MVS release of the product and began demanding additional features. Management bowed to this pressure and asked the teams to squeeze in as many additional requests as possible before the deadline. The development teams heroically worked 60+-hour weeks and delayed summer vacations in the final iterations to deliver the product at the deadline. This effort further skewed the team velocity reports since it showed the teams completing 100+ points per iteration. Some in management inevitably asked why they were so productive late in the project or commented how a fixed, non-negotiable deadline drives project completion. In reality, this experience further soured the development teams on our new “Agile” process since it certainly felt like the old “command and control meet the deadline or else” Waterfall process.
At the completion of the project the development teams, project management, and executive management began working towards a more agreeable Agile methodology that would be more in line with the actual Agile Manifesto. The keys to developing an Agile methodology that actually delivers the promises of greater efficiency are the following:
1). Recognize management’s comfort level with a command and control management style which dictates process and deadlines.
2). Recognize Project Management’s comfort level with dictating process and resisting team empowerment
The key to overcoming these realities is to develop trust between self-empowered development teams, project management, and executive management. Executive management must clearly define each role’s responsibility and authority within the project to ensure the teams are truly empowered to choose how they will utilize the Agile toolset. Once these roles and responsibilities are clearly understood, the development teams must earn the trust of management by consistently delivering working software while using the Agile tools. It takes time to establish trust. Allowing self-empowered teams to utilize the Agile tools according to their needs removes the friction associated with combining a command and control mindset with Agile.
I will provide a framework for communicating team responsibilities and authority levels to ensure Agile success in a future article.