“Ambition is a very pretty thing; but sir, we must walk before we run.” 1851 G. Borrow Lavengro II. ii
This truism applies to many areas of life and is highly appropriate for large organizations when attempting to adopt agile within a traditional software development environment (aka waterfall). This article explores what transpired when a large retail grocer invited a team of agile experts from a service vendor to partner with them to develop an enterprise agile transformation program that would fit their existing corporate culture. The program would pilot a newly devised agile delivery framework that would provide pragmatic, realistic measurements that will evaluate the effectiveness of the agile practices on the company’s software development projects.
The retail grocer, headquartered in the Midwest (USA), is one of the nation’s largest grocery retailers, with fiscal 2011 sales of USD90.4 billion; spanning many states with grocery and multi-department stores, convenience stores and mall jewelry stores.
Software development is a critical part of grocer’s business model today and in many regards has become viewed as a competitive advantage for the company. Over a four year period, the grocer embarked on a strategy to improve its’ software processes, to move away from a siloed-tiered IT organization to a service-tiered organization where infrastructure and enterprise wide IT resources were grouped into a set of shared services.
In addition, the application development function was further re-organized by functional area (grouping Project Management, Business Analysts, Enterprise Architecture, Software Development teams, etc. into discrete teams) where line managers were replaced by resource managers responsible for staffing and employee evaluation. Finally, all capital projects were required to adhere to more rigorous project controls. Unfortunately, these changes failed to achieve the desired results at the grocer. After a four year journey, teams were struggling with collaboration, product quality had not improved, and project delivery stood at 59% and project timeframes continued to average 30 to 41 months.
Admittedly, the company’s waterfall development approach worked well for projects that had a well understood timelines, domain space and technology, but often failed to meet business expectations in newer, innovative areas. In general, the grocer’s application development teams struggled to deliver value quickly, and most projects ended up costing more than expected and consumed unplanned resources.
The grocer’s journey of agile adoption started long before they ever engaged a service vendor for assistance. Experimenting with agile, several IT resources obtained Scrum Master Certifications and started piloting agile practices but ran into a variety of implementation difficulties which resulted in the grocer’s leadership defaulting back to their traditional development approaches as the “least risky option.” They cited several failed attempts to adopt agile practices in the past due to a lack of:
- appropriate transparency and visibility within the agile projects
- inability of agile teams to “stick to a plan”
- any meaningful measures/evidence showing that agile projects out performed waterfall projects
Regardless of these initial results, some members of the software development community felt confident that agile practices could indeed be successful at the grocer given the right level of planning and modification based on the company’s unique requirements. Knowing that a large multi-national service vendor had itself been undergoing an agile transformation initiative, the grocer’s delivery effectiveness team sought the counsel of the multi-national vendor services organization for assistance with their agile adoption.
Working with the grocer, the service vendor evaluated their software development practices to identify scaling factors that could be preventing the successful introduction of agile practices. They determined that practices such as Scrum and XP could be effective on agile projects, but would require significant modification. The service vendor’s Agile Scaling Model was used as a reference point to determine how to effectively adapt and scale classic agile practices. During an assessment of the grocer’s environment, the service vendor’s team discovered that the following scaling factors affected most of the large software development projects within the company:
- The development teams were relatively small but geographically distributed across various locations;
- The retail grocer had specific project / governance disciplines that must be followed;
- Project environments required the support of many operating platforms, including legacy environments;
- The product domain was innovative and rapidly changing;
- The retail grocer’s organization was highly distributed, involving multiple internal organizations and independent third parties; and
- The retail grocer’s organizational structure was quite rigid and steeped in existing procedures and practices
Enterprise transformations are difficult at best in large enterprises such as the grocer, and they typically require a gradual and phased adoption strategy that permeates the organization over a period of time. Armed with this organizational insight, the grocer realized that they would not be successful with a haphazard approach to their agile adoption. Instead, the grocer put in place a deliberate and strategic agile adoption plan based on a hybrid delivery model that applied an iterative and incremental approach to agile transformation based on a common set of guiding principles:
- Maintain a focus on business value / results
- Drive the vision for adoption from the top-down and implement from the bottom up
- Adopt agile practices iteratively in waves
- Adapt quickly and embrace new practices as projects and teams mature
- Strive for quick-wins and modify accordingly based on lessons learned (retrospectives)
The grocer’s delivery effectiveness team divided their multi-year agile transformation program into multiple waves, each lasting approximately 9-12 months. Each wave of their agile evolution would require commitment and action at multiple levels within the organization.
During Wave 1, the grocer’s delivery effectiveness team set several goals:
1) Assess the organization to identify the current state (current software practices, scaling factors and organizational challenges that could impede their agile adoption),
2) Define an initial agile framework to pilot in conjunction with their traditional process
3) Create a set of criteria, or decision matrix, to identify agile-friendly projects, and
4) Conduct an agile pilot using a “real-world” capital project to measure the effectiveness of the agile practices and the grocer’s adoption strategy.
In subsequent waves, it was planned that the service vendor and the grocer would use feedback from the Wave 1 results to update and adjust the grocer’s agile adoption strategy, agile framework, and decision matrix, as necessary, and would conduct additional pilots to expand the program in different areas of the both the business and IT. Given the organization’s previous issues with Agile, and more specifically, Scrum, and given the complexities inherent in the grocer’s software development projects, the grocer decided to implement a Disciplined Agile Delivery Framework and apply the service vendor’s Agile Scaling Model, two methodologies that were created and espoused by Scott Ambler, chief agile methodologist for IBM Rational®. Disciplined Agile Delivery (DAD) is defined as:
… an evolutionary (iterative and incremental) approach which regularly produces high quality solutions in a cost effective and timely manner via a risk and value driven life cycle. It is performed in a highly collaborative, disciplined, and self-organizing manner within an appropriate governance framework, with active stakeholder participation to ensure that the team understands and addresses the changing needs of its stakeholders to maximize business value provided. Disciplined agile delivery teams provide repeatable results by adopting just the right amount of ceremony for the situation which they face.
The Agile Scaling Model (ASM) is a contextual framework for effective adoption and tailoring of agile practices to meet the unique challenges faced by a software or system delivery team of any size. There are a several scaling factors that address a wide variety of agile bottlenecks, including but not limited to, geographic distribution, regulatory compliance, technical complexity, and others.
Using these scaling factors as a guideline, the service vendor conducted an assessment where the following scaling factors were identified: geographic distribution, enterprise discipline, problem domain, project complexity, and organizational structure. Based on the existence of these scaling factors, the grocer’s delivery effectiveness team implemented the following agile practices:
- Risk Value Life-cycle
- Release planning
- Whole Team
- Shared Vision
- Business Value Focus
Scrum is an agile project management framework which works in conjunction with other agile practices to provide teams with the following capabilities: time-boxed iterations, product backlogs, user stories, sprint plans, daily meetings, sprint demos, and sprint retrospectives, so teams can build and deliver small chunks (iterations) of “potentially shippable code”. DAD expands the Scrum life-cycle by incorporate the entire software delivery phase, not just focusing on the software development organization. DAD acknowledges that the development team is dependent on other critical enabling organizations (verification testing, integration testing, IT operations, etc) in order to get its software into the hands of customers.
While the grocer’s first wave of agile adoption was deemed as a success, it wasn’t without its challenges. Moving to a Disciplined Agile Delivery framework was a big adjustment for the pilot team. Culture wars ensued where some team members wanted to continue with “business as usual” (i.e. maintaining traditional practices) while other team members wanted the flexibility to use Scrum techniques “out of the box” with minimal coaching. Change is painful, and all of these issues had to be overcome gradually.
Like any large organization, the grocer continues to struggle with ongoing agile adoption issues, including:
- Organization Structure: the grocer has a three-tiered service model which has constrained their ability to create “cross-functional” agile teams due to the strong established roles. The company is still working through organizational changes to facilitate an agile process.
- Project Funding Model: The front and back ends of the grocer’s software development life-cycle still practices Waterfall processes, which limits the ability to do rapid software development iterations. For example, approval of projects occurs twice a year, and projects must come to within +/-5% of total funding to gain approval which requires that ‘all’ of the requirements and design activities are completed. The grocer recognizes the need for a more flexible funding approach to fit with a development model of more frequent development iterations.
- Internal Governance: The grocer’s Project Management Office (PMO) was not initially involved in the company’s Agile transformation and resisted it, but there is a recognized need for the PMO to be more engaged as a facilitator and enabler of an Agile process across the software delivery life-cycle.
- External Governance: The grocer is required to conduct end-to-end testing of their solution and submit it to a third party for a two week validation period before the product changes could go into production. This governance must be incorporated into the overall Agile process to get software quickly into the hands of internal and external customers.
- Team Dynamics: During the iteration planning meetings, the team struggled with understanding the value of estimating using story points and iterations with the extended team (people who are not part of agile pilot [core] development team). Secondarily, the story points needed to be converted to an hours based system where a final budget can be calculated for the project. Success with Disciplined Agile Delivery will mean the incorporation of an expanded group of stakeholders who are responsible for overall software delivery.
- Estimating Effort: The grocer is refining its Agile practices in the areas of estimation, including grooming the product backlog, conducting sprint planning as story points, and using Agile measurements such as velocity and release burn-down charts, product owner acceptance, total test cases planning and executed, and total number of defects.
- Use of Agile Planning Tools: The pilot team expressed frustration that there were “too many tools” involved in getting their job done. The tools the team did have access to were not being used effectively. The service vendor and the grocer have piloted the use of an enterprise agile project management tool to help manage the product backlog and improve team collaboration. The grocer pilot team team continues to struggle with what is the right set of enabling tools to manage their internal agile software development activities as well as those with an external third party vendor.
The grocer’s first wave of agile adoption achieved several benefits as a result of an enterprise adoption framework and the incorporation of disciplined agile practices.
- The increased collaboration between the Product Owner and IT led to higher business satisfaction, greater levels of organization trust, and better overall planning.
- The project was able to consistently deliver business value more rapidly than when the team was using waterfall – with at least a 4 times increase in the frequency of development releases compared to a traditional [waterfall] process.
- The project team delivered software that was closers to the product owner’s satisfaction and released higher quality products while receiving more frequent feedback from their key stakeholders.
The grocer’s story is a classic example of a company that was steeped in traditional [waterfall] software development processes that is now achieving incremental success and tangible results by incrementally adopting agile practices. Despite the real and ongoing technical and cultural challenges, the pilot reported measurable, positive outcomes in the delivery of working software which will have a measurable impact in driving additional revenue. In the grocer’s case, the organization simply had too many complexities to adopt Scrum in its purest form. By adapting agile practices in line with the grocer’ scaling factors (organizational complexity, governance, functional roles), the grocer was able to attain significant improvements in product quality, speed of delivery, and increased customer satisfaction that were unattainable in their previous attempts at agile adoption. For the grocer, a Disciplined Agile approach was the only path towards success.
“Enterprise Agile Adoption is a marathon, not a sprint.” Gerald Smith, CSP, PMP