Scrum
Scrum Methodology – Agile Software Methodologies
Introduction
In our last post, we discussed Agile, its origins, basic concepts, and how it fits into the organization. While we often speak of Agile generically, we have to discuss the development methodologies that share the common philosophical beliefs of Agile.
Scrum is the most prevalent of the Agile methodologies used in software development today, and rightfully so. With Scrum, development is executed via a series of short efforts called sprints. A sprint typically lasts from two to four weeks. The actual duration of each sprint is determined by the customer and the development team, but a sprint should not be too long. We’ll talk more that sprints later.
Scrum Terms and Concepts
Before we begin our discussion of Scrum, we first need to discuss some important terms and concepts associated with Scrum.
- Scrum Team – the Scrum team is generally made up of less than 10 developers, business analysts, testers, etc. but can be larger if the product necessitates it. The team is totally self-sufficient and able to execute sprints effectively. In Scrum, roles such as architect, engineer, developer, etc typically go out the window in lieu of a sense of a team effort and an “all-for-one and one-for-all” mindset.
- Scrum Master – the Scrum Master is a team member or manager whose primary responsibility is making the team as productive as possible. The Scrum master should be very well-versed in Scrum concepts and ideals and should insulate the team from outside influences and remove impediments that stand in the way of progress.
- Daily Scrum – one of the key concepts of Scrum is the daily scrum. This is a quick, stand-up meeting of the development team. The purpose of the daily scrum is to keep the team on track on a daily basis by discussing the following three things for each team member: (1) what did you do yesterday? (2) what are you going to do today, and (3) do you have any impediments? The Scrum Master leads the daily scrum and ensures that the meeting is productive and stays on track.
- Product Backlog – the Product Backlog is the all-inclusive list of every feature and component of every application that makes up the project as a whole.
- Sprint Backlog – the Sprint Backlog is created from the Product Backlog each sprint during the Sprint Planning meeting. It is made up of all the tasks to be completed within a specified sprint.
- Sprint Planning – the Sprint Planning meeting takes place before each sprint and is attended by the Scrum team members. The purpose of the Sprint Planning meeting is to define the work to be done by each person on the team for that sprint. Once this work has been defined and assigned, work items are moved from the Product Backlog to the Sprint Backlog.
- Sprint Retrospective – one of the most effective ways to ensure continuous improvement within a Scrum team is to learn from mistakes. The Sprint Retrospective meeting, which takes place and the end of a sprint allows the team members to discuss the happenings of the prior sprint and identify things that could have been done better.
- Sprint Review – the Sprint Review meeting is attended by the Scrum team and as many stakeholders as is necessary to demonstrate the new functionality and discuss its suitability to the needs of the business. The Sprint Review is kind of a show-and-tell where the team gets to present to the customer. This has always been my favorite part of a sprint! 🙂
- Product Owner – the Product Owner is the project’s primary stakeholder and is usually a member of functional management within the organization.
Scrum in Practice
Now that we know the key terms associated with Scrum, let’s talk about how the Scrum development process actually works!
The necessary members of the team meet with the customer before work ever begins and identify to whatever extent possible the overall needs of the customer. These needs are not necessarily going to be truly complete and remember that Agile methodologies love change! But we do want to at least have a good understanding of what needs to be done to satisfy the customer’s needs and ensure that the results are in alignment with these needs.
Once we have identified the needs, we build the Product Backlog. As we discussed in the previous section, the Product Backlog is the full, all-inclusive list of the features needed. The business analysts, who are Scrum team members as well then do their best to decompose those needs into quantifiable requirements. A good requirement is a requirement that is testable, so it is imperative that the requirements are developed in such a way that the developers are able to understand them and further decompose specific tasks necessary to develop the requirement in its entirety.
Once we know (to the best of our ability) the Product Backlog, we can decide roughly how many sprints are needed to develop the functionality and assess the actual duration of each sprint. Based on the complexity of the features and the dynamics of the team, we may say that a three-week sprint is optimal, but sprints are typically from two to four weeks in length. Once we identify this duration, we generally want to stick to it and not deviate.
Before we begin a sprint, the Scrum team holds a Sprint Planning meeting. In this meeting, they review the Product Backlog and each developer decides just how much he/she can take on and complete and often high-level estimates are given based on the concept of shirt sizes (S, M, L, XL, XXL, etc). Other approaches are used, but the point of the Sprint Planning meeting is to determine and vote upon just how long the team thinks each item should take to develop and test. The Sprint Planning meeting may last from an hour to an entire day, but the outcome is the movement of work items from the Product Backlog to the Sprint Backlog and assignment of specific tasks to individuals.
When the sprint begins, the Scrum team works to develop and test the exact functionality defined in the Sprint Planning meeting. During the sprint, the Daily Scrum is instrumental in keeping each member of the team and the team as a whole on track and it is imperative that each team member knows what is going on around him/her and how the team as a whole is performing toward delivery of the feature(s) or functionality at the end of the sprint. The Daily Scrum should be as short in duration as possible, and most teams prefer to literally stand up during the meeting. Though the Scrum team should have a deep sense of camaraderie and sometimes a need to get off of the subject 🙂 the Scrum Master must keep things on track and make the team stay focused on the purpose of the meeting. During the Daily Scrum, the Scrum Master goes around the room and asks every team member the following questions:
- What did you do yesterday?
- What are you going to do today?
- Are there any impediments preventing you from getting things done?
If there are valid impediments, it is the Scrum Master’s responsibility to remove those impediments through whatever means necessary. Generally, the Scrum Master should be a person with adequate authority and credibility with the customer to be able to communicate with the necessary people to resolve any impediments encountered. This is vitally important to the successful implementation of Scrum!
Remember that as part of Agile, stakeholders are actively involved as often as is possible. It is not uncommon for stakeholders to be physically involved on a daily basis. During the sprint, developers may routinely do desk reviews with a stakeholder or stakeholders to verify proper functionality during development. A desk review is an informal, impromptu discussion/demonstration of a specific function with the customer.
At the end of a sprint, the team formally presents the new functionality to the customer and receives feedback in a Sprint Review meeting. If the sprint has been properly executed and the customer has been actively involved during development, the functionality that the customer sees should bring no surprises. It is not uncommon during Sprint Review meetings for the customer to engage in in-depth discussion about the business needs and it is also fairly common for there to be a realization than maybe some key element is missing. Thus, change is introduced into the mix and this is documented by the Scrum team. This change may likely go into the Product Backlog for inclusion in a future sprint.
Finally, at the end of each sprint, the Scrum team conducts a Sprint Retrospective meeting. This meeting is attended solely by members of the Scrum team and the purpose is to give the team the chance to reflect on anything and everything that can be done to function and perform more effectively going forward.
Agile Methodology – Using Agile to Develop Better Software and Reduce Risk
Introduction
Imagine an enterprise software development project where the customer says “we are going to take a long time to get this done and we don’t expect to see any results for at least two years”. Can you imagine it? Me neither, and the truth is that it will probably never happen 🙂 So what is reality? In the real world of enterprise software development, the key for any development team is to provide maximum value to and work closely with the customer, to be able to build a culture of true ingenuity, and to be able to meet the customer’s changing needs in a way that there is minimal disruption, if any.
In the early days of software development, it was not uncommon for months to pass before any development began, and once development started, it could be months or years before any type of finished product was ready for testing. The requirements definition and gathering process was often very long, and in many cases the development team was isolated from the customer.
Once requirements were complete and development had begun, change was just not something that was easily entertained. Let’s keep in mind that concepts such as Continuous Integration and Configuration Management were unknown and use of source control repositories was not as mainstream as it is now. A change in requirements was just quite difficult to accommodate and was generally frowned upon.
The Agile Methodology
Agile was first introduced in February 2001 via the Agile Manifesto, a document created by a group of developers who met in Snowbird, Utah to discuss the principles behind a way to do lightweight software development. Since then, the Agile Methodology has grown and been widely adopted by software development teams and companies worldwide.
When we discuss Agile Methodologies, we must also mention Scrum, Lean Software Development, Kanban, Dynamic Systems Development Method (DSDM), and Extreme Programming, since these methodologies all share the same philosophy.
In a nutshell, Agile is about communication, teamwork, collaboration, adaptability, iteration, feedback, and of course, agility! The development initiative is broken down into efforts of short duration and change is not only expected, it is embraced by all stakeholders. To successfully implement Agile, an organization must embrace its concepts and philosophies at all levels.
Agile provides a framework with which teams can maintain focus on rapidly delivering working software and providing true business value, even in environments where the technical and functional assets and landscape may vary or change routinely. We can say that Agile allows development teams to provide maximum business value through the delivery of truly valuable, working software that meets the business needs. How do we know that the software truly meets the business needs? Because all of the stakeholders are involved and quality and scope verification take place in short, iterative cycles. Deviations from the true purpose of a feature or piece of functionality can be identified quickly and corrected in an agile manner.
If we go back to the Agile Manifesto, there are 4 key points that it outlines.
It favors:
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
The key principles behind these points are outlined below (read these carefully):
- Satisfy the customer through early and continuous delivery of working software
- Change is welcomed, even late in the development process
- Working software is delivered frequently, typically at intervals from two weeks to two months
- Developers work directly with functional personnel/SMEs on a daily basis
- Projects are built around motivated, capable people and they are given an environment that allows them to succeed
- Face-to-face communication is critical
- The primary measure of progress is working software
- The development pace must be sustainable
- Continuous attention to technical excellence and good design enhance agility
- Simplicity is essential
- The best architectures and designs emerge from effective, self-organizing teams
- The team routinely reflects on past performance and seeks ways to do things better
If Agile is properly implemented, with buy-in from stakeholders at all levels of the organization, productivity and competitive advantage are maximized and cost is minimized. Of course Agile is not necessarily about reducing cost, but when properly implemented and managed that is a side effect that is very nice.
Let’s discuss the key points above in greater detail.
Favor Individuals and Interactions over Processes and Tools
The greatest processes and tools in the world are worthless without the right people effectively communicating and interacting. Regardless of the size or maturity of the organization, we should start with people then decide the appropriate processes and tools to make our Agile development more effective.
Favor Working Software over Comprehensive Documentation
In the days of waterfall development, I can remember the latter stages of larger projects being consumed with the creation of mounds of documentation! I remember working with teams of technical writers as they produced both functional and technical documentation for software deliverables. With Agile, any documentation that is created is usually created while development takes place. The rapid develop/release approach facilitates concurrency among developers, business analysts, and writers, and in an Agile environment the business analysts often produce the documentation.
Regardless of the use of Agile or not, it is rare that a customer not require some type of documentation and there is nothing wrong with that. But, in an organization that is truly Agile-oriented, working software is always the primary, core deliverable.
Favor Customer Collaboration over Contract Negotiation
Let’s face it, as long as development teams provide services for customers, there will always be contractual obligations. But when we use the term “contract negotiation” we imply an us versus them mentality and this is detrimental to the Agile process! For the Agile process to be effective, we need contractual vehicles that are flexible and that are developed and written to effectively handle change.
It is not uncommon to work with a client via a Firm Fixed Price (FFP) contract. From the customer’s perspective, FFP is preferable because it transfers risk to the service provider. In this case, Agile is still a valid development methodology IF the customer understands and truly embraces Agile concepts. The difficulty sometimes comes into play when the customer insists on defining functionality up front, forces the service provider to sign a contract whose estimates are based on these initial requirements, then tries to introduce scope creep as the project progresses. I sometimes refer to this as “agile under waterfall”, but Agile is still a good fit for such an endeavor. Obviously, a FFP contract is not the preferred vehicle under which to execute Agile, but it is still attainable if all stakeholders are well-versed in and embrace Agile concepts.
Favor Responding to Change over Following a Plan
Although detailed project plans and fancy Gantt charts are impressive, they are not useful with Agile. You read that right! Agile is based on release schedules where the prescribed functionality may be defined, but it is understood that it may change. Project progress within Agile is based on burndowns. Regardless of the actual functionality delivered, progress is still made over time. The total estimate may change due to newly-identified requirements or scope changes from the customer.
Agile and Risk Management
Prior to the emergence of Agile, a large number of software development projects failed or were cancelled with little or no working functionality in place. Teams often spent months or years working on a project with nothing tangible to show for their efforts. In many cases, projects were developed and delivered only to find that they did not meet the true needs of the business! Imagine after months or years of work and possibly millions of dollars of investment to discover that your needs haven’t even been met!
From the Project Management Institute’s (PMI) standpoint, Risk Management is a key knowledge area and something that is very high on the Project Manager’s priority list. All project managers should understand risk. It is just an inherent dynamic within any project and one that has to be understood, and either avoided or mitigated. So, what is risk? By its formal definition, risk is something that can or may occur and that could cause unexpected or unanticipated outcomes. Project managers know that risk is not always something negative. Opportunities are risks as well. But risk is something that, positive or negative, has to be identified, quantified, and managed. The situation, environment, project, people, etc determine when, where, and how risks are managed.
Agile reduces risk through through stakeholder involvement and rapid, iterative development and release. This means that evaluation of scope verification take place routinely, which effectively reduces risk.
Organizational Threats to Agile
The greatest single threat to Agile is management! More specifically, functional management with unrealistic expectations. In some organizations, Agile is nothing more than a buzzword because the stakeholders have not been educated in its fundamental concepts.
Earlier in this post, I mentioned the need for Agile to be understood and embraced by every stakeholder, from the top down. Without this understanding and support, it will likely fail or at the very least leave managers with a bad taste due to the fact that the development Project Manager tells them “we can certainly modify our approach and give you functionality X but requirement W is going to have to be pushed back to a future iteration.” In the case of FFP, requirement W may just have to drop off entirely!
With Agile, change is welcomed, even late in the development process, but in the case of FFP, it is possible that certain changes can significantly affect the project end date and thus necessitate contract extension.
Conclusion
So, Agile is a software development methodology that fosters rapid delivery of valuable, working software in an iterative manner. It values people and communication over processes and tools. It prefers working software over comprehensive documentation. It favors active and dynamic involvement of the customer and proper, effective identification of the true needs of the business over contract negotiation. It advocates the ability to nimbly respond to change, even late in the development process to following a detailed, pre-defined plan.
It can be argued whether or not it negates the need to perform Risk Management, but it is safe to say that with constant and active involvement of the customer and self-organizing, professional, competent, and productive development teams with a true dedication to the customer’s mission and a clear understanding of the customer’s needs, it can be enormously successful and a win-win for both the customer and the development team.
I will reiterate this final point because it is so hugely important – Agile MUST be understood and embraced by every member of the organization, from the CEO to the day-to-day tactical and functional personnel. Without this complete organizational buy-in and effective communication and interaction between all involved stakeholders, it will not be successful.
In the next post, we will discuss Scrum in detail.
John Nelson's Blog 