Agile
Agile Methodology – Testing within an Agile Environment
In the days of waterfall, software testers created test scripts from the mounds of requirements and rigidly and methodically executed those test scripts against the functionality that the development team delivered. The bug remediation process was often a sizable endeavor that took place in a large way as we approached milestones and immediately before we had to deliver software. In many cases, development stopped, and developers sat around and waited for testing results for hours or sometimes days. The testers and developers often did not work together as a cohesive group and in the case of larger projects with dozens of team members, the communication between testers and developers was routinely done late in the process and was not very effective.
With Agile, testers and developers not only work closely together during development, they communicate openly and can foster a great deal of synergy. The “us versus them” mindset that often crept into waterfall development is gone and the two roles work together to develop and deliver the best products possible. We no longer “throw the software over the wall” for the QA group to pick apart, but we utilize our own individual roles, skill sets, understanding of the problem domain, and unique insights into not just the product we are developing but the problems and processes that our customer lives with everyday.
Continuous Testing and Perpetual Quality
Testing in an Agile environment is continuous. Developers test rigorously as they develop, and often turn to the testers and business analysts to provide functional expertise. In other words, though testing is done continuously, the routine interaction between developers and testers often puts the testers in an advisory role as well. Since the development team often publishes or deploys new functionality into the development/test environment(s) on nearly a daily basis, verification and feedback from the testers is immediate.
Testing and Quality Assurance under Agile is not a phase at the end of a development effort. Rather, it is seamlessly woven into the process and is an integral part of the total endeavor. There is no separation between developing and testing – testing is just an ongoing activity that ensures the validity and correctness of the functionality that is being developed.
Agile Testers are Involved!
In an Agile environment, testers are actively involved in discussions with the customer and the creation of user stories. Along with the business analysts, they develop an intimate knowledge of the true needs of the customer and are directly involved in the software design strategy from the standpoint of testing. Beyond this, they actively participate in the segregation and decomposition of user stories and the estimation of the overall development and testing efforts for each.
In contrast to the days of waterfall where testers were often viewed as bottlenecks and impediments, with Agile they actively propel the team and the effort forward with their insights, understanding of the customer’s needs, and they do all of this with a keen eye on not just the act of testing, but the testability of the functionality created.
Agile Testers Become Subject Matter Experts (SMEs)
As we stated in the previous section, with Agile, testers are actively involved in discussions with the customers. This includes senior management, functional management, and the people who are in the middle of the day-to-day operations of the business. The testing practice within an Agile environment places the testers in an advisory role in a way that waterfall never could have. This means that part of the normal, continuous interaction between developers and testers includes testers providing subject matter expertise in many situations. Therefore, testers should take it upon themselves to be SMEs. In fact, the testers and the business analysts should collectively possess nearly the entire body of knowledge of the functional personnel with whom they gleaned the user stories that drive development. That sounds like a tall order but it is necessary to ensure the effective delivery of acceptable products. It allows the Agile development team to be as self-supporting as possible while providing a basis from which software that truly meets the needs of the customer can be developed.
What Makes a Good User Story?
A good user story is complete, accurate, detailed, and from the standpoint of QA, it is testable! Though this may sound nonsensical because after all, don’t we already know that? We should know that, but it is important to explicitly make the statement. A good user story is testable.
Must Haves – Continuous Integration and Automated Testing
Continuous Integration is a concept that calls for every developer to continuously integrate his/her changes into the master code base. This obviously necessitates a suitable source control repository such as Microsoft Team Foundation Server (TFS), and there are many others that serve the purpose very well. Regardless of the chosen source control tool, the process dictates that every change be compiled/built against the latest code base prior to any changes being checked in. Each developer must adhere to this, and it’s pretty simple.
Beyond that, automated builds should be implemented. The build server(s) should be configured to get the latest version of the code base and compile/build the code periodically throughout the day or at least based on a pre-defined schedule. Once the latest code has been retrieved, the build server(s) should then execute any prescribed automated tests. If the code coverage is high enough, problems that would go unnoticed for days or weeks can be uncovered very quickly to facilitate rapid remediation. The topic of code coverage is pretty important because the purpose of automated tests is to holistically ensure the integrity of the code base at all levels. This cannot be done unless the code coverage is as close to 100% as realistically possible.
As the software we build becomes more complex and we have multiple layers that are ideally loosely-coupled, the need for automated unit testing becomes paramount. With unit testing, individual tests have extremely narrow focus. As we move up the layer stack, we don’t deviate from this but we understand that when we reach the business logic layer (BLL), we may have dozens of business services with potentially hundreds or thousands of methods, each of which relies upon some lower-level object or objects. If we build our unit tests from the bottom up, consider all the possible scenarios and outcomes, and we orchestrate our automated tests to exercise all layers of our code from the bottom up, our code coverage approaches 100%. The daily or periodic execution of these batteries of tests ensures that the integrity of our application’s layers remains in tact as we move through development. Remember, Continuous Integration requires us to always know that our code base is sound.
Automated testing tools and cohesive automated testing methodologies are critical to the delivery of quality software under Agile. Developers must have a sense of ownership in every piece of functionality they develop. Unit testing is vital to their development process and code coverage should be very high. I don’t want to go too deeply into the specifics of automated testing in this post, but it is important to recognize its importance with regard to Agile.
Finally…
In summary, within an Agile environment, testing is not a phase at the end of the development initiative. Instead, it is an integral part of the overall development effort. It is executed daily by developers who not only write unit tests for their code, but who diligently test all functionality they build. Testing and QA in general is sewn into the fabric of the overall initiative and testers work directly with the developers on a daily basis, not just specifically as QA providers, but as SMEs and advisers. Testers in the Agile environment work directly with the customer and serve a very important role in combination with the business analysts. With Agile, our overall view of testing changes dramatically from the days of waterfall and we live by the concept of perpetual quality.
Agile Methodology – Agile under a Firm Fixed-Price Contract (FFP)
I
n our first post on using Agile to develop better software and reduce risk, we discussed Agile in a general sense, talked about its origins, and covered the basic underlying principles.
Project Managers and development teams in today’s business world are becoming more and more familiar with Firm Fixed-Price contracts (FFP). They struggle to adhere to the philosophies of Agile while being confronted with customers and functional managers who want specific features and functionality and want accurate estimates up-front. Many of these customers already have (or believe they have) knowledge of Agile, but want to capitalize on the risk transference that FFP gives them. Having worked in such environments for several years, I have learned just how interesting these situations can be for Project Managers and development teams, and how the development path can be affected as a result.
A large part of the solution is to think outside of the box a little and consider Agile’s strong points and how they actually help us contend with perceived challenges of FFP contracts. Before we can consider ways to make Agile work under FFP, we first have to understand exactly what Agile is, understand the types of constraints inherent to any development project, and gain some insight into why organizations favor Firm Fixed-Price contracts.
An Agile development team can definitely succeed under FFP! Let’s get that out of the way. As we discuss project constraints and the motivations of customers and contractors/service providers we will see just how to make it work.
Why Does Agile Work in the First Place?
Ignoring the concept of FFP for a moment, why would an organization benefit from Agile in the first place? Let’s start by reviewing the 4 key points of the Agile Manifesto and discussing each a little bit. We originally discussed these in our first Agile post, but let’s take a slightly different approach here. Agile favors:
- Individuals and interactions over processes and tools – people define functionality and people build software. People decide on and implement processes and people choose and implement tools. Pretty simple, right? By placing emphasis on the people and their interaction we also place emphasis on productivity, quality assurance, accountability, and knowledge transfer not just at project completion but throughout the development process.
- Working software over comprehensive documentation – although documentation is important, a working product is more important. When we say “comprehensive documentation” what does that mean? In the days of waterfall, it meant massive mounds of functional and technical documentation and quite often the technical documentation was not too far from a regurgitation of the code itself! What good is that? Though I avidly support the provision of adequate functional documentation to support the normal operation of the software and to provide training/reference information (based on the customer’s needs and specifications of course), Agile forces us to take a look at exactly what we are producing, how we are producing it, and its true value, and we produce what is needed concurrently with development. From the standpoint of technical documentation as an example, if we write our code properly and provide the documentation and commentary throughout to adequately allow a developer who comes behind us to understand what is being done, there is just no need to reproduce this information again.
- Customer collaboration over contract negotiation – regardless of whether or not a customer demands an FFP contract, it is highly unusual to NOT want a reasonably accurate estimate of the total cost of the product development and all services to be provided. That’s just good business! It would be irresponsible of a service provider to not exercise due diligence in providing such an estimate anyway. That is good business for the service provider. But with Agile, the “us versus them” environment that can be created with contract negotiation is counterproductive. Through close collaboration, effective communication, and expectation management, a service provider can do a pretty good job of ensuring that the functionality that is truly needed is properly developed and that the customer is completely up-to-date on all functionality developed. Furthermore, the service provider can ease the customer’s fears and apprehension by building trust through day-to-day communication and frequent impromptu reviews and demonstrations. As they see functionality being produced, they tend to relax a bit and allow themselves to become even more involved.
- Responding to change over following a plan – with Agile, change is expected. When we say change, we do not mean chaos! Uncontrolled, frequent change is not feasible and though change is welcomed, it has to be handled in an orderly manner.
Now, from our original post on Agile, let’s also review the principles behind the points discussed above.
- Satisfy the customer through early and continuous delivery of working software
- Change is welcomed, even late in the development process
- Working software is delivered frequently, typically at intervals from two weeks to two months
- Developers work directly with functional personnel/SMEs on a daily basis
- Projects are built around motivated, capable people and they are given an environment that allows them to succeed
- Face-to-face communication is critical
- The primary measure of progress is working software
- The development pace must be sustainable
- Continuous attention to technical excellence and good design enhance agility
- Simplicity is essential
- The best architectures and designs emerge from effective, self-organizing teams
- The team routinely reflects on past performance and seeks ways to do things better
Project Constraints – Time, Cost, and Scope
Every project has the following constraints: time, cost, and scope. When a customer drafts a Firm Fixed-Price contract, they generally attempt to lock all three. How do they do this? It’s pretty simple:
- Scope is locked by the production of a comprehensive requirements specification. The customer does this in an attempt to ensure that no requirement is missed and to give the service provider a detailed view of who they are and what they expect their software to be and to do.
- Cost is locked because in their minds the requirements are detailed, explicit, and comprehensive so it should be easier for the provider to give a very accurate estimate up front. Furthermore, by creating the requirements specification, the customer generally also uses their business knowledge to formulate their own idea of how much it should cost to build the software.
- Time is locked directly and indirectly from the first two constraints. Quite often, customers define their expected period of performance (POP) based on their estimate(s) from scope and cost.
Project Managers and developers understand this triangle of constraints and how the adjustment of one can affect the values of the others. So let’s start by looking at Agile under FFP from this standpoint.
Exploring the Customer’s Motivation for a Firm Fixed-Price Contract
Let’s think for a moment about the prospect of building a house. It’s an exciting topic to discuss, and just the thought of it gets my creative juices flowing! But as quickly as my creativity springs into action, my risk management facilities do too. My first thoughts deal with ensuring I hire a reputable, ethical, and licensed contractor. Obviously this isn’t something that I want to entrust to someone who will screw me in the end or whose work is shoddy. I am going to want to do my homework, call around and get references, look closely at past performance, consider quality of work, and many other things. Who wouldn’t do that?
I am also going to set myself a budget and communicate that budget to my contractor. I am then going to work with him to determine whether my desired plan, square footage, materials, landscaping, etc. are going to allow me to com in at or below my budget. I am going to want him to sign a contract that sets the maximum cost to me. Oh, wow! Guess what I just did? I hired a contractor under a Firm Fixed-Price contract! How dare me to do something like that! 😉
See, it’s not a bad idea, but just hold this thought and we will revisit it later in the post. Well, my motivation for wanting to ensure that the contractor I hire can deliver the product I want at or below my pre-defined cost limit is very logical and sensible. After all, I am not made of money and have a limited budget! That’s perfectly fine.
Before we begin discussing ways to execute Agile under FFP, we first have to take a look at the reasons why customers are preferring FFP contracts more and more these days. At a high level, the reasons are:
- Internal politics and forces within the organization
- Lack of trust in the potential contractor or service provider
- Lack of understanding of development processes and methodologies
- Belief that they know the true scope of the project better than anybody
So let’s continue our discussion of Agile under a Firm Fixed-Price contract with this in mind. The desire to lock a contractor into a set cost is not evil. From the customer’s standpoint, there is a limited budget, the consideration of ROI, political influences (both internal and external), trends in the business’ marketplace that are necessitating the software, and the list goes on and on. These are all very sensible and real considerations.
Internal Politics and Organizational Forces
Though we like to think of companies, departments, and teams within departments as cohesive, cooperative, supporting groups of people, in reality this more often than not is far from the truth. Whenever groups of people work “together” on a daily basis, there are dynamics that come into play that influence people’s behavior and decision-making. There is very often competition among managers, distrust, and empire building at work. Egos play a very large role in shaping the true function and nature of companies, departments, and teams. That is just the way it is. One fantastic way to tarnish your reputation as a manager is to be held accountable for a botched or failed development project.
We won’t dive too deeply into this discussion in this post, but I do plan to cover it in more detail in a later post because there are indeed forces at work within companies that may not be readily apparent on the surface but that become evident as we become more intimate with the organization and learn the dynamics and relationships among stakeholders. We should be mindful of that before we ever meet with a potential customer!
Lack of Trust in the Potential Contractor or Service Provider
Let’s face it, there will initially be a level of distrust of the contractor or service provider. That’s just human nature. If we go back to our example of building our house, we can immediately relate. Because of this, organizations feel compelled to exercise due diligence in fully understanding their own requirements before they ever seek a contractor to build a product. In the process, they uncover additional requirements, often debate the need for these requirements, and over the course of days, weeks, or months spend a great deal of time and money just understanding their own processes and needs. The end result of this effort is usually a comprehensive requirements specification.
Armed with this, they can, with a clear conscience seek contractors and bids because they now fully understand what they need and will be able to offer a unified front to all potential bidders that shows understanding of their needs and knowledgeable SMEs who can interface with these potential bidders. All in all, it sounds like a pretty good idea from the customer’s perspective.
Lack of Understanding of Development Processes and Methodologies
Although most organizations have good intentions in drafting requirements specifications for an FFP contract, the truth is that many of them just don’t truly understand how software development processes work and how the discovery, design, development, delivery, and quality assurance activities should really be carried out. When the comprehensive requirements specifications are created and finalized, they often contain all of the features and functionality that the customer believes to be the most important and the most crucial to the value of the software to be developed. They believe that by being as detailed as possible, the need for change later in the process is eliminated and therefore this document provides the true blueprint for what they need. Because of the belief that change will not be needed, they believe also that the provision of an estimate from the contractor should be easy and straightforward.
Belief that They Know the True Scope of the Project Better than Anybody Else (We Know What We Want!)
Functional managers and personnel have in-depth knowledge of the business processes and tasks that are required to carry out the day-to-day operations of the activities within their purview. This is invaluable to the development team during discovery and development. When it comes to creating requirements specifications though, this business process knowledge combined with lack of understanding of how development processes work can cause the customer to focus on defining scope based on their perception of what is needed.
Building our House – An Evolutionary and Agile Process?
As I promised, we are now going to return to our example of building a house 🙂 So, I am now back in the driver’s seat and I am contacting potential contractors and discussing my wants and needs with them. Let’s assume that I find a contractor and when I meet with him I show him the blueprints that I have already paid an architect to create, and I give him an exact list of materials for everything within the house. I have already picked out external siding, roofing material, tile for the kitchen, bathroom, carpet for the living room, den, and bedrooms. In summary, I am a great customer and I just know what I want!
Based on my requirements, I have determined that the job should take 16 weeks and my budget is exactly $225,000. Great! I know where I stand and what I want! So I hire the contractor, he gets started, and is doing a great job! Then, as construction progresses and I see what the kitchen layout really looks like and how the tile I selected goes with the cabinetry, I realize that I don’t really like it! So I tell the contractor that he needs to stop laying that tile, and that I want to change the design. I also realize that I don’t really like the cabinets and want them taken out. He tells me that he can do that, but it will cost me $1,000 in labor to pull up the tile that is already down, and since they are several days into the tile laying, probably another $3,000 to put the new tile down, not to mention that I have to return what is left of the original tile to the tile store and buy the new tile. The cabinets have to be taken down, repackaged, and returned to the store and I now have to purchase new cabinets. He tells me that it will take a couple of days to remove the old cabinets and repackage them at an additional cost of about $1,000. This epiphany of mine could cost me an additional $15,000 and blow my schedule by more than a week!
Because I built my requirements specification based on what I honestly thought I wanted but didn’t account for added cost of changing my mind, I have now begun to blow my budget. The price he quoted me was based on my exact specifications, and we both agreed that those reflected what I wanted. Let’s not forget that my time estimate of 16 weeks is now blown as well.
So now let’s rewind to the beginning of my project and start over…
Imagine now that I hire a contractor who understands Agile principles. That’s right! Agile can be applied to processes other than software development 🙂 He reviews my specifications in detail, and when we meet the next time he tells me that he acknowledges that my budget is $225,000, and that I really want to see the house completed in about 16 weeks. But then we discuss my detailed requirements, and though he definitely believes that based on the blueprint he can build the house within my budget, he asks if I would be in favor of taking an Agile approach to ensure that I get the best value and that I become involved in the construction process on a routine basis. He tells me that we will take my general requirements as a starting point, but that we will manage the construction process in such a way that we identify the absolute must-haves first, then give ourselves the ability to adjust and make changes as we progress.
The more I think about it, the more I like this approach. If we get to a point where we realize that one of my requirements turns out to not be the best choice, we can make a change. After all, I did my homework and talked with several of his prior customers and all of them raved about his work and his methodology. At this point, I am feeling really good about things!
So we agree on the basic floor plan and we both sign the contract and he begins work. The foundation is poured, plumbing set in place, and the walls begin going up. On a daily basis, I go out to the site in the afternoon and look at what has been done that day and he and I discuss what he plans for the next day. Occasionally, I see things about which I am not sure, and he and I discuss the options. As they begin framing the internal walls, I have a chance to see the house everyday, and I am able to visualize the finished product and because I see things progress on a routine basis, there is always opportunity to make slight modifications without hugely affecting the timeline. Obviously, the contractor guides me as I decide upon changes and no change or decision for change is made in a vacuum.
The important point here is that he and I agreed to the cost and the time. Those just should not change. What we left slightly fluid is the scope. Now this may sound as if we just left this constraint undecided, but that would not be a true statement. We did agree on what the finished product should be, but because we are using an Agile approach, there is flexibility to make slight modifications as construction progresses. Early in the process, I communicated to him the things that are non-negotiable, the features and options that I just have to have. But because I am actively involved, at times I may realize that some of my must haves either really don’t make sense or that maybe I just didn’t think something through adequately.
This approach actually maximizes my value and allows the contractor and his crew to work at a somewhat pre-planned pace. The scope is somewhat flexible, but the time and budget/cost are fixed. We can still say that this is a firm fixed-price contract because it is. But we have implemented an Agile approach to allow us to get the most out of the things that truly matter, and because I am actively involved, I can set the focus on a daily basis if needed. It’s a win win for everybody! And even if there is some need to slightly extend the 16-week time constraint, I am more focused on the cost, so this isn’t a big deal.
Initial Discussions with the Customer – Fix the Time and Cost, Not the Scope
If we take our home building example and put ourselves in the contractor’s place, we are now faced with executing Agile under a firm fixed-price contract.
To give the customer the most bang for the buck, we should be willing to spend some time with them truly understanding their needs. Even though they have developed their comprehensive requirements specifications based on what they know, we may have an opportunity to present the value of Agile to them and attempt to convince the to work with us to define the true must haves, then allow the Agile process to let us give them the best value possible, with their continuous involvement.
The challenge here is to convince them that we can fix the time and cost, but we should approach the scope from an Agile perspective. This job requires salesmanship and confidence. Of course, past performance is never a bad thing and if we can provide them with satisfied customers from past projects who are willing to give us a great reference, we may harness enough leverage to sway them in our direction. If we can successfully convince them that their requirements are indeed valid but that we should rely upon user stories to help us use our professional business analysis and process engineering skills to guide them to the optimal solution, we have gained valuable credibility up front!
This is sometimes a delicate process and unless we have the right people to present the optimal approach (while not deviating from the fixed time and cost constraints) this may not be a feasible endeavor for us. I have known service providers to walk away from potential engagements because they realized that the customer was too set in their ways or that the customer refused to alter their stance.
Now, don’t misunderstand me. Walking away should be a last resort. We should first assess every RFP/RFQ that comes our way, and objectively evaluate the potential customer and project. If we go after projects that are either not a good fit for us or that we believe to be setting us up for failure from the beginning, we should opt to not pursue.
Again, I do not advocate just walking away from potential business. I have, however been doing this long enough to know that you should never set yourself up for failure. If we believe the customer’s requirements to be unreasonable and there is no sign of a change of heart on their side, we can never ignore the fact that not pursuing the project is always an option!
What if we Must Fix all Three Constraints?
So what happens if we pursue a potential customer, and are unsuccessful in convincing them to allow us to apply Agile in such a way that the scope is not necessarily set before the project begins, but we know the project is something that we can successfully develop and deliver?
Well then we can still use an “Agilistic” approach within our development model, involve the customer in the way that we would normally, but we have to make them understand that change will be something that is going to happen, due to the nature of development projects. As a result, they have to accept the risk that some requirements that have been pre-defined in the requirements specification may drop off completely if change does indeed come into play during development and the change affects the critical path with no option for extension.
I have seen this before. We could for example use Scrum and build our Product Backlog from the customer’s requirements specification. The main point of consternation for the Agile team though is the fact that we aren’t truly doing Agile, but a watered down version of it. Though we can say “we use Scrum”, the truth is that our endpoint is immovable. If we have to push tasks back into another sprint/iteration, those tasks still have to get done at some point, and barring the need for contract extension due to introduction of change by the customer, our development team may find themselves working overtime. Since this is an FFP contract, we are not going to be additionally compensated, so we just have to “suck it up” and get it done! That’s the painful truth and it happens all the time.
Promote the Agile Mindset Anyway!
But guess what? All is not lost here! As we progress through the development effort, we include the customer and enlist their active involvement. As the team conducts sprint planning meetings, we can invite stakeholders to observe. We can even invite them to our Daily Scrum meetings to observe the process and learn. We certainly sell the benefits of Agile in the Sprint Review meetings where we present completed functionality to them. A key point here is that by involving them continuously, conducting desk reviews routinely, and doing mini-demos throughout each sprint, nothing they see in the Sprint Review meeting will be a surprise! Wow, they already know what they are going to see. This minimizes the chance of change anyway, and if an issue is spotted during a sprint and change is truly warranted, every stakeholder is involved immediately and a decision can be made and documented.
Why is anything in the previous paragraph important? Because of one golden point: though we may have been locked into time, cost, and scope for this project, if we can educate the customer about the true benefits of Agile on a daily basis, we stand a better chance of swaying their thinking for the future!
But Very Few Organizations Fully Use Agile anyway! What’s the Big Deal?
The truth is that in reality, most teams don’t truly implement “pure” Agile anyway. I’m sure that we all know that, but it has to be said. Whose fault is this? That’s a little complicated, and it is not my intent to cast blame. I do however think there is a need to have an open dialog about this subject.
If you are of a pure Agilist mindset, this is a source of pain! If you are someone who believes the end justifies the means, then you don’t really lose any sleep over the discussion. I am a mix of the two. I truly want to be able to enjoy the full benefits of a purely Agile environment, but at the end of the day I also know that due to the nature of what we do, there will be times when we just can’t get what we want.
It is ultimately the contractor’s responsibility to effectively sell the methodology to each and every customer. As they do this and successfully deliver quality software and provide great value for the customer, they gain credibility in the form of past performance. But they have to start somewhere!
I think ultimately, it all goes back to fixed budgets, inherent distrust in potential contractors, and organizational dynamics.
Scrum Methodology – Agile Software Methodologies
Introduction
In our last post, we discussed Agile, its origins, basic concepts, and how it fits into the organization. While we often speak of Agile generically, we have to discuss the development methodologies that share the common philosophical beliefs of Agile.
Scrum is the most prevalent of the Agile methodologies used in software development today, and rightfully so. With Scrum, development is executed via a series of short efforts called sprints. A sprint typically lasts from two to four weeks. The actual duration of each sprint is determined by the customer and the development team, but a sprint should not be too long. We’ll talk more that sprints later.
Scrum Terms and Concepts
Before we begin our discussion of Scrum, we first need to discuss some important terms and concepts associated with Scrum.
- Scrum Team – the Scrum team is generally made up of less than 10 developers, business analysts, testers, etc. but can be larger if the product necessitates it. The team is totally self-sufficient and able to execute sprints effectively. In Scrum, roles such as architect, engineer, developer, etc typically go out the window in lieu of a sense of a team effort and an “all-for-one and one-for-all” mindset.
- Scrum Master – the Scrum Master is a team member or manager whose primary responsibility is making the team as productive as possible. The Scrum master should be very well-versed in Scrum concepts and ideals and should insulate the team from outside influences and remove impediments that stand in the way of progress.
- Daily Scrum – one of the key concepts of Scrum is the daily scrum. This is a quick, stand-up meeting of the development team. The purpose of the daily scrum is to keep the team on track on a daily basis by discussing the following three things for each team member: (1) what did you do yesterday? (2) what are you going to do today, and (3) do you have any impediments? The Scrum Master leads the daily scrum and ensures that the meeting is productive and stays on track.
- Product Backlog – the Product Backlog is the all-inclusive list of every feature and component of every application that makes up the project as a whole.
- Sprint Backlog – the Sprint Backlog is created from the Product Backlog each sprint during the Sprint Planning meeting. It is made up of all the tasks to be completed within a specified sprint.
- Sprint Planning – the Sprint Planning meeting takes place before each sprint and is attended by the Scrum team members. The purpose of the Sprint Planning meeting is to define the work to be done by each person on the team for that sprint. Once this work has been defined and assigned, work items are moved from the Product Backlog to the Sprint Backlog.
- Sprint Retrospective – one of the most effective ways to ensure continuous improvement within a Scrum team is to learn from mistakes. The Sprint Retrospective meeting, which takes place and the end of a sprint allows the team members to discuss the happenings of the prior sprint and identify things that could have been done better.
- Sprint Review – the Sprint Review meeting is attended by the Scrum team and as many stakeholders as is necessary to demonstrate the new functionality and discuss its suitability to the needs of the business. The Sprint Review is kind of a show-and-tell where the team gets to present to the customer. This has always been my favorite part of a sprint! 🙂
- Product Owner – the Product Owner is the project’s primary stakeholder and is usually a member of functional management within the organization.
Scrum in Practice
Now that we know the key terms associated with Scrum, let’s talk about how the Scrum development process actually works!
The necessary members of the team meet with the customer before work ever begins and identify to whatever extent possible the overall needs of the customer. These needs are not necessarily going to be truly complete and remember that Agile methodologies love change! But we do want to at least have a good understanding of what needs to be done to satisfy the customer’s needs and ensure that the results are in alignment with these needs.
Once we have identified the needs, we build the Product Backlog. As we discussed in the previous section, the Product Backlog is the full, all-inclusive list of the features needed. The business analysts, who are Scrum team members as well then do their best to decompose those needs into quantifiable requirements. A good requirement is a requirement that is testable, so it is imperative that the requirements are developed in such a way that the developers are able to understand them and further decompose specific tasks necessary to develop the requirement in its entirety.
Once we know (to the best of our ability) the Product Backlog, we can decide roughly how many sprints are needed to develop the functionality and assess the actual duration of each sprint. Based on the complexity of the features and the dynamics of the team, we may say that a three-week sprint is optimal, but sprints are typically from two to four weeks in length. Once we identify this duration, we generally want to stick to it and not deviate.
Before we begin a sprint, the Scrum team holds a Sprint Planning meeting. In this meeting, they review the Product Backlog and each developer decides just how much he/she can take on and complete and often high-level estimates are given based on the concept of shirt sizes (S, M, L, XL, XXL, etc). Other approaches are used, but the point of the Sprint Planning meeting is to determine and vote upon just how long the team thinks each item should take to develop and test. The Sprint Planning meeting may last from an hour to an entire day, but the outcome is the movement of work items from the Product Backlog to the Sprint Backlog and assignment of specific tasks to individuals.
When the sprint begins, the Scrum team works to develop and test the exact functionality defined in the Sprint Planning meeting. During the sprint, the Daily Scrum is instrumental in keeping each member of the team and the team as a whole on track and it is imperative that each team member knows what is going on around him/her and how the team as a whole is performing toward delivery of the feature(s) or functionality at the end of the sprint. The Daily Scrum should be as short in duration as possible, and most teams prefer to literally stand up during the meeting. Though the Scrum team should have a deep sense of camaraderie and sometimes a need to get off of the subject 🙂 the Scrum Master must keep things on track and make the team stay focused on the purpose of the meeting. During the Daily Scrum, the Scrum Master goes around the room and asks every team member the following questions:
- What did you do yesterday?
- What are you going to do today?
- Are there any impediments preventing you from getting things done?
If there are valid impediments, it is the Scrum Master’s responsibility to remove those impediments through whatever means necessary. Generally, the Scrum Master should be a person with adequate authority and credibility with the customer to be able to communicate with the necessary people to resolve any impediments encountered. This is vitally important to the successful implementation of Scrum!
Remember that as part of Agile, stakeholders are actively involved as often as is possible. It is not uncommon for stakeholders to be physically involved on a daily basis. During the sprint, developers may routinely do desk reviews with a stakeholder or stakeholders to verify proper functionality during development. A desk review is an informal, impromptu discussion/demonstration of a specific function with the customer.
At the end of a sprint, the team formally presents the new functionality to the customer and receives feedback in a Sprint Review meeting. If the sprint has been properly executed and the customer has been actively involved during development, the functionality that the customer sees should bring no surprises. It is not uncommon during Sprint Review meetings for the customer to engage in in-depth discussion about the business needs and it is also fairly common for there to be a realization than maybe some key element is missing. Thus, change is introduced into the mix and this is documented by the Scrum team. This change may likely go into the Product Backlog for inclusion in a future sprint.
Finally, at the end of each sprint, the Scrum team conducts a Sprint Retrospective meeting. This meeting is attended solely by members of the Scrum team and the purpose is to give the team the chance to reflect on anything and everything that can be done to function and perform more effectively going forward.
Agile Methodology – Using Agile to Develop Better Software and Reduce Risk
Introduction
Imagine an enterprise software development project where the customer says “we are going to take a long time to get this done and we don’t expect to see any results for at least two years”. Can you imagine it? Me neither, and the truth is that it will probably never happen 🙂 So what is reality? In the real world of enterprise software development, the key for any development team is to provide maximum value to and work closely with the customer, to be able to build a culture of true ingenuity, and to be able to meet the customer’s changing needs in a way that there is minimal disruption, if any.
In the early days of software development, it was not uncommon for months to pass before any development began, and once development started, it could be months or years before any type of finished product was ready for testing. The requirements definition and gathering process was often very long, and in many cases the development team was isolated from the customer.
Once requirements were complete and development had begun, change was just not something that was easily entertained. Let’s keep in mind that concepts such as Continuous Integration and Configuration Management were unknown and use of source control repositories was not as mainstream as it is now. A change in requirements was just quite difficult to accommodate and was generally frowned upon.
The Agile Methodology
Agile was first introduced in February 2001 via the Agile Manifesto, a document created by a group of developers who met in Snowbird, Utah to discuss the principles behind a way to do lightweight software development. Since then, the Agile Methodology has grown and been widely adopted by software development teams and companies worldwide.
When we discuss Agile Methodologies, we must also mention Scrum, Lean Software Development, Kanban, Dynamic Systems Development Method (DSDM), and Extreme Programming, since these methodologies all share the same philosophy.
In a nutshell, Agile is about communication, teamwork, collaboration, adaptability, iteration, feedback, and of course, agility! The development initiative is broken down into efforts of short duration and change is not only expected, it is embraced by all stakeholders. To successfully implement Agile, an organization must embrace its concepts and philosophies at all levels.
Agile provides a framework with which teams can maintain focus on rapidly delivering working software and providing true business value, even in environments where the technical and functional assets and landscape may vary or change routinely. We can say that Agile allows development teams to provide maximum business value through the delivery of truly valuable, working software that meets the business needs. How do we know that the software truly meets the business needs? Because all of the stakeholders are involved and quality and scope verification take place in short, iterative cycles. Deviations from the true purpose of a feature or piece of functionality can be identified quickly and corrected in an agile manner.
If we go back to the Agile Manifesto, there are 4 key points that it outlines.
It favors:
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
The key principles behind these points are outlined below (read these carefully):
- Satisfy the customer through early and continuous delivery of working software
- Change is welcomed, even late in the development process
- Working software is delivered frequently, typically at intervals from two weeks to two months
- Developers work directly with functional personnel/SMEs on a daily basis
- Projects are built around motivated, capable people and they are given an environment that allows them to succeed
- Face-to-face communication is critical
- The primary measure of progress is working software
- The development pace must be sustainable
- Continuous attention to technical excellence and good design enhance agility
- Simplicity is essential
- The best architectures and designs emerge from effective, self-organizing teams
- The team routinely reflects on past performance and seeks ways to do things better
If Agile is properly implemented, with buy-in from stakeholders at all levels of the organization, productivity and competitive advantage are maximized and cost is minimized. Of course Agile is not necessarily about reducing cost, but when properly implemented and managed that is a side effect that is very nice.
Let’s discuss the key points above in greater detail.
Favor Individuals and Interactions over Processes and Tools
The greatest processes and tools in the world are worthless without the right people effectively communicating and interacting. Regardless of the size or maturity of the organization, we should start with people then decide the appropriate processes and tools to make our Agile development more effective.
Favor Working Software over Comprehensive Documentation
In the days of waterfall development, I can remember the latter stages of larger projects being consumed with the creation of mounds of documentation! I remember working with teams of technical writers as they produced both functional and technical documentation for software deliverables. With Agile, any documentation that is created is usually created while development takes place. The rapid develop/release approach facilitates concurrency among developers, business analysts, and writers, and in an Agile environment the business analysts often produce the documentation.
Regardless of the use of Agile or not, it is rare that a customer not require some type of documentation and there is nothing wrong with that. But, in an organization that is truly Agile-oriented, working software is always the primary, core deliverable.
Favor Customer Collaboration over Contract Negotiation
Let’s face it, as long as development teams provide services for customers, there will always be contractual obligations. But when we use the term “contract negotiation” we imply an us versus them mentality and this is detrimental to the Agile process! For the Agile process to be effective, we need contractual vehicles that are flexible and that are developed and written to effectively handle change.
It is not uncommon to work with a client via a Firm Fixed Price (FFP) contract. From the customer’s perspective, FFP is preferable because it transfers risk to the service provider. In this case, Agile is still a valid development methodology IF the customer understands and truly embraces Agile concepts. The difficulty sometimes comes into play when the customer insists on defining functionality up front, forces the service provider to sign a contract whose estimates are based on these initial requirements, then tries to introduce scope creep as the project progresses. I sometimes refer to this as “agile under waterfall”, but Agile is still a good fit for such an endeavor. Obviously, a FFP contract is not the preferred vehicle under which to execute Agile, but it is still attainable if all stakeholders are well-versed in and embrace Agile concepts.
Favor Responding to Change over Following a Plan
Although detailed project plans and fancy Gantt charts are impressive, they are not useful with Agile. You read that right! Agile is based on release schedules where the prescribed functionality may be defined, but it is understood that it may change. Project progress within Agile is based on burndowns. Regardless of the actual functionality delivered, progress is still made over time. The total estimate may change due to newly-identified requirements or scope changes from the customer.
Agile and Risk Management
Prior to the emergence of Agile, a large number of software development projects failed or were cancelled with little or no working functionality in place. Teams often spent months or years working on a project with nothing tangible to show for their efforts. In many cases, projects were developed and delivered only to find that they did not meet the true needs of the business! Imagine after months or years of work and possibly millions of dollars of investment to discover that your needs haven’t even been met!
From the Project Management Institute’s (PMI) standpoint, Risk Management is a key knowledge area and something that is very high on the Project Manager’s priority list. All project managers should understand risk. It is just an inherent dynamic within any project and one that has to be understood, and either avoided or mitigated. So, what is risk? By its formal definition, risk is something that can or may occur and that could cause unexpected or unanticipated outcomes. Project managers know that risk is not always something negative. Opportunities are risks as well. But risk is something that, positive or negative, has to be identified, quantified, and managed. The situation, environment, project, people, etc determine when, where, and how risks are managed.
Agile reduces risk through through stakeholder involvement and rapid, iterative development and release. This means that evaluation of scope verification take place routinely, which effectively reduces risk.
Organizational Threats to Agile
The greatest single threat to Agile is management! More specifically, functional management with unrealistic expectations. In some organizations, Agile is nothing more than a buzzword because the stakeholders have not been educated in its fundamental concepts.
Earlier in this post, I mentioned the need for Agile to be understood and embraced by every stakeholder, from the top down. Without this understanding and support, it will likely fail or at the very least leave managers with a bad taste due to the fact that the development Project Manager tells them “we can certainly modify our approach and give you functionality X but requirement W is going to have to be pushed back to a future iteration.” In the case of FFP, requirement W may just have to drop off entirely!
With Agile, change is welcomed, even late in the development process, but in the case of FFP, it is possible that certain changes can significantly affect the project end date and thus necessitate contract extension.
Conclusion
So, Agile is a software development methodology that fosters rapid delivery of valuable, working software in an iterative manner. It values people and communication over processes and tools. It prefers working software over comprehensive documentation. It favors active and dynamic involvement of the customer and proper, effective identification of the true needs of the business over contract negotiation. It advocates the ability to nimbly respond to change, even late in the development process to following a detailed, pre-defined plan.
It can be argued whether or not it negates the need to perform Risk Management, but it is safe to say that with constant and active involvement of the customer and self-organizing, professional, competent, and productive development teams with a true dedication to the customer’s mission and a clear understanding of the customer’s needs, it can be enormously successful and a win-win for both the customer and the development team.
I will reiterate this final point because it is so hugely important – Agile MUST be understood and embraced by every member of the organization, from the CEO to the day-to-day tactical and functional personnel. Without this complete organizational buy-in and effective communication and interaction between all involved stakeholders, it will not be successful.
In the next post, we will discuss Scrum in detail.
John Nelson's Blog 