Shadow Backlog

ShadowBacklog

The team was under pressure to deliver a rewrite of the flagship product after having been handed an agressive delivery date. In spite of the Scrum board and other information radiators in their area, the Scrum Master had to deliver progress reports to the CEO every day. The company president and vice presidents attended sprint reviews and grilled the team whenever a story was not closed or sprint goal not met.

One could easily argue that a fixed scope and fixed delivery date is in no way agile – end of discussion – but this still happens in organizations that use (at least partially) agile practices leaving Scrum Masters to navigate choppy waters.

Of course the team developed some coping mechanisms, the most insidious of which was kicking the can down the road.

Whenever the team encountered unexpected complexity or scenarios when working on a story, they said “this wasn’t in the story that we planned, so let’s call the current story done so that our sprint metrics will look good and handle the extra stuff sometime later.” Everyone except for the Scrum Master was happy – for awhile. Team velocities looked good. Stories were being closed as planned. On the surface it looked like features were being completed. The teams were going to deliver on schedule. The Scrum Master knew deep down that they were either going to deliver late or deliver an application with enormous amounts of technical debt. She also knew that the organization could not handle to truth so she soldiered on.

Then the bubble burst. The collective “oh crap” moment happened when people realized that the system should be production-ready but was not. Some features were incomplete. Bugs were still there from parallel testing that had been performed months ago. There was no room in the time before the expected completion date to resolve all of the open issues. Market launch would be delayed.

The ‘shadow backlog’ existed in the developers’ minds and in the list of open defects that were logged elsewhere from ongoing user testing. There were no product backlog items for them and no sprints in the release plan for addressing them.

Reflections:
– Make sure that all remaining work is visible. Create backlog items for stories, technical debt, and bugs. The backlog consists of Product Backlog Items (PBIs), not just user stories.
– Get user feedback during the sprint to avoid surprises later. Ideally users are integrated into the process via acceptance test driven development (ATDD). At a minimum they have an opportunity to see stories in action and provide feedback in the sprint. Any feedback and new ideas that cannot be incorporated during the sprint should be captured as real backlog items and sized/prioritized.
– Don’t be afraid to keep a story open if it was more difficult or took longer than expected when sprint planning was originally done. Better to take a little heat now rather than have the bubble burst later.
– Avoid ‘bucket stories’ near the end of a release that become a dumping ground for stuff that did not get completed in the sprints. Bucket stories can quickly grow exponentially and delay the release.

The Heartbeat of an Agile Team

What is the one focal point that catches every team member’s attention at the same time every day?  Of course it is their Scrum board. Some teams go through the motions and use their Scrum board because they are told to; for effective teams the board is a way to collaborate, manage work in process, keep track of impediments and know whether or not they are going to meet the sprint goal. Effective teams turn their board into a highly-visible collection of critical information about their effort. It becomes the heartbeat of the team and helps set the cadence for each sprint.

Their board might look something like this:

Board

 

Sprint Dates and Goal – as a reminder of the goal and the timeline. Seeing the sprint dates can be helpful if users who will be testing attend the standups so that they understand the sprint timing.

Columns for each step in their development process – This can be as simple or as complex as the team wants – it usually evolves as the team learns and tweaks its process to ensure a smooth flow of stories. In the example above, the team experienced issues with too many stories waiting for user testing. They created a column to show which stories are ready for user testing to make it clear to users when something is ready for them to test. During standups, the Scrum Master points out any columns where an excessive number of stories are piling up. Even with Scrum, the team may set limits for how many stories may accumulate in each column before team members swarm to address a bottleneck.

Sprint Calendar – during sprint planning the team members estimate on which days of the sprint they think stories will be Done and also when they will likely reach any other significant milestones in the team’s process such as being ready for user testing. This is a way of performing a sanity check for the sprint plan.  It is also a way for the team to gauge whether or not it is on track during the sprint. When user testing is part of the Definition of Done, it allows the users to plan when they will be performing this testing. Putting team members’ planned vacation days on this calendar avoids surprises in the middle of the sprint. It can also help to put key SME’s or users’ vacation days on the calendar if these could impact the sprint.

Impediments – unresolved impediments. Making these visible is a way to track them and may have the side benefit of prompting outside stakeholders who attend the standup or visit with the team to help the Scrum Master by finding other ways to resolve them.

Additional Stories – these are stories at the top of the Product Backlog that are ready to be brought into a sprint but that are not part of the current Sprint Backlog. These are posted on the board so that they are readily available in case the team reaches the sprint goal early and can complete some additional stories in this sprint. Note that these are not called “Stretch Goals”. That term can be perceived negatively by the team, almost as if they are not pushing hard enough in the first place to meet the sprint goal.

Release Burnup – the bigger picture of what the team is working towards. This help avoid the sense of being in an endless churn of sprints with no end in sight.

For teams using Kanban, many elements on this board still apply but may need slight modification. Note the use of colors throughout the board.  It is amazing how teams will adopt color schemes or symbols that communicate the state of their work, highlight important events, serve as key reminders, etc. One type of information not included on the example above is an area for Technical Debt.  Creating an open area on the board for this invites the team members to note it as soon as it becomes apparent to them.  These notes can then be translated into backlog items during grooming. This encourages open discussion about technical debt and surfaces this information rather than just having it in the team members’ heads.

A note for teams that use online/electronic Scrum or Kanban boards: additional information such as the PTO calendar, Impediments and release burnup can be captured in story cards and placed in a column dedicated to that background information. The sprint calendar can be captured by putting Due Dates on each electronic story card. On a physical or electronic board, the story cards can contain as much information as the team finds helpful. Typically the story number, title, and number of story points are included.

Reflections:

  • The Scrum or Kanban board can be used to communicate much more than just the status of the stories and tasks.
  • The information on this board should reflect each team’s personality and process. It will evolve as the team learns.
  • The Scrum Master can help the team during the standup by directing their attention to potential trouble spots on the board.

Essential Agile Skills

As organizations adopt agile practices, there are several key skills that differentiate high-performing teams from mediocre teams:

1) Working in short timeboxes

For a team that is accustomed to delivering working code every six or eight months, two or four week long iterations can be unfathomable. But doing this is essential to the success of the project in order to get the benefits of continuous code integration into a testing environment and frequent user feedback. The sprint’s timebox forces the business stakeholders to constantly question the scope of each feature at a micro level and decide when to carve out functionality.

2) Splitting work into small pieces

The most common excuse for teams wanting to plan sprints using large user stories is that it is “more efficient” to do all of the coding work together and test it at the very end or in the next sprint. This perceived efficiency is self-deception. Successful agile teams know that they must put working code in front of their users early and often. Remember the old saying that “users often don’t know their requirements until the software fails to meet them?”

3) Communication and transparency

Old-school silos between IT, Product/Marketing, and business operations groups are an agile anti-pattern. Political games, finger pointing, and one-upmanship are a huge waste of an organization’s resources and energy. A grown-up culture of transparency allows all of these players to collaborate and solve difficult challenges. For the Scrum Master, this means posting project information in high-traffic areas of the office. More importantly, it means being open with stakeholders about the challenges that the team or project is encountering. Team members can feel secure speaking freely about impediments and project risks in a public forum without fear of reprisal. The essential skill here is knowing how to voice issues in a way that does not sugar-coat them but does set the stage for a constructive discussion.

4) Swarming, collaboration, team first

“Your side of the boat might be leaking but my side is fine” attitude just doesn’t cut it anymore. Lone gunmen need not apply. The most effective agile team members will gladly work outside of their skill sets and comfort zones if it helps the team succeed. Agile developers welcome and seek opportunities to discuss functionality with end users to understand their goals and needs. They leverage techniques such as User Acceptance Test Cases to capture and understand detailed requirements.

5) Focus and finish mindset

This is another one of those things that is counter-intuitive at first for people new to agile. Coders are used to getting all the coding tasks done so that they can get all of the stuff to the testers. Understanding that high work in process = low throughput is key. Developers working in lockstep with testers results in higher throughput, even if that means that the developers have to swarm on non-development tasks every now and then.

6) Scope negotiation

Scope negotiation means being relentless both when talking about items in the current sprint or when doing release planning. Sometimes it takes two, three, or four conversations about an item to admit that it is not really needed.

7) Talking about requirements at the right level of detail at the right point in time

Skilled agilists recognize that the conversation that happens in a user story writing workshop is at a much higher level than the conversation about a story right before that story is pulled into a sprint. They also shun detailed requirements until the sprint starts, knowing that a large collection of detailed requirements is inventory that is subject to decay.

8) Automated testing

Refactoring is a fact of life as requirements are progressively elaborated. Effective implementation of test-driven development and automated testing blunts the effort for regression testing as the application evolves.

Introducing Scrum to a Team

GoFor a Scrum Master introducing agile to a team, there is a balance between implementing concepts and practices at a pace that does not overwhelm the team and demonstrating the benefits as early as possible to build momentum and support within the organization.

Each situation is different, but let’s assume that an organization wants to use Scrum and there is an experienced Scrum Master who has been asked to lead a team of people that are experienced with waterfall approaches and are new to agile. Also assume a reasonable level of management support. How should the Scrum Master go about implementing agile practices on the team? For an experienced agile practitioner, the practices are second nature so it can be tempting to try to introduce it all at once, but the reality is that it must be gradually introduced in bite-sized chunks. The key question is which practices should be introduced first and which practices must be implemented in the longer term? Here are some suggestions:

Groundwork

Before starting the first sprint, there are some things that need to be put in place:

  1. Secure a Product Owner (PO). In many organizations this might be a product manager or representative from a business unit. This requires a serious time commitment and product insight so it should not be an afterthought.
  2. Secure a team where the people will be dedicated just to this team, rather than juggling several different projects at once. This can be a big shift for organizations accustomed to managing project assignments via resource utilization/optimization models tied to specific skillsets.
  3. Educate the team (including the Product Owner) about agile principles and Scrum. Hiring a Scrum trainer to conduct a two or three day course is ideal, but if lead time or budgets do not allow this, prepare an overview that touches on the agile principles as well as the mechanisms and ceremonies of scrum. The Scrum Reference Card and Build Your Own Scrum  are good tools to facilitate this.
  4. Be careful in comparing agile to waterfall. Disparaging waterfall will not endear agile to people who have delivered successfully using waterfall.
  5. The new PO will require additional training about what the role involves.  If they were a product manager accustomed to working with a project manager (PM), they will need to understand the additional responsibilities that get transferred from the PM to the PO. To get them started, help them understand that they will need to work with the team on a daily basis.
  6. Change the team’s physical workspace to foster improved collaboration.  Sometimes it is as simple as taking down a few cubicle walls. Other times, it involves locating people in the same area if they are spread out around the building. Be thoughtful about this. Done wrong, too many people are jammed into a small area and they can’t move around without bumping into each other. This builds frustration and resentment that leads people to resist and question the approach that brought this change. Also, an area with plenty of wall space is best.
  7. Pick an iteration length. Start short so that the team can get used to the different ceremonies and rapidly learn from its experiences. This also quickly gives the team and the organization a feel for what happens before/during/after an iteration.
  8. Populate the initial product backlog:
    • Conduct a story writing workshop. Utilize a technique such as story mapping to organize the stories. Don’t get hung up on teaching story mapping techniques – just do it by organizing stories that way on the wall. It will actually be very intuitive to people who are accustomed to structured analysis techniques. A big challenge here will be in writing the stories. If a work breakdown structure (WBS) exists as a result of the initial business case or previous project planning that was done before deciding to use agile, it may be possible to use it as a starting point for populating the product backlog.  This depends on the extent to which the WBS aligns with product features vs. technical tasks.  The more aligned it is with features, the easier it is to translate. This is an opportunity to reinforce that product backlog items are features and not tasks. Also, team members may correlate this to the design phase of a waterfall project. The focus needs to be on “what” the requirements are and not “how” they will be implemented. Stories need to be written in just enough detail for sizing, not with detailed requirements.
    •  Get the PO to prioritize the backlog by business value. Don’t be surprised if every item is given a priority of ‘1’ the first time around.
    • Create a Definition of Done. This provides the basis for sizing stories. This should include user acceptance testing (avoid the use of “done” and “done done” at all costs!). This will be challenging for a team accustomed to a UAT phase after long design and development phases. Doing this reinforces that the team should write stories that represent testable feature increments rather than individual technical layers of the application.
    • Conduct a planning poker session where the team sizes an initial set of backlog items using story points. This can be a big change for teams accustomed to having senior team members provide the estimates. During this process, the Scrum Master needs to make sure all voices on the team are heard.  Planning poker is ideal way to do this. The Scrum Master solicits input from all team members, especially when there are significant differences in estimates. So that the team does not feel as though it needs to define every last detail of each story, help team understand that this is not the last time these stories will be discussed; detailed requirements will be covered in depth once the sprint starts.
    • This next step depends on the project and organization.  If the organization views the project as an agile incubator, the team might have the leeway not to commit to an overall release plan for the product until a few sprints have been completed and the velocity is understood. If not, you may need to use an old project planning approach to derive a delivery date to satisfy the organization until a release plan can be formulated. Be very careful about providing enough buffer for unknowns – any dates that are published may become the benchmark against which success or failure of the effort will be measured. If possible, characterize any release dates as preliminary until the team has had a chance to run a few sprints.

The First Sprint

  1. Plan the sprint.
    • Set firm start and end dates for sprint. Go with either a 1 or 2 week sprint. A 3 or 4 week sprint length is not recommended because it is harder to convince PO not to change the contents of a sprint if it is longer. Also, longer sprints involve larger numbers of stories for a larger team, which can be difficult for them to manage. A shorter sprint length allows the team to demonstrate results quickly.
    • Calculate the team’s hours of capacity to do work on user stories during the sprint, factoring in sprint length, time for meetings, vacation time, etc.
    • Conduct the sprint planning meeting. The first part should involve the development team and the PO. Review the user stories on the backlog starting with the highest priority items. The team should ask the PO any questions they need to help them understand how to task out each story. The PO can stay for second part if they want. Create tasks for each story. Make tasks granular enough (max 6-8 hours) so that it will be easy to have multiple people working on each story. Continue to add stories until team’s hours of capacity for the sprint are used up.
    • Ask everyone on the team if they are comfortable committing to delivering these stories in the timeframe of the sprint and if they have any concerns or reservations.  Go around the table and ask each person – it can reveal some insights that people might not otherwise volunteer.
  1. Set up a story/task board in the team area. Even though the team might have software for sprint planning, go with high-visibility, tactile information radiators. This is important because it helps the team learn that it needs to be self-organizing/self-assigning. The team should update the board throughout the sprint, not the Scrum Master. The Scrum Master needs to change culture of having the project plan buried in project planning software like MS Project and only being maintained by the PM. This also starts to build culture of transparency with stakeholders. Invite stakeholders to visit the team’s work area to view status of current iteration. Of course, be careful that they don’t interfere by trying to change priorities or introduce new stories.
  2. Start the first sprint.
  3. Do daily standups. Start by helping team understand the purpose and the basics.
  4. Some people on the team may have heard the agile myth about “no documentation”. This is the time to start establishing culture of lean, maintainable documentation that has value through the lifetime of the product. A new team will only be able to bite off so much new stuff at one time. If they are accustomed to some type of UX or use case document, use that artifact (during the first sprint) as a transition into user acceptance test cases.
  5. Coach team members that the first thing that they need to do when they pick up a new user story to work on is to get with the PO and applicable users to discuss detailed requirements (since the team didn’t hammer out every last detail of the story during grooming or planning, right?). Reinforce with the PO that they need to be prepared to work with team members on a daily basis.
  6. While the Scrum Master’s eventual goal is a team that limits WIP and individuals self-assign tasks, don’t necessarily expect that to happen in the first sprint. Remember – people on the team are probably accustomed to having their tasks assigned to them by the PM or tech lead via project plans.  Initially, the Scrum Master may need to ask for volunteers for tasks, or as team members report that they are wrapping up a task, the Scrum Master may need to ask them which task they will work on next. Team member will typically look at the next available story that no one is working on. Steer them instead to first look for tasks they can work on for stories that are already underway but not yet complete.
  7. Start tracking the team’s burn down and post in the team work area.
  8. Run interference if the PO or other business stakeholders try to change the stories in the sprint. This is another reason to keep sprints short (max 2 weeks) at first. It is easier to argue against changing the contents of a sprint when the next sprint is less than 2 week away.
  9. Conclude the sprint on the planned end date.  DO NOT extend the sprint under any circumstances, even if it will take ‘just one more day’ to finish all of the stories. It is important to establish this early as it speaks to the sprint commitment being firm and the importance of sizing/organizing the work so that it can be completed within the iteration.
  10. Do the retrospective.  Keep the format simple initially. “What went well, what did not go well, ideas for improvement” is an easy format. Do not allow anyone from outside the team to attend. Make sure the PO attends because they are part of the team and this meeting is an important way to build trust between the PO and the developers. Ensure attendees know that the Vegas rule applies. At the conclusion of the retrospective ask attendees if it is okay to share ideas that were generated with outsiders – stakeholders will naturally be curious.  Ideas that the team does not want to share are not shared outside the team.
  11. In preparation for the sprint review, prepare a sprint summary document.  Show the team’s burn down, stories committed and status of each story at end of sprint.
  12. Conduct the sprint review. Encourage as many people from around the organization to attend as possible. This is about
    • Demonstrating results
    • Team members taking pride in their work by showing it off
    • Showing the organization that the team is transparent

The development team members (developers, QA people, etc.) should conduct the demos.

The mechanics of the first sprint described above are relatively straightforward, but the reality is that this is all occurring against the backdrop of organizational change:

  • Team members are experiencing dramatic change.  Their physical work area has changed. They are being asked to commit to delivering something without detailed specifications. No one is telling them which tasks to work on.
  • The relationship between business and IT may be very different when using an agile approach. If the PO comes from a business unit and because of the organization’s culture there exists a lack of trust or poor communication between the business unit and IT, the lack of trust and communication issues will not necessarily disappear overnight.
  • The organization’s culture may not value or emphasize teamwork. Part of management and the Scrum Master’s jobs is to build a culture where teamwork is highly valued.

The Second Sprint

In the second sprint, the Scrum Master continues to emphasize agile principles and the mechanisms of Scrum.

  1. The Scrum Master continues to coach regarding self-assignment of tasks. As the Scrum Master backs away a little bit and the team starts to self-organize, others (such as the PO) may feel that the team needs to be directed and step in to direct the team and tell them what to do.  The Scrum Master must also coach them not to direct the team.
  2. The Scrum Master continues to encourage team members to swarm on stories by default rather than only when necessary. Lessons learned from first sprint (e.g. too many stories in progress at the same time) make the purpose of this apparent.  Swarming will not happen overnight, but the Scrum Master needs to keep encouraging it.
  3. Conduct user story grooming meetings during the sprint. In the meetings, continue the mantra of discussing features at “the right level of detail at the right point in time.” Use these meetings to size out the rest of the backlog and to revisit stories likely to be worked on in the next sprint. The product owner may be unprepared for these meetings because they are already overwhelmed by working with the team on the current sprint’s user stories. Sometimes it helps for the Scrum Master to meet separately with the product owner a day or two before a grooming meeting to survey the backlog and discuss the next set of stories to be groomed.
  4. Encourage the PO to regularly tend to the backlog – not just before grooming or planning meetings.
  5. Using the team’s velocity from the first sprint, begin working with the product owner to formulate the release plan. It may not be appropriate to go public with it yet. Velocity cannot necessarily be trusted after just one sprint, but doing a preliminary release plan is valuable to get the PO thinking in terms of using the velocity and backlog to formulate a plan.
  6. Start tracking the team’s burn up and also post in the team work area.  The burn up emphasizes that hours worked do not provide business value, only completed user stories do. A graph that shows a steady burn up of completed story points during the sprint (rather than a ”hockey stick” where most of the stories get closed at/near the end of the sprint) is the goal. Don’t necessarily beat the team over the head with this…at this point just introduce it to them and call attention to each story as it is closed during the sprint.

Beyond

Over subsequent sprints, the Scrum Master should consider introducing some of the following techniques.  Of course each situation is unique so he/she will have to gauge whether or not the team is ready for each new practice.

  • For teams that are having trouble swarming and have too many stories in progress at the same time, reinforce the concept of WIP limits.  For teams that already have a task/scrum board with steps in the process laid out, the WIP limits can easily be established for each step. The team can decide whether they are firm limits not to be exceeded or soft limits that are triggers for conversations.
  • Help the team learn the finer points of writing user stories. A book study (e.g. “User Stories Applied” by Mike Cohn) is a good way to do this. Be sure to include the PO in this.
  • Test-Driven Development (TDD).  The benefits of this can be difficult for a team to grasp. Once the team (including the Product Owner) understands that it is okay to refactor and that it is simply part of progressive elaboration, the benefits of TDD become much more apparent.
  • Acceptance Test Driven Development, manual or automated.
  • Introduce additional retrospective techniques to help the team learn about itself and identify areas for growth. The Scrum Master will need to judge what type of retrospective is appropriate depending on how the sprint went.

In addition to learning new practices, the team overall needs to learn to be agile, not just to use agile practices.

  • Work with team members’ managers to help them develop a broader, t-shaped skillset that allows them to perform a larger variety of technical and non-technical tasks.  For specialists, this can be a big shift.
  • Help the team to develop an agile mindset by only learning enough about a feature to size and plan it and then really dig into the details of it once it has been pulled into a sprint. One of the most gratifying moments for a Scrum Master can be when the PO says, “We did not think of that aspect of the feature in when we discussed the story. We will just create a new story and handle it in the next sprint” – without taking the team to task for not delivering it in the current sprint.
  • When the Scrum Master has been steering the team towards new practices but the team has resisted using them, the Scrum Master may need to let the team fail for a sprint. The Scrum Master needs to help stakeholders understand that this is part of the learning process, and two-week sprints will minimize most of the damage if the team gets off-track.

Reflections

  • Every situation is different so of course there is no “one size fits all” approach to starting with agile. Organization and team dynamics impact the speed and extent to which agile is adopted.
  • As the team becomes more comfortable with the agile approach, the Scrum Master should adapt his/her approach to be in more of a supporting role for the team.
  • Occasionally the team will forget lessons learned from a few sprints ago and revert to old habits.  The Scrum Master can bring previous retrospective conversations back into focus to help the team avoid making the same mistakes.

The ScrumMaster as Servant Leader

eadership in a modern organization is a paradox whereby leaders provide a vision of what the organization should strive to achieve but then serve those in the organization in order to realize that vision.  This is embodied in the management philosophy of servant leadership.

As defined by the Alliance for Servant Leadership, the principles of servant leadership are:

  • Transformation as a vehicle for personal and institutional growth.
  • Personal growth and its importance to the individual and the institution.
  • Enabling environments that encourage individuals to share ideas and express opinions.
  • Service as a fundamental reason that people want to lead.
  • Trusting relationships based on mutual respect, trust, civility, acceptance, and reciprocity.
  • Creating commitment rather than imposing controls in order to accomplish valued missions.
  • Community building where individuals work effective together, as teams.
  • Nurturing the spirit as a way to provide joy and fulfillment in meaningful work.

These principles fit squarely with an agile organization’s values of teamwork, collaboration, trust, continuous improvement, extraordinary service, and empowerment.

Exceptional leaders are confident enough to and capable of following a servant leader approach; those who are less skilled are autocratic. A Scrum Master can practice servant leadership by:

  • Seldom, if ever, telling anyone on the team what to do. Becoming a Scrum Master involves learning the various mechanisms and ceremonies associated with Scrum.  The most profound aspect of leading a Scrum team, however, is not in learning a methodology. Rather it is about asking very deep, probing questions and helping the team find its way to a solution rather than giving them the solution.  The benefits and impact of this are significant both in terms of the outcomes but also in the satisfaction and commitment that this builds in team members. It is one thing to say that team members are empowered; it is quite another to actually empower them.
  • Not assigning tasks to team members. For an organization transitioning to agile, other authority figures may misinterpret this as a lack of leadership by the Scrum Master.  If those authority figures attempt to direct the team, the Scrum Master should educate them regarding what it means for a team to be self-organizing.
  • Reminding the team about the product vision and sprint goals. Note that it is also key for the Product Owner for articulate product vision throughout an effort.
  • Removing roadblocks that stand in the team’s way.
  • One of the characteristics of an effective Scrum team is that most of its members have a general enough skill set to perform a variety of tasks and do whatever it takes to “move the ball down the field” as a unit. This requires that people who used to be very focused and very good at a specific type of work or technology develop and grow a broader skill set.  A manager serves their team by helping to steer them towards personal growth opportunities.  He/she enables this personal growth by encouraging team members to seek training for new skills and to take on assignments that are outside of their comfort zone.