VP Engineering Playbook

A practical guide for new leaders in software development

Tag: agile

Agile Revisited

We recently celebrated the 15 year anniversary of the publishing of the Agile Manifesto.  As you’ll recall, in February 2001, 17 thought leaders in software development gathered in Utah to discuss better ways to build software. A set of values and principles were established which provided guidance for planning, building and releasing software. As agile evolved, several methodologies emerged to provide specific practices for the application of agile. Examples of these are Scrum, XP, Lean, Crystal, etc. These methodologies can be opinionated, but do provide companies with ready-made blueprints for structuring their agile programs.

At a few past companies, I have had the opportunity to craft software development processes that align with agile principals. In these cases, I haven’t utilized the strict application of a single methodology.  Rather, I have tried to mold the agile principals themselves into a set of practices that best match the culture of the target organization. Collaborating with other product stakeholders, this usually results in a unique interpretation of agile for that company. Since agile espouses introspection and continuous improvement, adaptation is natural.

Given the 15 year milestone, I thought it would be useful to revisit the values behind the agile manifesto and how those can be applied to a modern software delivery organization. This may be rudimentary to experienced agile practitioners, but might offer insights for new software development leaders charged with being “agile”.

Individuals and interactions over processes and tools

One of the core shifts introduced by agile was recognition that great software is produced by empowered individuals working closely together on small teams. This ran counter to the approach used by large organizations attempting to coordinate activities across hundreds of developers. In order to manage enormous teams, tools were championed that allowed disparate teams to schedule project work into sequences of events (Gantt charts) through desktop applications. With this move to formalized project management, teams stopped discussing the software in person and focused their effort on managing a tool.

Agile shifts this mentality back to individuals and their personal interactions. It encourages practices like the ones below.

  • Limit team size. Agile teams should be small (less than 10) and should include representatives from each function required to address the project work. For Internet apps, this means the product manager, UX/UI designer and engineers from each functional area (front-end, back-end, services, devops, etc.).  The key is that the team doesn’t have a major dependency on someone not part of the core team.  There can be other influencers, like from marketing or customer service, who participate as extended team members.
  • Colocation.  The core team should sit together. Physical proximity encourages high bandwidth collaboration. I realize that newer communication methods like IM and group chat facilitate non-spoken communication, but as humans, we are wired to rapidly broadcast information and gather feedback from non-verbal cues. As one of the agile principles states, “the most efficient and effective method of conveying information to and within a development team is face-to-face conversation.” I am also a big proponent of fluid seating arrangements, where individuals can shift their seating based on their current agile team assignment. They may still have a “permanent” desk, but for the duration of their team assignment, they sit with the other members of the agile team.
  • Encourage informal discussions. I love to see the following situation. One member of an agile team has a question about a requirement. Then, one or more team members suggest they grab a conference room to discuss. The group heads to the room, closes the door and can be seen feverishly working out the problem on a white board. This occurrence stems from the agile concept that these types of informal, in-person communications should occur frequently during a project. These should not be categorized as interruptions and discouraged.
  • The team should have the proper environment, tools and support needed to accomplish their mission. In order to be self-empowered, the team should have full control over their work environment, the tools they use and necessary support from other teams. This is where engineering leadership can contribute to the team’s success. If you are a member of leadership (director, VP Eng, etc.), a big part of your job is to monitor your teams’ updates and identify any outside circumstances that might be creating impedance. These could be dependencies on other teams, difficulty getting their work environment set up or access to the right tools. Look for these obstacles and clear them.
  • Trust the team. If the team is staffed with empowered individuals armed with a clear direction, they should be able to accomplish their goals. As a member of leadership, you should trust that they can do this and get out of the way. Resist the temptation to tweak or guide. As long as expected outcomes are clear, maintain your role as an observer.

Agile anti-patterns:

  • Heavy emphasis on updating tools. On a functioning agile team, you should see a lot of informal discussion. The tools used to manage the agile process are useful for reflecting status, but are not a replacement for personal interactions.
  • Rushed meetings.  Over the last 10 years, I have seen a growing resistance to holding meetings. The observation has been that many companies load up employee schedules with so many meetings that they don’t have time to actually perform work. The knee-jerk reaction has been to question the value of all meetings or rush them in order to “get some time back”. Meetings, which I will simply define as a conversation between two or more persons at a pre-scheduled time, have a  purpose in an agile environment. They allow for necessary team member interactions to answer questions, review designs, plan work, or do anything that requires agreement. Allowing these discussions to occur at a scheduled time simply makes it easier for individuals to plan their day.  So, I think meetings are fine, as long as they are limited to necessary participants and produce an outcome. If you see meetings avoided or rushed, step back and determine if the team has a mechanism available to foster debate and make decisions.

Working software over comprehensive documentation

Another key tenet of agile development is to focus the team on delivering software versus waiting for a full set of requirements. Documentation is encouraged in agile. However, agile doesn’t gate the start of development on the completion of documentation. I still see this practice in some shops that profess the use of agile. If the developers wait until the requirements document is complete before they start coding, it will serialize the project and extend the end-to-end delivery schedule. Also, complete documentation is a farce. It is impossible for a product manager to anticipate every facet of a new feature. Many design decisions can be made at time of implementation.

An emphasis on working software provides other benefits in the form of agile practices for structuring work.

  • Break up large projects into incremental deliverables. Agile’s focus on delivering working software is the foundation for the convention of a sprint.  A sprint’s purpose is to time box project work, forcing segmentation of the overall project. Sprints should always include a release to production. This release can either be represented by many small features or portions of a very large feature.  Even if a large feature isn’t fully “live” at end of sprint, it is still important to conduct the release. Access to the feature can be limited to internal users by a feature flag. Each release forces integration and testing, spreading that cost over more cycles.
  • Track progress. Working software is the currency of a project – it provides a clear means of measuring progress. Work items in an agile sprint are usually organized into a burn down chart. This provides an advancing view of completed versus remaining work items. Remaining work should approach zero as the sprint proceeds. Delivering working software throughout the sprint provides the clearest way to validate progress.
  • Lower technical risk. As engineers plan their work items, they should address the technically hardest deliverables first. This allows the team to clear the biggest obstacles early in the project. Addressing these items in working software proves the feasibility of the solution.
  • Apply the simplest approach. Organizing project work into discrete deliverables released frequently encourages product and technical designs to be simple. In product design terms, this is often referred to as the minimum viable product. The value of a MVP is well understood. Until users provide real feedback, it is difficult to estimate the potential success of a new feature idea. Similarly, for infrastructure design, simplicity prevents over-engineering.  While sound technical design is critical to scalability and extensibility, trying to address future scenarios with extra technology capabilities can become wasted effort if those requirements never materialize.  As one of the agile principles states – “simplicity–the art of maximizing the amount of work not done–is essential.”

Agile anti-patterns:

  • Perfecting the design spec before sharing with engineering.  Teams may sometimes delay showing the design spec for a project to engineering until it is fully baked and signed off. The reasoning is that the team doesn’t want to waste engineer time by reviewing an incomplete spec. There is some credence to this, but I also think there is value in sharing a spec before it is complete. This allows the engineers to begin processing the future design – conceiving of improvements and planning their technical approach. Waiting until the spec is fully baked usually creates schedule pressure that limits feedback or negotiation of items that would lower delivery cost.

Customer collaboration over contract negotiation

Agile imbues the software development process with a strong emphasis on the end customer. For consumer applications delivered over the Internet, the end customer represents the millions of users of a company’s apps. Since it isn’t feasible to involve all these users in the software development process, we rely on the product manager to represent their feedback. In this model, the product manager is the customer. There are other customers to an agile project – like outside stakeholders. Even the core team members can be viewed as customers of agile.

Promoting a customer-centric view is critical to the agile process.

  • Conduct regular demos. Outside of the release at the end of a sprint, the team should strive for frequent, informal demos of their work. When a task is completed that can be “shown”, the responsible developer should grab the product manager and do a quick demo. The purpose of this is to solicit feedback as early as possible. Any time delay will increase the cost of making changes. Also, I like to see agile teams demo to each other periodically. One team I managed conducted “Demo Fridays”, where each team member showed what they had built the week before.
  • Speak in terms of customer goals. When the team conducts sprint planning, the product manager should lay out their objectives for the sprint in terms of end user benefits. Focusing the sprint outcomes on the customer allows everyone to align their tasks with what will have the most impact on the business.
  • Maintain a feature backlog that includes tech debt. Agile teams should construct a ready backlog of work items. These usually include feature ideas and product enhancements, prioritized in terms of expected impact. The feature backlog should also include tech debt tasks.  If you aren’t familiar with the term, tech debt represents work items that make future feature delivery faster, easier or more reliable. These are generally items sponsored by engineers that don’t contribute to the product’s feature set. They are important nonetheless, as they represent an investment in future delivery of the product. Teams can determine the appropriate mix for their situation, but I like to see at least 25% of available cycles in a sprint allocated towards tech debt.
  • Radiate information. Agile advocates public display of information related to the project. This could be UI designs, the burn down chart, database schema, system diagrams, or anything else that provides meaningful information to members of the team. This public display is generally done on the walls surrounding the team’s work area, using print-outs or monitors. Information radiators also have the added benefit of providing a convenient mechanism for presenting the team’s work to any visitors. Some teams I have managed publicized periodic “office hours”, encouraging other employees to drop by.

Agile anti-patterns:

  • Formal sign-offs. If you find members of the team requesting formal “sign-off” of delivered items, then the team may have a trust issue. Usually, this occurs at hand-off points in the process, like presenting product requirements or validating completed work items. If sign-off is requested, then the team likely is struggling with their ability to respond to change.
  • Only demoing UI features. Some teams think that only features with a user interface are worth demonstrating. In the spirit of delivering working software, any completed component of a project should be demoed. Even a back-end service with no interface or a database design can be presented. The key is that the developer shows the work to another person.

Responding to change over following a plan

In Internet time, the product landscape is constantly shifting. Yesterday’s great feature idea may no longer be viable today. Teams need to be able to adapt quickly and shift priorities to capitalize on new business opportunities as they emerge. The agile preference for responding to change allows this flexibility. While a plan is important, the team should orient around an expectation for change. As one of the agile principles states – “agile processes harness change for the customer’s competitive advantage.”

  • Prioritize sprint tasks based on expected impact. As agile teams plan their sprint work, they should schedule the highest priority items first. This also implies that if the priority of an active item changes, then its position on the sprint plan will be adjusted as well.
  • Daily stand-ups. One practice to ensure that team members are kept updated on changes to the schedule is the daily stand-up.  I like to keep these focused on the standard set of updates – what each team member accomplished yesterday, plans to address today and if they are blocked. These updates give the rest of the team a sense for the progress of other team members, and most importantly, where the schedule may be deviating from plan. Any issues raised during the individual updates can be addressed in smaller break-out discussions following the stand-up meeting. The stand-up meeting should be short, no more than 15 minutes.  However, enough time should be allowed for sufficient updates and treatment of any follow-up items. Occasionally, I have observed stand-ups being rushed, as if finishing a stand-up in under 5 minutes is an accomplishment.
  • Mid-sprint reviews.  As team members complete sprint tasks, these should be marked off in the sprint tracking tool. This progress against the schedule will be reflected in the burn down chart for the sprint. While progress can be tracked daily, I think it is a good practice for the team to formally review sprint progress at least once during the sprint. I like to do these check-ins mid-sprint, either at the end of a daily stand-up or in a scheduled meeting. The team should display the burn down chart on a shared monitor and discuss progress. Ideally, half the work items are completed at this point.  If not, the team should explore why and make adjustments to the plan for the second half of the sprint.
  • Retrospectives.  Another theme of agile is the notion of introspection and continuous improvement. The process by which the team works should be constantly reviewed using a reliable feedback loop. Retrospectives are formal meetings at the end of each sprint in which the team reflects on the sprint’s activities.  Discussion revolves around what went well and what needs improvement. This review should be conducted with an objective, blameless tone. For the items that need improvement, the team should identify actionable tasks to address them. Those tasks are then added to the backlog for future scheduling.

Agile anti-patterns:

  • No changes to the sprint plan as the sprint progresses. If the sprint plan always proceeds exactly as expected, then there is likely something wrong. The team should take advantage of the fluidity of agile to ensure that the most important work items are being addressed first.
  • Retrospectives with no improvement items. I have observed retrospectives for some teams narrow to a list of compliments between team members. Recognition is important, but retrospectives should also result in improvements. I like to see a 2:1 ratio of items that went well versus those that need improvement. This ensures there is space for recognition, but enough critical thinking to drive the team towards continuous improvement.

Structuring your Agile Sprint Schedule – Part 2

This is a continuation of a post on how to structure your agile sprint schedule.  In Part 1, we discussed approaches to sprint length and the planning process.  In this post, we will examine testing and release.

Testing

Developers should be writing unit tests for their code.  In test-driven development, these are written first. I don’t strongly advocate for this approach, but I do insist on unit tests being included in any reasonably sized code check-in.  Unit tests should not be left to the end of the sprint, or rolled into tech debt.  Also, you should have a continuous integration system in place that runs the full set of unit tests against the code base periodically (ideally on every check-in).  If a unit test fails, the issue should be corrected by the associated developer immediately.  I have seen continuous integration runs result in increasing numbers of unit test failures, to the point where they become pointless.  Ensure you enforce the discipline within the team to prevent this.

Your QA team should be involved throughout the sprint process.  Don’t just pull in a QA team member at the end of the sprint to conduct regression testing.  A representative from QA should offer input into the planning process and should work with developers to test components of sprint deliverables as they are built.  These  front-loaded tests are very valuable, as they can flag issues early in the sprint.  Otherwise, all the issues will accumulate to the final regression test.  As the sprint end is approaching, the team should try to taper down code commits representing new features and allow the QA representative to focus on end-to-end testing.  Bugs that surface from full integration testing can be addressed in the final days of the sprint.

Release

I define the release process as the activities that occur between the final planned code commit and the point at which the code is stable in production.  With this definition, release includes the following activities:

  • Branch Cutting.  Depending on how your team manages its code base, you will likely create a branch off of your main code line for the sprint release.  Often called the “release candidate”, this separate branch allows your team to test and apply fixes to the release candidate in isolation from the main code line.  The main code line may continue to get changes for the next sprint, without blocking developers while final regression testing occurs.  Your release engineer (often a designated developer on a small team, or a dedicated role on a larger team) will coordinate and drive the branching activities.
  • Unit Test Run.  Following the branch cutting, you will want to run your full suite of unit tests on the release candidate in your continuous integration environment.  Any unit test failures should be addressed by the developers immediately.
  • Regression Testing.  Your QA team members will install the release candidate into a dedicated test environment.  They will run through their suite of regression tests.  Some of these may be automated (ideally), but they will likely need to conduct a good percentage of tests manually.  Particularly if your application has a rich UI and complex business logic, having a set of manual “sanity” tests is important.  As the QA engineer finds bugs, they should log them for developers to fix.
  • Launch Planning.  As the QA engineers and developers are wrapping up the regression test process, the release engineer and team leads should conduct release planning.  This involves determining the sequence of steps required to move the release code to production.  This includes a plan for any database schema changes, with details of how those will be applied. Also, a list of post-launch checks should be assembled.  These will likely include business metrics and application performance monitors.  Finally, a rollback plan should be determined, in the event that there is an issue with the release.
  • Release.  Once QA has signed off on the release candidate and all bug fixes have been committed, then the team is ready to start the actual code release.  At Zoosk, we would conduct a quick stand-up with all sprint participants to review the launch plan and ensure that post-release checks were delegated out.  This stand-up also included the product manager(s), so that they were aware of the imminent release.  After code was pushed live, release checks are conducted.  Once all release checks are cleared, the release engineer calls the release stable.  If a release check indicates an issue, the team can troubleshoot it quickly and determine if a rollback is required.

Other Considerations

  • Holidays.  If a holiday occurs during your sprint, just subtract that day from the number of days available in that sprint.  I don’t recommend that you extend the sprint by a day, as this would likely break the integrity of the Monday-Friday sprint schedule.
  • Progress Checks.  While your team(s) will report on sprint status at daily stand-ups, for larger teams it makes sense to have a formal check mid-sprint on progress.  For a multi-week sprint, this usually works best on the Friday in between.  Participants should be the product manager and team leads.  At this progress check, the participants should review completed versus planned work items and gauge the likelihood of completing all items by the end of the sprint.  If it appears that all work items will not be completed in the second half of the sprint, then the product manager and team leads should determine a course of action to take. This can involve reducing scope of a work item or scheduling it into a future sprint. Planning for developer “overtime” as a means to catch up should be avoided, but does represent an option in some cases.

Structuring your Agile Sprint Schedule – Part 1

A key tenet of the agile software development process is that all work is divided into repeatable sprints.  There are many forms of agile (Scrum, XP, etc.), each of which prescribes a different approach to structuring your sprints.  While these approaches to agile can differ radically, several general patterns exist, which you can use to decide how to structure your team’s sprints.  Most companies that I have worked with take the principles of agile development and apply them to their team’s unique culture and preferences.  The result is a set of practices that fits for that software development team.  This two-part post will provide some considerations for deciding how to structure your sprints.

Sprint Length

It is critical to the agile process to have a fixed sprint length.  The sprint is usually defined in a number of days.  Work items are scheduled into the sprint in the order in which they need to be addressed.  The highest priority items are started on Day 1 and the team works through items until the end of the sprint is reached, which we will call Day X.  X is then the number of days in the sprint.  The sprint period should rarely change and be recorded on your company’s shared calendars.  Sprints will have distinct names and are usually referred to in ascending numbers.  For simplicity, you can just start at sprint 1 and increment from there.

In terms of selecting the sprint length, as with many things, it depends.  I don’t think that a particular sprint length applies to all teams and all situations.  I do think there are some book ends – sprints need to be at least a week and shouldn’t be longer than a month.  Sprint work also should align with the business week.  The weekend provides a natural break in the work cycle, that also creates a strong dynamic around “starting” work on Monday and “ending” work on a Friday.

Determining  the length of your sprints can be influenced by a few factors:

  • The size and inter-dependencies of your team(s).  If you have a small team (5 people or less), who do not have significant dependencies on other teams in order to finish their sprint work, then a shorter sprint (1 week) will work.  If you have many developers on multiple teams contributing to the same sprint, the coordination overhead will be large enough to justify a longer sprint.  Since this overhead is paid every sprint, developer cycles can be optimized by lengthening the sprint and including more items in the coordination.  In this case, a 2 or 4 week sprint is probably appropriate.
  • The breadth and complexity of the application.  A large, monolithic code base will often require a large set of regression tests that must be run in order to verify that the code changes made in the sprint are ready for production. While automation can reduce the amount of time these require to run, if you have a complex, user-facing application that spans multiple device types, you will still likely have the QA team and product managers test the sprint’s release candidate manually before release.  If this regression testing requires a day or two to conduct, then you will incur this cost on every sprint.  In this case, a longer sprint makes more sense.  On the other hand, if the sprint participants are contributing to a smaller, encapsulated application (or service without a front-end), then the regression testing requirements may be less, or even fully automated.  This would allow for a shorter sprint, like a week.  I realize that a four week sprint may be counter to the “move fast and break things” mentality, but there are many types of mission-critical applications where the final QA process must be thorough.  A banking application would be an obvious example.  In these cases, a longer sprint would allow a higher percentage of sprint time spent on development.
  • The amount of overhead in your sprint release process.  While you can perform production releases during the sprint, these are often done on separate branches, representing small fixes or activation of feature flags.  The assumption is that your sprint ends with a substantial amount of code released to the production environment. If your change management or release process requires substantial time to actually plan, stage and push the code, then longer sprints would reduce this repetitive overhead.

Where you have leeway, I would lean towards a shorter sprint.  The sprint length will set the general cadence for your organization and create the sense of urgency around how work is planned.  A long sprint will naturally reduce that sense of urgency and the healthy pressure to complete items.  Where you have time-consuming test requirements or an encumbered release process, look for opportunities to streamline and automate.

Planning

Sprint planning represents the activities that determine what work will be scheduled into a sprint.  Most teams maintain a “backlog” of work items that they would like to address in the future.  These can be product changes or technical debt.  These items are prioritized by the product manager and team lead.  As part of the planning process, the engineering team lead will apply a “cost” to each item.  This cost is usually expressed in developer time (2 developers for 3 days).  The planning exercise results in an agreed set of work items that will fit into the time available in the sprint.

There are a couple of items to think about as part of planning:

  • When to conduct planning.  Planning can be conducted within that sprint’s boundaries, where the team spends Day 1 of the sprint on planning.  Alternately, planning can be performed during the final days of the prior sprint, and complete before the new sprint starts.  I prefer to conduct planning activities outside of sprint boundaries.  The reason this works is that the team lead and PM normally conduct this planning work, leaving the individual developers to continue finishing their sprint work.  If your team is very flat, or your team lead can’t represent cost without developer input, then you might consider making planning a part of the actual sprint.  In this case, you would allocate a day or two at the beginning of the sprint for everyone to work on planning.
  • Balancing feature work with technical debt.  Items planned for a sprint should not just represent product features.  You should also pull in a healthy amount of tech debt in each sprint. In case you aren’t familiar with the term, tech debt refers to work items that improve the state of your infrastructure or code base.  They usually generate a developer efficiency or create a new technical capability.  They are often sponsored by the engineering team, versus the product manager.  I usually shoot for a 2:1 ratio of  features to debt in a sprint.  The ratio between features and tech debt can vary from one sprint to another.  In some sprints, your product manager may ask to knock out several high value features, pushing out tech debt items.  Just remember to get the postponed debt addressed in the next sprint.

This post is continued in Part 2.