VP Engineering Playbook

A practical guide for new leaders in software development

Page 3 of 6

Challenges of Maintaining a Monolithic Back-end

As new Internet-based companies get started, they will often organize all back-end code into a single codebase. This code is deployed onto one tier of servers, representing their primary back-end infrastructure. Using a single application back-end is often referred to as a monolithic approach.  This approach is contrasted with a service-oriented architecture, where application logic is divided into a suite of services (more recently referred to as microservices). Each service is independent from the others, with a separate codebase and deployment target.

Maintaining a monolithic back-end can offer some advantages to a small team, but will create challenges as the application grows. In this post, we will examine some of the considerations for maintaining a monolithic back-end, versus segmenting out logic into discrete services.

Duplication of Logic

A single application often contains duplicate logic in many places.  As changes are planned to the application, those changes will need to be applied to all places where the duplicate logic exists. By breaking the application up into services, this duplicate logic can be consolidated into a single service. Changes to that logic will then need to be made in only one place. In a monolithic application, this code duplication can be reduced through an object-oriented approach, defining independent classes for each major type of functionality. However, the functionality provided by these shared classes will still need to be reviewed and understood before the class can be consumed. This creates overhead for the team, requiring each developer to maintain an understanding of a vast set of classes that can be incorporated into the main application.

Clear Contracts between Services

Migrating to a service-oriented approach requires the definition of clear contracts between services. With a well-defined interface, service functionality becomes intuitive. In order to allow other teams to interact with the service, the owning team will generally publish documentation specifying the interface.  This will be structured like an API, with function calls, inputs and outputs delineated. This interface documentation will be very useful for new team members, allowing them to learn about the service independently. Consuming teams can also ignore the inner workings of the service, as long as they maintain clear interactions through the service’s interface. This will allow development for consuming teams to proceed more quickly.

Speed of Product Evolution

Organizing around services allows small, nimble teams to be assigned to each service. These small teams usually consist of a few developers, a product manager, a devops engineer and a tester. A team this small can enhance their service very quickly. Vetting proposed changes to an application with all affected parties is usually the largest encumberence to making rapid decisions. Decision making is slowed down as the organization involved becomes larger.  Having a small, autonomous team assigned to each service will allow this decision making to proceed quickly, as all of the individuals necessary to review and approve a decision is minimal. On a large monolithic application, the engineering team will usually be segmented into smaller sub-teams, like search, content management, community, etc.  This does offer some separation of concerns.  However, if all of these sub-teams are contributing to a common code base, then dependencies will still emerge and the decision making overhead will continue.

Code Sharing

Part of establishing a new service is to provide the boilerplate functionality to perform many common functions. These can include call marshaling, security, response formation, database access, etc. Once a single service is established, much of the service interface code can be templatized and re-used in other services. This will lower the initial roll-out cost for services over time. It is helpful to organize this shared code into a set of libraries or independent code modules by function. The shared code should be stored in a separate project within your code repository. You can also consider whether to assign a particular developer or team ownership over the shared code. Even if ownership is initially distributed, like an open source project, you should still assign a single individual to conduct code reviews of new contributions.

Testing

Testing is much easier on a set of individual services than within a large application. With a monolithic application, there will generally be more logic intertwined and linked dependencies between functions. The set of regression tests necessary to ensure that a change in the application hasn’t affected other parts of the functionality will be large. Your QA team will need to run through these regression tests before every major release. As the monolithic application grows in size, this regression testing overhead will increase. By breaking out the application into discrete services, code changes can be more isolated. A code change in a single service will then require only regression tests within that service’s codebase to be run.

Risks with Service Oriented Approach

One disadvantage of using a heavily service oriented approach is the processing overhead associated with making the remote calls to services from the application that provides the initial entry point into your back-end. A call to a service on a separate server tier will always take longer than running that code on the same server. Also, if a single service is down, that can impact performance of the entire back-end application, as incoming requests continually wait for a response from the slow service.

These risks can be mitigated.  First, the performance of each service should be measured and optimized as much as possible. Response times should be tracked over time, and flagged for refactoring when they cross a threshold. To mitigate the risk of cascading slowdowns stemming from a single service failure, the application receiving client requests should be designed with service failure logic. This would include a time-out on a service call and utilization of a default value in constructing the overall response to the client.

Team Size

With a small team, it is easier to maintain a single monolithic application. As we discussed, the main cost of maintaining a single application is the coordination overhead required when making design changes. With just a few engineers, this coordination overhead is minimal. As the team expands, this communication overhead will require more time. I have seen large teams working on a monolithic application even have to schedule meetings between sub-teams to vet proposed design changes.

I think a team size under 5 engineers can easily maintain a monolithic application.  As the team expands beyond this, it’s better to start dividing the monolithic application into separate code bases, deployed as services. This allows different developers (and eventually teams) to be assigned to each service, and their work to be isolated from the others.

Improving Engineer Retention

As the VP of Engineering at a rapidly growing software company, one of your primary responsibilities is to build your engineering team. Initially, this is primarily a recruiting function. As your team expands, though, you want to ensure that the engineers you have hired stay with the company. There is nothing more costly to organizational development than to have a new engineer leave after 6 months.

Granted, you shouldn’t expect engineers to stay on your team forever. I think a good target in a competitive hiring market is four years.  Beyond four years at a company, an engineer will start to feel that there is an opportunity cost to remaining. Outside recruiters will reinforce this perception, as they dangle other offers in front of the engineer and warn them about being pigeonholed.

The way I measure retention is based on the number of employees who voluntarily depart the company in a trailing 12 month period as a percentage of total employees. I don’t include terminations in this number. As an example, if you have 100 engineers on the team, and 50 quit in the last year, then you have a 50% retention rate. Using the four year engineer longevity target, then a good retention rate is 75%. Of course, you can strive for higher than this, and may do so occasionally. I passed the 80% retention rate during one 12 month period at a pre-IPO company in San Francisco. But, that isn’t sustainable over the long term in a tight hiring market.

In this post, I will offer some tactics to improve your engineering team’s retention levels.

Company Mission

First, we should consider engineers as employees. Think of why an employee would want to work for your company. Often it boils down to your company’s mission and how that contributes to the greater good. Is the company delivering a product or service that benefits other people or makes the world a better place? Is it something that employees can feel passionate about? Most companies have a compelling mission, or they wouldn’t be in business. You want to articulate your company’s mission in a way that excites your employees. Generally, the marketing or HR team will have this messaging worked out and available in presentable sound bytes. You should promote it to new hires and reinforce the message during group meetings.

As an example, at Zoosk, where I led engineering for several years, we went through this process. On the surface, Zoosk could be described as a dating app. The service allows individuals to discover and communicate with other members, often resulting in a date. However, after receiving notes of gratitude from married couples, we realized that our mission was broader than helping members get a date. Zoosk helps people find their life partner, perhaps the most significant event of their lives. Painted in this way, it was very easy to embrace the mission of the company. We reinforced that mission through internal company messaging. We even printed photos of happy couples with the text of their testimonials, and posted them on the office walls for all employees to see.

As another example, you may have noticed GE’s latest ad campaign in which a developer describes getting a job at GE, which will allow him to write software that makes hospitals run more efficiently.  This is contrasted with another developer’s job at “Zazzies”, where his app puts fruit hats on animals. The point is clear – that developers, as employees, want to work somewhere in which they feel their contribution has a positive impact.

Company Progress against Goals

Employees want to see that their company is moving forward. This measure of forward motion is usually represented as progress against goals. Engineers generally care about two types of goals – product usage and company financials.

Product usage goals reflect the number of users of a product and the activities they perform.  Examples might be user sessions per day, messages sent, photos uploaded, items purchased, etc. Engineers like to work on a product that is being used, as it gives meaning to their work. They also like to take ownership over the performance of their product – having frequent updates on performance against goals is very motivating.

Another significant set of metrics is the company’s financials.  These represent items like revenue, expenses, income, etc. Financial metrics are important because they provide feedback on whether the company is viable as a business. Engineers care about these because they want to work at a successful company. Also, they generally will have an equity stake in the company and want to ensure their equity will be worth more in the future.

For these reasons, I think it is important to share product usage metrics and financial results with the engineering team. I realize this requires a bit more trust than some companies are comfortable with, but the transparency goes a long way. If the company is exceeding all of its goals, that’s great. The team will be very excited. If the company is not meeting its goals, then you have the opportunity to provide perspective on why and how the company will fix that. Engineers are generally solution-oriented. By sharing the company’s challenges with them, I often find that they will make suggestions. Even if these aren’t good ideas, problem solving reflects positive thinking, which is good for morale.

You can share these product and financial metrics at team meetings. I usually summarize them at quarterly engineering all-hands. For timely updates, though, I will grab 5 minutes at the end of recurring team meetings, like sprint retrospectives or even stand-ups. This openness is very important and the engineers will appreciate the candor. I realize that a publicly traded company can’t disclose detailed financials regularly outside of the quarterly reporting cycle. If that is the case, at least try to spend some time reviewing the financial information after it is public. If your company is private, then I would share as much as possible, with the appropriate admonitions to not disclose the information outside the company.

Engineering Team Culture

Much is made about building engineering team culture in our industry. A lot of that focus is often on social factors, like having a “no a-holes” policy, providing snacks or encouraging participation in volunteer activities. I think building a strong engineering culture is more foundational. Like any team in a competitive space, the participants want to feel like they are “winning”. While we don’t have direct competitions between engineering teams at technology companies, a measure of “winning” can at least be translated into doing work that is noteworthy in the industry.

Having a “winning” culture means that the team is continually pushing itself to be better. They are looking for new ways to solve problems – better technology, improved process, skills development, etc. Working towards being better will ultimately lead to a sense of pride. And if your engineers feel proud of what they do, that will be a strong driver of culture.

Here are some ideas to build culture:

  • Continuous improvement. Encourage activities that focus on making improvements to your team’s processes, infrastructure and capabilities.  Conduct retrospectives.  Relentlessly ask if there is a better way to accomplish a task.
  • Encourage automation. The enemy of continuous improvement is complacency. One way to become complacent is to fill available development time with repetitive tasks. By automating those, time is freed up to find new ways to improve.
  • Promote yourself. You can reinforce the drive for continuous improvement by promoting the team’s accomplishments in your industry.  One output for this is through an engineering team blog. Most progressive Internet based companies have active and insightful engineering blogs. See UberSpotify and Airbnb for examples. The engineering blog provides a medium for team members to share technology accomplishments and best practices with the rest of the industry. This naturally drives improvement – if your team doesn’t have anything worth promoting to the industry, then it isn’t doing anything remarkable. Additionally, encourage your team members to speak at relevant conferences and allocate time for them to prepare. Landing a speaking engagement at a popular technology conference can generate a lot of visibility.
  • Support open source efforts. Make contributions to open source projects, or start a new one. This shows that your engineering team is advanced enough to improve the state of shared industry solutions.
  • Share wins. When an engineer accomplishes something notable, you should broadcast that to the rest of the team. This reinforces the positive behavior throughout the organization. Further, it makes the engineer feel good about what they accomplished and want to do more of it.

Professional Development

Probably the most important factor for an engineer’s job satisfaction is their professional development.  As an artisan, good software engineers value the ability to improve their skill set. This makes them more valuable in the workforce and satisfies their desire to expand their knowledge. Here are some ideas to provide engineers with professional development opportunities:

  • Training. Encourage your team members to find training opportunities that interest them and pay for these. Training can take the many forms – a 3-5 day intensive course on a particular technology or an evening class at a local university. Many of these programs are offered online.
  • Conferences. Similar to training, let your engineers attend relevant conferences. These are usually associated with the technology they focus on. Conference attendance may require travel. The company should pay for this. I usually budget for one conference for each engineer a year. I offer to pay for more than one conference if an engineer lands a speaker role.
  • Team rotations. Based on how your teams are structured, allow engineers to rotate onto different teams. This gives them exposure to new technologies and products. Rotations can be done for a short duration, like a quarter. You will want to manage the number of concurrent rotations so that productivity isn’t severely impacted by the associated ramp up.
  • Hackathons.  Conducting hackathons allows engineers to experiment with new technologies and try out product ideas. Most companies structure these as a 1-3 day event where engineers form teams and work on a new idea. I like to keep ideas focused on one of two areas – either a product feature or a productivity improvement. The teams should produce a working prototype. Ideas are then demo’ed at the end of the hackathon and prizes are awarded to the top teams. Many good product ideas will come out of these hackathons and the teams really enjoy the experience. I try to conduct at least 2 hackathons a year.
  • Independent work. Google’s 10% time is a well-known practice. I have cast this concept as “tech time” at other organizations. The idea is to allocate some regular time window for engineers to work on a technology problem outside of their normal sprint development tasks. I usually make this optional for engineers, but request that they document their independent work on a shared medium if they participate. Often this independent work allows the engineer to experiment with a new technology on their own schedule, resulting in more approaches for solving problems.
  • Feedback. Your company’s formal employee review process should include productive mechanisms to give feedback. I often try to go beyond this and ensure that managers give engineers feedback on their performance regularly. Weekly one-on-one’s are a good vehicle for this. Providing frequent, timely feedback to an engineer helps them improve.

Compensation

Finally, we get to compensation. Most good engineers view compensation as being important to their decision to join or stay with a company, but it isn’t the strongest decider. My experience is that engineers need to feel that they are compensated fairly relative to their peers in the industry, but will give a company latitude if the other retention criteria are strong. I am an advocate of paying engineers generously, as the leverage that a skilled, satisfied engineer can create is immense. If budget is exerting pressure on compensation levels, then ensure that the salary for an engineer is at least within 10% of the average for your industry and company size. Your HR team can get access to data showing compensation levels by job type and location. I have used Radford for this in the past. Don’t try to get away with significantly underpaying your engineers. They will figure it out, and leave for other higher-paying companies.

Technology Selection

A big part of your job as VP of Engineering will be to oversee technology selection for your team.  Some examples might be a programming language, an application framework or a particular data storage technology.  When your company is first starting, there are many of these decisions to make.  As your technology stack matures, these become less frequent.  However, your technology choices should be visited regularly to ensure they are still meeting your needs and perform better than alternatives.

Your role in technology selection is to establish the process and to shepherd the team through the steps.  This means determining who will be involved in the evaluation, establishing evaluation criteria, tracking the team’s progress and ultimately guiding the team to the final decision.  This blog post provides some guidance on how to structure a technology selection process.

Approach

Whether you are building your technology stack for the first time, or making an incremental addition, it’s a good idea to establish a process for your evaluation.  The process should be documented, ideally in a public forum or shared document, like a wiki.  That way, all engineers can understand the rationale for the technology choice.  This is particularly useful long after the choice has been made, so that team members can understand the requirements and assumptions that existed at that point in time.

Here are some items to consider as you construct your team’s evaluation process:

  • Problem Statement.  What is the problem you need to solve or capability that you are seeking to add with the technology choice? Try to narrow this down to a single statement, or at least a short list of capabilities to add.
  • Requirements. For your problem statement, what criteria would indicate a successful choice? If there are benchmarks that must be met, like queries per second, list those. For a technology choice associated with development, what outcomes might indicate success? Perhaps time to learn, developer productivity, compatibility with other frameworks, etc.
  • Timeline.  How much time are you allowing to make the technology decision?  If you have an immediate problem to solve, this may be short.  Ideally, you can allocate a couple of sprints (2-4 weeks) to the process.
  • Evaluation Plan.  What are the rough steps the team will take to evaluate the technology?
  • Participants.  Who will participate in the evaluation? Who will be part of the group making the final decision? Evaluation and decision making participants should be kept small.
  • Support. Do you need support from other departments, like marketing or analytics, in order to conduct your evaluation? This support should should be committed before the evaluation starts.

In terms of the process, most teams use an approach of starting with a list of all possible technology choices.  Then, they narrow the candidate list based on their criteria, trying to stack rank the choices.  Usually, teams will create a scoring function for each criteria, and then add up the totals.  They will work towards identifying the top three choices and discard the rest.  With the top three choices, they will conduct some sort of proof of concept where they build a basic prototype of a feature. The proof of concept provides real world feedback on how the technology works within their environment.

Other Possible Evaluation Criteria

While you want to confirm that the technology choice will solve the problem you have, or deliver the capabilities you need, it’s also important to consider some other criteria that can give an indication of its maintainability over time.  Here are some suggestions.

Knowledge within the Team

One big influence over your technology choice is how many of your team members have worked with the technology in the past.  If most of your team is adept in the technology choice, your time to being productive with it will be shorter.  If most of the team hasn’t worked with the technology, the ramp up will be longer.  You can train the team in the new technology.  That kind of training will be viewed as a professional development benefit, contributing to employee satisfaction and retention. Just keep in mind that training will take time, and extend the period to being productive.  For example, if you are choosing a programming language for server side processing, candidate technologies might be Java, Scala and Go. Check the proficiency of your existing engineers in these languages.  Most will be familiar with Java. Scala proficiency is likely lower.  Scala can have a steep learning curve.  So, you should expect a fair amount of time between making the choice and having the team be productive. However, Scala has some capabilities that might outweigh the time delay.  These considerations should be taken into account, as you make your choice.

Popularity and Momentum

For each technology, consider how “popular” it seems to be.  Popularity can be measured by interest and activity within the broader development community.  Some ways to gauge popularity:

  • Searches for the technology term on Google
  • Activity on developer sites, like StackOverflow.  Also, examine the support forums associated with the technology for an indication of maturity. Look at the posted questions from developers and see how many are answered.
  • Frequency of commits and releases.  For an open source technology, check the project’s site.  Look for active commits and regular releases.  If a commercial product, how often is the vendor providing updates?  Also, check the completeness of documentation.
  • Conferences focused on the technology. Have whole conferences been dedicated to the particular technology?  Do these seem to be well attended?
  • Podcasts.  Do a Google search for “[technology] podcast”.  If you get back a lot of results, check a few.  Are the podcasts getting updated regularly, or did they die off a few years ago?
  • Choices by other technology companies.  Check the engineering blogs for other companies in your space. What technologies are they using for the problem or capability?  Most engineering teams at progressive internet companies have an active blog.  See Uber or Spotify for examples.

Popularity is important because it is an indicator for how much support you can expect and how complete the implementation will be over time.  Also, developers want to broaden and maintain their exposure to relevant skills, so having an opportunity to work on a popular technology will help with recruiting and retention.

Momentum is another dimension to consider.  Momentum represents the change in a technology’s popularity over time.  Does popularity seem to be growing or fading? Also, momentum encompasses how long a technology has been in place.  That will generally be an indicator for how much longer you can expect it to be popular.  An established database technology, like MySQL, that has been popular for a long time, will likely continue to have support.

Tooling and Frameworks

Related to popularity is the state of tool and framework support for your technology choice.  If you are selecting a new programming language, how many IDE’s are available that include it?  Similarly, have development frameworks been built to apply the language to your use case?  A simple example is with PHP – there are tens, if not hundreds, of frameworks for building web applications with PHP.  Also, you should check for libraries available to integrate with the other technologies in your infrastructure.  For example, if you have Cassandra data stores, does your language choice have a client library that connects to Cassandra?

Costs

Vendor supplied or proprietary technologies will have an obvious cost associated with licensing and support fees.  However, don’t assume that that open source cost is zero. There can be many costs associated with an open source roll-out.  Some examples include getting support and training. For example, I have established support contracts for MySQL (with Percona) and Cassandra (DataStax) in the past.  These are necessary because your team will sometimes reach a point where they can’t solve a particular problem with an open source technology quickly.  Or, more simply, it’s more efficient to get support from the open source project committers, than dig through the source code.  It is also useful in the heat of troubleshooting a major issue to have the ability to reach out to one of these vendors for help. Many offer 24/7 support for outage resolution.  Finally, getting your engineers trained on a new technology can accelerate adoption.  These same support vendors often offer training classes, either at a central location or will come to your office. So, you should budget for support and training costs, and include them in your comparison.

The Decision

After you have collected all the information necessary for your technology choice and completed your proofs of concept, you and your team need to make a decision. My recommendation is to make this decision quickly, and not deliberate indefinitely.  While it is important to go through the process, you can change technologies in the future.  Granted, there will be a switching cost, but there is also an opportunity cost for your decision delay.  Keep in mind that the requirements for your problem statement will likely change over time, pushing you to re-evaluate the technology choice later.

Once you have made the decision, share it with the team.  Make the documentation associated with the evaluation process available to everyone.  As work with the technology progresses initially, keep track of your assumptions and ensure that the technology is delivering in  the ways you expected.  Patterns for utilizing the technology will begin to form during these initial implementations, so it’s important to get these initial patterns correct.  And, if the technology isn’t doing what you expected, examine that immediately.

« Older posts Newer posts »