A big part of your job as VP of Engineering will be to oversee technology selection for your team.  Some examples might be a programming language, an application framework or a particular data storage technology.  When your company is first starting, there are many of these decisions to make.  As your technology stack matures, these become less frequent.  However, your technology choices should be visited regularly to ensure they are still meeting your needs and perform better than alternatives.

Your role in technology selection is to establish the process and to shepherd the team through the steps.  This means determining who will be involved in the evaluation, establishing evaluation criteria, tracking the team’s progress and ultimately guiding the team to the final decision.  This blog post provides some guidance on how to structure a technology selection process.

Approach

Whether you are building your technology stack for the first time, or making an incremental addition, it’s a good idea to establish a process for your evaluation.  The process should be documented, ideally in a public forum or shared document, like a wiki.  That way, all engineers can understand the rationale for the technology choice.  This is particularly useful long after the choice has been made, so that team members can understand the requirements and assumptions that existed at that point in time.

Here are some items to consider as you construct your team’s evaluation process:

  • Problem Statement.  What is the problem you need to solve or capability that you are seeking to add with the technology choice? Try to narrow this down to a single statement, or at least a short list of capabilities to add.
  • Requirements. For your problem statement, what criteria would indicate a successful choice? If there are benchmarks that must be met, like queries per second, list those. For a technology choice associated with development, what outcomes might indicate success? Perhaps time to learn, developer productivity, compatibility with other frameworks, etc.
  • Timeline.  How much time are you allowing to make the technology decision?  If you have an immediate problem to solve, this may be short.  Ideally, you can allocate a couple of sprints (2-4 weeks) to the process.
  • Evaluation Plan.  What are the rough steps the team will take to evaluate the technology?
  • Participants.  Who will participate in the evaluation? Who will be part of the group making the final decision? Evaluation and decision making participants should be kept small.
  • Support. Do you need support from other departments, like marketing or analytics, in order to conduct your evaluation? This support should should be committed before the evaluation starts.

In terms of the process, most teams use an approach of starting with a list of all possible technology choices.  Then, they narrow the candidate list based on their criteria, trying to stack rank the choices.  Usually, teams will create a scoring function for each criteria, and then add up the totals.  They will work towards identifying the top three choices and discard the rest.  With the top three choices, they will conduct some sort of proof of concept where they build a basic prototype of a feature. The proof of concept provides real world feedback on how the technology works within their environment.

Other Possible Evaluation Criteria

While you want to confirm that the technology choice will solve the problem you have, or deliver the capabilities you need, it’s also important to consider some other criteria that can give an indication of its maintainability over time.  Here are some suggestions.

Knowledge within the Team

One big influence over your technology choice is how many of your team members have worked with the technology in the past.  If most of your team is adept in the technology choice, your time to being productive with it will be shorter.  If most of the team hasn’t worked with the technology, the ramp up will be longer.  You can train the team in the new technology.  That kind of training will be viewed as a professional development benefit, contributing to employee satisfaction and retention. Just keep in mind that training will take time, and extend the period to being productive.  For example, if you are choosing a programming language for server side processing, candidate technologies might be Java, Scala and Go. Check the proficiency of your existing engineers in these languages.  Most will be familiar with Java. Scala proficiency is likely lower.  Scala can have a steep learning curve.  So, you should expect a fair amount of time between making the choice and having the team be productive. However, Scala has some capabilities that might outweigh the time delay.  These considerations should be taken into account, as you make your choice.

Popularity and Momentum

For each technology, consider how “popular” it seems to be.  Popularity can be measured by interest and activity within the broader development community.  Some ways to gauge popularity:

  • Searches for the technology term on Google
  • Activity on developer sites, like StackOverflow.  Also, examine the support forums associated with the technology for an indication of maturity. Look at the posted questions from developers and see how many are answered.
  • Frequency of commits and releases.  For an open source technology, check the project’s site.  Look for active commits and regular releases.  If a commercial product, how often is the vendor providing updates?  Also, check the completeness of documentation.
  • Conferences focused on the technology. Have whole conferences been dedicated to the particular technology?  Do these seem to be well attended?
  • Podcasts.  Do a Google search for “[technology] podcast”.  If you get back a lot of results, check a few.  Are the podcasts getting updated regularly, or did they die off a few years ago?
  • Choices by other technology companies.  Check the engineering blogs for other companies in your space. What technologies are they using for the problem or capability?  Most engineering teams at progressive internet companies have an active blog.  See Uber or Spotify for examples.

Popularity is important because it is an indicator for how much support you can expect and how complete the implementation will be over time.  Also, developers want to broaden and maintain their exposure to relevant skills, so having an opportunity to work on a popular technology will help with recruiting and retention.

Momentum is another dimension to consider.  Momentum represents the change in a technology’s popularity over time.  Does popularity seem to be growing or fading? Also, momentum encompasses how long a technology has been in place.  That will generally be an indicator for how much longer you can expect it to be popular.  An established database technology, like MySQL, that has been popular for a long time, will likely continue to have support.

Tooling and Frameworks

Related to popularity is the state of tool and framework support for your technology choice.  If you are selecting a new programming language, how many IDE’s are available that include it?  Similarly, have development frameworks been built to apply the language to your use case?  A simple example is with PHP – there are tens, if not hundreds, of frameworks for building web applications with PHP.  Also, you should check for libraries available to integrate with the other technologies in your infrastructure.  For example, if you have Cassandra data stores, does your language choice have a client library that connects to Cassandra?

Costs

Vendor supplied or proprietary technologies will have an obvious cost associated with licensing and support fees.  However, don’t assume that that open source cost is zero. There can be many costs associated with an open source roll-out.  Some examples include getting support and training. For example, I have established support contracts for MySQL (with Percona) and Cassandra (DataStax) in the past.  These are necessary because your team will sometimes reach a point where they can’t solve a particular problem with an open source technology quickly.  Or, more simply, it’s more efficient to get support from the open source project committers, than dig through the source code.  It is also useful in the heat of troubleshooting a major issue to have the ability to reach out to one of these vendors for help. Many offer 24/7 support for outage resolution.  Finally, getting your engineers trained on a new technology can accelerate adoption.  These same support vendors often offer training classes, either at a central location or will come to your office. So, you should budget for support and training costs, and include them in your comparison.

The Decision

After you have collected all the information necessary for your technology choice and completed your proofs of concept, you and your team need to make a decision. My recommendation is to make this decision quickly, and not deliberate indefinitely.  While it is important to go through the process, you can change technologies in the future.  Granted, there will be a switching cost, but there is also an opportunity cost for your decision delay.  Keep in mind that the requirements for your problem statement will likely change over time, pushing you to re-evaluate the technology choice later.

Once you have made the decision, share it with the team.  Make the documentation associated with the evaluation process available to everyone.  As work with the technology progresses initially, keep track of your assumptions and ensure that the technology is delivering in  the ways you expected.  Patterns for utilizing the technology will begin to form during these initial implementations, so it’s important to get these initial patterns correct.  And, if the technology isn’t doing what you expected, examine that immediately.