Building a Roadmap to Improve Developer Productivity

More and more evidence is available that investing in developer productivity leads to better outcomes and higher retention. This type of investment provides clear business benefits such as reducing the overall cost of engineering and more consistent outcomes across planning cycles and teams.

However, improving developer productivity also makes developers happy. The work environment is more creative and collaborative and engineers feel like they are delivering more value more often, which is highly rewarding. Keeping developers happy keeps them in the flow more often and for longer. These developers then deliver their best work to customers and to each other, and stick around to continue to contribute.

Shippo highly values developer productivity and are building our Path to Green by forming a Developer Experience (DevEx) team. As part of building our DevEx roadmap, we implemented a Developer Experience survey to help us understand our current state and identify our growth opportunities.

A new team charter

We started a temporary virtual team (or V-team) initially inspired by a new templating framework that some of our engineers had begun to build out to support microservice development and shared libraries (look for a future blog post about our templating engine). This project needed focused effort to build out what our engineering teams needed including Kafka support as we move to an event-driven architecture, shared CICD support using ArgoCD, and Terraform and helm template standardization. However, once we got some of those building blocks out of the way, it became clear that we had more opportunities to improve engineering productivity in general and we needed to define a charter, a roadmap, and to figure out what exactly we were trying to accomplish.

When starting up a new team, there are a few core tasks that need to happen during the Forming and Storming stages to help the team know what to do and how to get started. These include answering the following questions:

• What are we trying to accomplish?
• How will we know that we have achieved that goal?
• What is the next best step toward achieving that goal?

The result from this exercise should be a mission statement, some KPIs, and a clear and focused MVP in the right direction.

We created an initial measurement plan for our team that used existing metrics we had available through LinearB integration, which includes out-of-the-box DORA metrics as well as other measurements that evaluate developer productivity by using GitHub and Jira metadata. When searching the internet for more guidance on DevEx best practices, we found several references to the Google Quarterly Developer Survey and how that team found a lot of value in asking key questions to their developers on a regular cadence.

It became clear that we needed our very own Shippo Developer Experience Survey to get a clear picture of how our engineering organization is functioning from the perspective of the engineers. And we didn’t want to just focus on one type of engineer. We wanted the perspective of front-end, back-end, data engineers, and SRE.

Drafting a survey

We started by reviewing some of the key issues that increase developer productivity, including cognitive load and effective feedback loops and started creating questions that identify when those issues are present or not present in our developers’ day-to-day. We created a wiki page with all the potential questions organized into key areas we wanted to measure. We then solicited feedback from folks who watch the DevEx slack channel as well as some members of our People Operations team that regularly measure Employee Engagement using surveys.

Once we got the questions edited and clear, we reduced the set to 10 key questions. We needed to keep the survey lean to ensure we had high participation and collected a representative sample of our organization. We want to be able to send it out either quarterly or every 4 months to get regular feedback, but we do not want to overwhelm engineers with too many questions. We formulated the questions as positive, desirable statements and asked engineers to rate how much they agree with this statement on a scale of 1 to 10. We also allowed write-in comments on each question so that we could collect additional feedback as needed.

The following are the questions that we included in our survey:

For all of these questions, think about your normal day-to-day experience across all code bases and answer 1-5 (1 = Strongly Disagree, 5 = Strongly Agree)

• It is easy to validate a change before raising a PR.
• It doesn’t take me long to get started on a new engineering project or task.
• I am often able to reuse or apply existing code to new projects or tasks.
• It is easy to debug and resolve production issues.
• Our documentation is helpful for understanding our code, architecture, and other infrastructure.
• I never have to make manual updates in production environments.
• I find the alerts my team receives to be helpful and actionable.
• I don't spend much of my time doing repetitive or manual maintenance tasks.
• It is easy and enjoyable to work on code at Shippo.
• What is one area of your developer experience that you would like to see receive more attention and improvement? (short answer)

Launching the survey

We chose to set up the survey in Lattice, which is the tool we use to run our Employee Engagement surveys. To prepare for sending out the survey, we wrote up a wiki describing what we were doing and why, including the questions we were planning to ask. We also wrote up a summary to include in the email notifications that would be sent with the survey to participants.

Finally, once we were ready to launch the survey, our team started the Slack Hype Machine, sending out regular reminders of the importance of this survey. We also sent DMs to managers who did not have high participation rates while the survey was running.

Making sense of the results

When the survey deadline was met and all the results were in, we started to analyze the results. We reviewed statistical analysis across different teams and other management trees (at director level), as well as based on employee tenure with the company, and title. We maintained anonymity by obfuscating results for teams that have under 5 responses. We also reviewed sentiment analysis for all short-answers and comments by question using our survey tool results.

We analyzed the results per director and manager for any patterns. So, for example, in the following screen, I could see that the engineers reporting into a given director were 45.8% more Positive in their comments in response to “I am often able to reuse or apply existing code to new projects or tasks.” and had a 9.1% lift in the score for that question as well.

Upon sharing the results with directors and managers, some of them took it upon themselves to follow up with our team to ask for guidance on how to use the data. Some managers and directors created action plans of their own for their organizations.

Beyond the numerical and sentiment analysis, we also took an exported CSV of the 191 comments engineers left and manually went through the comments and tagged them for given themes. These themes were derived when identifying patterns in the comments, but included such themes as “Missing Docs,” “Noisy Alerts”, and “Request Tracing.” This tagging was performed by engineers on the team who could understand the terminology and tools mentioned in the more terse or indirect comments. Also, as we performed the tagging, we sometimes modified or renamed the themes based on greater visibility into the full context of the patterns presenting themselves in the comments. The overall numerical scores were useful, but these comments were a goldmine of specific details of what tools or parts of our process was breaking down and what parts were working well.

One thing we noticed during this time is that folks were often confused by our terminology for the “It is easy to validate a change before raising a PR.” question so we decided that next time we will word it slightly differently to ensure people understand that it is talking about testing.

Deriving the DevEx roadmap

Once the results were organized and tagged, the DevEx team created a slide deck to analyze the themes and identify the priorities as we saw them based on this feedback and a small MVP that we could act on within the next quarter for each of these themes. We were having our in-person offsite for the team during this time which made this brainstorming and design work especially fun. We used a whiteboard and post-it notes and came up with a lot of ideas that we then filtered down to focused and manageable first steps as well as some long range vision for the future.

The following themes and action plans outlined what we could implement with a small team within 3 months.

Theme: People are enjoying component reuse and templating/core libraries

What we are doing to improve: 

• Continue to invest in templating and common libraries.
• Support migrating existing projects to latest template standards
• Support better publishing of Python libraries internally
• Rollout support for Github Actions in templates
• Ensure that our DORA metrics are accurate

Theme: We do not have the observability we expect

What we are doing to improve:

• Invest more in observability support including trace coverage across the application
• APM agent enabled on monolith
• Evaluate gaps in request traceability
• Lunch and Learn on OTEL best practices and New Relic usage
• Ensure that health checks are working and recording metrics for monolith
• Make a recommendation on go-forward strategy for synthetic monitoring

Theme: Our documentation is outdated, missing info, and hard to find

What we are doing to improve:

• Reduce Slack channels that are no longer needed
• Implement top level information architecture for our Confluence wiki
• Consolidate wiki spaces and archive outdated docs
• Add CODEOWNERS to all monolith code

Theme: We often modify things manually in production

What we are doing to improve:

• Improve and consolidate feature flagging capabilities across services
• Roll out Argo Workflows to enable interfaces for production access

Theme: Our local developer setup is impacting productivity

What we're doing to improve:

• Consolidate and update our local dev documentation
• Define long term roadmap for local dev environment improvements

Theme: We have environmental issues impacting our effectiveness

What we're doing to improve:

• Add alerting and team ownership in a shared integration test project for our QA environment
• Consolidate our config management to simplify managing and keeping different environments up to date

For each of these MVPs, we assigned a Directly Responsible Individual on the team to create stories or epics in Jira for the work and share back to the team during backlog refinement. Once we felt good about our plan, we felt ready to share results and our prospective next steps with the Engineering team at large.

We also created a dashboard for following along with progress for each of these epics and pinned it in our Slack channel.

Sharing the results

We took the slides that we had created for our internal team and stripped them down to a short 7 minute presentation to share the results at the next R&D All Hands meeting. We discussed the participation rate (74% Engineering ICs participated, with 191 comments submitted) and shared some of the themes and the DevEx team’s action items. We also discussed how each team could take on some of the growth opportunities within their own domain. For example, adding and updating CODEOWNERs files; cleaning up documentation; ensuring that alerts are actionable, etc.

After the meeting, we made the overall results and more detailed slides available via Google Drive and shared the focused results to managers and directors for their given organization. We also hosted a more in-depth conversation with the Platform Enablement team to discuss what this means for our organization at large and to elicit more feedback on action items.

Up next

One interesting take away from the survey is that it helped our team and others to feel more empowered to take on some of the shared tech debt in the organization as it brought light to these areas. So for example, when DevEx started assigning Code Owners to all areas of our monolith codebase, people already knew why this was necessary and were prepared to accept responsibility for more explicit code ownership. Additionally, we felt empowered to make requests on documentation improvements and archiving old documentation on our Wiki so that they can be as helpful as possible with our current environment.

Now that we have the survey baseline, we are ready to measure again early next year to see if we have made any improvements to the metrics given our investment in the initiatives we outlined. With this survey, we didn’t just kickstart our own DevEx team, but we also started to build greater awareness and buy-in for addressing these growth opportunities across team boundaries. As the quote goes, “Measurement is the first step that leads to control and eventually to improvement. If you can’t measure something, you can’t understand it. If you can’t understand it, you can’t control it. If you can’t control it, you can’t improve it.”