Building a Successful Developer Platform: 3 Lessons from Toyota’s Production System

Gartner recently predicted that “80% of software engineering organizations will establish platform teams by 2026, and 75% of those will include developer self-service portals”. Such platforms promise to enhance productivity and streamline the software development process by combining cloud infrastructure, middleware software, and developer tools to create accelerated “Golden Paths” to production.

In addition to my own discussions with dozens of business leaders this year, there are strong signals that Gartner might just be right. In the latest Gartner Software Engineering Hype Cycle, we saw platform engineering enter the “Peak of Inflated Expectations” and there was a whopping 350% rise in attendance at PlatformCon 2023, compared to the previous year.

It’s safe to say then that many infrastructure teams are actively engaged in building an Internal Developer Platform (IDP) for their business. My concern though, is that most of those efforts will fail… not because of a lack of capability or technology, but because they’ve neglected the most important part … developers.

In this article, we’ll look at the renowned Toyota Production System to discover how its creators combined people, process, and technology to build sustainable lean pipelines. Although you may recognise terms like “Kanban” or “Kaizen”, in this article we’ll explore the what platform teams can learn from the mysterious methods of “Genchi Genbutsu”, “Nemawashi”, and “Poka-Yoke”.

The Toyota Way

The Toyota Production System (TPS) is a manufacturing methodology that revolutionized the automotive industry. It was developed by Toyota in the 1940s as a response to the post-World War II economic challenges in Japan. TPS revolutionized manufacturing by introducing innovative concepts like Just-in-Time production and continuous improvement through Kaizen.

TPS became the foundation of lean manufacturing, inspiring companies globally to adopt its principles to improve efficiency, reduce waste, and enhance quality. Its principles were adapted and applied to software engineering, giving rise to concepts like Agile methodologies and Lean software development. Organizations found adoption delivered improved efficiency, reduced waste, enhanced quality, and increased collaboration among teams.

Most critically, TPS places a strong emphasis on people and their involvement in the manufacturing process. By empowering those involved in the process, Toyota was able to build a culture of reliability, quality, and continuous improvement. The influence of TPS on modern software development has transformed the way projects are managed, leading to more efficient and customer-centric software delivery.

Build The Right Thing with Genchi Genbutsu

The most common mistake I see when working with organisations interested in building an Internal Developer Platform is that they fail to involve any developers in its design or build. The problem is that many infrastructure teams believe their own knowledge of software development is sufficient. This bias can be dangerous and does nothing to help build a culture of collaboration.

To understand how we can add value to developers, we need to understand the problems they face on a day-to-day basis by immersing ourselves in their workflow. Genchi Genbutsu, which translates to “go and see for yourself,” is a practice that involves physically going to the location or process where the problem exists. In a manufacturing context, imagine how many long-winded conversations can be avoided by everyone getting out of their seats and visiting the factory floor.

As platform engineers, we tend to think about software as something in a code repository, ready to be packaged and deployed to production. In reality, developers have a read-write-run loop that they will iterate through on their own workstation many times before code ever reaches a repository. It’s at this stage that many platform teams learn that their automation efforts could be best served creating clean development environments rather than trimming a few minutes from production deployment scripts.

By immersing ourselves in the developer’s workflow, we can gain insights into the tools and technologies they use, the challenges they face, and the areas where we can help. This understanding allows us to prioritize the right features and build a developer platform that truly meets their needs.

Understand the Context through Nemawashi

Understanding developers is important, but building software takes collaboration across many different teams. Nemawashi, meaning to dig or to work around the roots, is the process of building consensus and laying the groundwork for decision-making by involving all relevant stakeholders. During Nemawashi, a company seeks the opinion of employees to ensure everyone is included. Just as strong roots provide stability and support to a tree, nemawashi involves building a solid foundation of trust and collaboration among stakeholders before taking action.

To build something that benefits everyone, we must understand the viewpoints of all stakeholders. One effective way to do this is through a path-to-production workshop, where all relevant parties come together to focus on the problem at hand. Developers can share their experiences, pain points, and requirements. Infrastructure teams can provide insights into scalability, security, and performance considerations. Product managers can gather feedback from users and define the overall vision for the platform. By involving all stakeholders, we can ensure that the developer platform meets the needs of everyone involved.

Armed with a box of sticky notes and an empty wall, the group builds a timeline of every step in the process, from identifying a new requirement through having that feature implemented and in operational support. Next, the group identifies the teams responsible for each of those steps. Finally, the group can estimate how much work is associated with each task, and the average wait time as a task moves from one team to the next.

A workshop like this is a great way to bring teams together and help them build a picture of the obstacle course of processes, team handoffs, and approval steps each new feature must navigate as it progresses from an idea to a funded project and from code on a developer’s laptop to various testing environments, until it eventually arrives in production.

Approval steps in particular are often an eye-opening experience for the group because a few minutes of review can easily add days or weeks to the overall delivery timeline due to wait times. Likewise, tech teams might find that their automation efforts could be best served creating clean development environments rather than trimming a few minutes of runtime from production deployment scripts.

In addition to the path-to-prod workshop, continuous feedback loops are essential to understand the evolving needs of developers. Regular meetings, open channels of communication, and feedback mechanisms help us stay connected with the developer community and adapt our platform to their changing requirements.

Poka-Yoke led Developer Experience

Once we know what we want to build, it’s equally important to consider how our users will interact with it. Poka-Yoke, which means “mistake-proofing”, is any part of a process that helps people avoid (yokeru) mistakes (poka). For example, a cable connector that has an insert so that it can only be inserted one way. This proactive approach to user-centric design improves efficiency by reducing mistakes, but also makes the process more approachable to newcomers.

When we build an Internal Developer Platform (IDP), we want our users to have a positive experience by helping them to avoid mistakes. When assessing a product, customers want to achieve simple things quickly, and know that complex things are possible. Great documentation is the primary tool that we have available to demonstrate those, but unfortunately, is usually low down on the list of priorities for an infrastructure engineering team. When documentation does exist, it’s often too technical, incomplete, difficult to read, or out of date.

I would encourage all teams to write a specific “Getting Started” guide written from the perspective with no prior knowledge of the platform. What does a developer need to know to launch a simple “hello world” application? If you can get new users up and running with something quickly, then they will generally be more willing to invest time later to research how to achieve more complex things. Inversely, if it’s painful to do something simple, most will assume doing something complex is just impossible. It’s a great idea to have a peer-review process, particularly for any how-to documentation — it’s so easy to miss a step that might feel obvious. Because software changes quickly too, I’d also suggest checking documentation is still correct at least each quarter.

Summary

Technology is exciting, but people should always be at the core of everything we do. Toyota has built a culture that revolves around their technology, and we can learn valuable lessons from their approach. By including developers in the platform engineering process, understanding the context and viewpoints of all stakeholders, and building an intuitive platform, we can create a successful developer platform that benefits everyone involved.

In conclusion, platform engineering teams have much to gain from the principles of the Toyota production system. By adopting practices such as Genchi Genbutsu, Nemawashi, and Poka-Yoke, we can ensure that our developer platforms are built to meet the needs of developers and deliver value to the organization. Building a successful developer platform requires collaboration, empathy, and a deep understanding of the developer’s workflow. By following the lessons from Toyota, we can create a platform that developers are excited to use and that truly enhances their productivity.

Originally published at https://semaphoreci.com on February 7, 2024.