Trunk-Based Development: The Key to Better and Faster Software

New businesses that master software delivery can reach valuations of 1 billion in a matter of months, disrupting established ecosystems. To do this, they have to have the capability to deliver software with quality and speed enabling them to rapidly scale valuable offerings based on experimentation. In that context, existing businesses are pressured to compete at the same pace or risk falling behind.

The rise of Agile methodologies and DevOps practices have improved the capability to deliver in faster cycles by leveraging techniques like Continuous Integration and Continuous Delivery (CI/CD). The main challenge with such practices is to ensure that quality requirements are met, as faster iterations increase the risk of defects and the accumulation of technical debt.

While the latest technologies play a critical role in accelerating the delivery of valuable software, efficient and effective software development processes are at the core of streamlined software activities. The development team must be able to perform rapid code changes and deliver them in small and reliable increments to support business goals.

Trunk-based development is a branching practice, identified as a core capability in the DORA research program, for elite performers delivering multiple times per day. This article compares trunk-based development with other branching strategies and how to implement it in your organization to deliver software with quality and speed.

What is trunk-based development?

Code versioning, for example, can quickly become a nightmare with an accumulation of small decisions and defects that pile up. When developers work in their own branch with a local version, the effort to merge multiple changes made in the meantime can take hours, if not days. All that time lost merging issues and the accidental complexity is a waste of effort for the company.

Trunk-based development is a software development methodology that focuses on keeping a single source of truth for the code repository, often referred to as the “trunk”, or “main” in the Git ecosystem. One important point is that this master software lane must always be considered as ready for deployment to production, without depending on other branches.

Trunk-based development is based on the following ingredients:

• Single source of truth in the “trunk” branch containing the latest code version that must be deployable to production.
• Short-lived branches used to continuously contribute to the central source of truth and work on an up-to-date version to reduce merge risks and defects.
• Small and frequent commits merging all code changes, even if features are not fully finished, to continuously ensure flawless integration with the rest of the codebase.
• Frequent communication between development teams as they often touch similar modules while performing an increased rate of code changes.
• Continuous integration and delivery to keep fast cycles of deployment as the whole team is collaborating in the same place.
• Automated testing to support fast feedback on code changes while continuously iterating through frequent commits and production deployments.
• Feature flags to keep the flexibility to activate or deactivate features without blocking the flow of iteration that is always moving forward.

The Lean Principle of limiting the work-in-progress is central to trunk-based development, helping to reduce waste and the number of defects on the production line. In software, its application means that small code changes are very frequently integrated into the main line and incorporated into the releasable version once they pass the quality requirements.

This paradigm encourages developers to commit their code regularly, resolving conflicts early on rather than letting them accumulate over time. By reducing the number of branches with a single source of truth with the latest code version, trunk-based development helps reduce the risk of merge conflicts, improves code quality, and accelerates software delivery.

Trunk-based development is not dependent on software architecture like modular monoliths or microservices — yet, it requires an alignment with the software system. For example, trunk-based development enforces frequent commits that must be reviewed quickly to keep up the fast flow of iteration, and is supported by practices like a release train and feature flags.

How trunk-based development differs from other practices

Gitflow, GitHub Flow, or Feature Branch structure team collaboration around different branches based on workflows like development versions, release, or hotfix. While developers feel safe working in a given branch, the release process is painful due to complex merging processes, slow testing, and multiple versions to manage at any given point in time.

Trunk-based development is different from other branching models with just-in-time integration and release of code changes on a single branch. In this model, software delivery constraints cannot be hidden as pending changes, making it the best way to streamline the software pipeline.

The key differences of trunk-based development lie in:

1. Number of branches: a single branch is used, whereas other branching models use multiple long-lived branches, such as development, feature, and release branches with different lifecycles.
2. Branching and merging: many short-lived branches that are quickly merged back into the trunk are used, whereas other branching models use long-lived branches merged back at specific points in time, and with complex rebase scenarios.
3. Collaboration and code sharing: developers are encouraged to commit code changes multiple times per day, and to resolve any conflicts early on, making it easier for teams to collaborate, while other practices support longer periods without merges.
4. Code quality and delivery: helps to reduce the risk of merge conflicts, and accelerate the delivery of small software increments, while alternatives lead to more complex merging procedures, with more code conflicts and less frequent delivery.

The choice of branching model depends on the specific needs of the team and context of the organization and its alignment with the development team. Google, Spotify, Etsy, Netflix are popular examples of organizations applying the practice, but other branching models may be more suitable for teams with a different organization and culture.

Furthermore, experience matters with trunk-based development. The State of DevOps Report (page 31) identified that new developers tend to make more mistakes and have lower performance, whereas developers with 16+ years of experience take full benefit of trunk-based development and perform better overall.

How to implement trunk-based development

Trunk-based development is like the cherry on the cake, as it requires multiple software practices to support its performance. Its implementation requires a gradual approach for developers to have the time to incorporate new practices in their work routines, and also for the team to grow in maturity with this paradigm.

It all starts by making it clear that the trunk or main branch is the only one considered as the source of truth for team collaboration. The following 4 steps support the implementation of mature trunk-based development practices to effectively harness its benefits. The key point is to align the software’s technical and organizational systems to work in concert.

1. Ad-hoc/initial maturity

The first level of implementation focuses on convincing the team of the benefits by making them understand the differences from other models they may be used to, and making small yet important changes in their working routines, i.e. to code and review more frequently.

The actions to drive are:

• Educating the team on the benefits of trunk-based development and how it differs from traditional branching models.
• Encouraging small changes and coaching the team to perform incremental changes to the codebase rather than large, monolithic changes.
• Setting up a synchronous code review process and making it systematic so that all changes are reviewed by at least one other developer before they are merged.

2. Repeatable maturity

A team with an accelerated rhythm of iteration needs specific elements of software automation and environments to keep changing code frequently in small increments with a good degree of confidence.

The following practices must be implemented for this to work:

• Continuous Integration and Continuous Delivery (CI/CD) tools to automate the build, test, versioning, and deployment of code.
• Setting up testing with a staging environment where changes can be tested before they are deployed to production, and where you will build your test automation infrastructure.
• Using feature flags that enable/disable new features in production to let developers commit unfinished work and be able to deactivate a feature if needed.

3. Managed maturity

Once a team masters the repeatable process of trunk-based development, the focus shifts to make the process more robust and scalable over-time. That’s where a clear release process that can be improved based on outcomes becomes necessary.

These elements become necessary at that stage:

• Establishing a release train process that includes testing, staging, and production environments ensuring software versioning with fixed deployment constraints.
• Setting up a rollback plan in case of issues with a release, as an emergency backup plan.
• Measuring and monitoring key metrics such as code quality, build and deploy times, and rollback success rates to continuously improve the process.

4. Scalable maturity

The various parts of the process must be delegated to dedicated roles when the codebase reaches a certain size. This final stage consists of supporting a continuous flow of changes with practices requiring an advanced degree of maturity.

The main practices relevant for this stage are:

• Establishing a platform team responsible for managing the CI/CD pipeline and ensuring that it is always running smoothly, and starting to provide advanced services.
• Using advanced testing techniques such as chaos engineering and continuous monitoring to proactively identify and fix potential issues.
• Continuously optimizing the process using data and feedback from the team to make improvements based on identified limiting factors.

Which technologies support trunk-based development?

The technology foundations of trunk-based development are similar to other branching strategies as the code still needs versioning, automated build, deployment and testing stages. The key differences for the trunk-based development tooling is to support a faster flow of iteration, requiring advanced capabilities.

The core tooling of trunk-based development is based on:

1. Version Control to manage the continuous versioning and keep the latest version in the shared codebase that can supported by tools like Git and Git version.
2. Continuous Integration to support the continuous build pipeline with unit test execution based on the multiple commits to the trunk. This can be achieved via Public Cloud providers like Google, Amazon, or AWS, or dedicated platforms like Bazel or Semaphore.
3. Continuous Delivery enables the deployment of the same artifact in multiple environments as well as providing progressive delivery capabilities. The tooling can be the same as for the CI, or a platform like Semaphore, GitLab, GitHub, or Harness.
4. Automated Testing will provide fast feedback loops so that the team can move forward quickly with the proposed changes. The key typologies of testing to start with are to run unitary tests early on as part of the CI build, to then complement them with non-regression tests on the functional and non-functional parts like security and performance. Unit testing tools are available for each programming language, whereas the other layers of testing require specific tooling like Cypress, Katalon, or Cerberus Testing.
5. Feature flags are critical for the team to perform multiple commits per day on unfinished work and progressively deploy finished features. Tools can be available as part of CD, but can also be done via configuration or specialized solutions like LaunchDarkly or Featureflags.io.

This tooling will support the progressive maturity of trunk-based development that needs to evolve to support the scalability of the practice. The increased flow of developers committing multiple times per day to the same codebase will drastically increase the number of builds, tests and deployments to be performed in a given timeframe.

As a result, if the integrated tooling takes too much time to perform the cycle of build to deployment, code changes will start to accumulate, increasing the work-in-progress and associated risks of defects, as is often the case with other branching strategies. Specific parts of the system require specific attention so they don’t become limiting factors.

The key potential choke points to address with the trunk-based development pipeline are:

• Merge requests rate to provide instant notification, alerts and automated delegation capabilities to avoid pending merge requests accumulating, which will translate into costly rework by developers afterwards.
• Build time execution with fast compilation and package creation to quickly have artifacts ready for deployment. Unit tests are a particular area to optimize with tools that can choose which part of the code requires testing using intelligent impact analysis.
• Automated test execution may become the biggest bottleneck of all once execution times change from seconds to hours. Use parallel testing and tooling to run critical tests first, and execution tests with less constraints afterwards.

This integrated tooling will set the right foundations, so that you can scale trunk-based development across your organization. As we have seen, experience matters and there are best-practices and recommendations that will accelerate the deployment of trunk-based development in a more structured way.

Overcoming the challenges of trunk-based development?

Trunk-based development is a structuring practice that impacts the day-to-day workflow of all developers and, in the end, the quality and speed of software delivery. Common challenges have been identified and can be overcome with good preparation and progressive changes in your organization.

The following challenges will be faced when migrating to a trunk-based development pipeline:

1. Adoption: changing our own habits takes time, and this is so for changing an organization. Start small, gradually, and focus on early adopters to show the value of habits like committing frequently and running tests locally.
2. Communication: coordination among team members has to be stronger as people interact much more on the same codebase. Clear communication channels and systematic processes will structure interactions and help with growing maturity.
3. Merge conflicts: developers face code conflicts early on with trunk-based development and must learn to promptly address them to avoid regressing to the old ways. Ensure code ownership, work allocation in different areas, and synchronous code reviews.
4. Feature management: multiple changes in a shared codebase that is continuously moving forward require ways to deactivate problematic bits of code. The team must gradually become proficient with feature flag implementation and removal.
5. Code quality: a codebase with frequent changes is harder to refactor, especially for structural changes. Organizations have to set clear code ownership, refactor in small increments, and rely on automated tests, intelligent refactoring, and code review.
6. Technical debt: the increased flow of iteration with multiple feature flags and more challenging code quality will accumulate technical debt much faster. That’s where the team must incorporate addressing technical debt as part of its standard workload, leveraging the pipeline with controlled limiting factors to keep improving quality at speed.

These challenges are continuous with trunk-based development and must be overcomed during its implementation, by leading by example, sharing the greater vision and forecasted difficulties, and defining a plan to gradually implement changes in the organization. As we have seen, trunk-based development is more than a just coding process–it’s a way of thinking about the coding process.

Conclusion

Trunk-based development can be a game-changer for organizations looking to deliver software with quality at speed. Similarly to the flow of lean factories, organizations able to deploy this practice can benefit from the fast flow of code changes, which can be quickly adapted and delivered to customers to support business goals.

Implementation challenges are fundamental points to consider before launching such an initiative. Development team maturity must be assessed first to determine if the practice would suit the current maturity and character of the organization. For instance, a fully decentralized development team with many juniors would not be a suitable context for implementing trunk-based development.

Trunk-based development represents the imperative to master the end-to-end software pipeline integration to support a fast flow of iterations. It is not only about mastering the code workflow between developers — it also requires organizational alignment with mature processes and tooling that can keep up with a growing codebase.

It represents the move from Software Engineering to Quality Engineering for better and faster software delivery with trunk-based development at its core.

Interested in trying it out? You can start implementing powerful trunk-based development today with Semaphore.

Originally published at https://semaphoreci.com on February 28, 2023.