Editor’s Note: The following is an article written for and published in DZone’s 2024 Trends Report. The Modern DevOps Lifecycle: Changing CI/CD and Application Architectures.
Software supply chains (SSC) have become a prevalent topic in the software development world, and for good reason. As software development has matured, so has our understanding of the dependencies that can affect the security and legal status of our products. We only need to hear names like Log4Shell to remember how crippling a single vulnerability can be.
SSCs are the nexus of development and operations, and as we begin to take SSCs more seriously, it is important to establish an effective balance of responsibilities within our existing DevOps culture. In this article, we will look at how DevOps plays a key role in SSC Management (SSCM) and how we can effectively manage our SSCs with our existing DevOps structures.
Software supply chain
AND supply chain is a network of resources that are needed to procure products. In software, this means all software artifacts that our product depends on and all artifacts that we release, including:
- Binary files
- Configurations
- Scripts
- Licenses
There are generally three parts to SSC:
- Upstream – dependencies on which our product relies
- Build a system – the infrastructure used to build our product
- Downstream – the artifacts we publish
This conceptual SSC is illustrated below in Figure 1:
Picture 1. SSCs consist of upstream artifacts, building systems, and downstream artifacts
Image Source: Software Supply Chain Security, DZone Refcard
The first step in managing SSC is to answer some basic questions, including:
- What artifacts, including transitive ones, does our product use?
- What artifacts does our product publish?
- Is our building infrastructure safe?
- Are we checking our dependencies are reliable?
- Do we provide proven secure forms of trust for our downstream artifacts?
Answering these questions is no simple task and it quickly becomes apparent that both development and operations play a key role in supply chain management.
Integrating DevOps and Software Supply Chain Management
Managing an SSC can be a difficult task as there are many pieces to consider. Responsibility for these pieces of the puzzle is also distributed among different groups. For example, if a dependency is vulnerable, it will likely fall to development to upgrade and mitigate the dependency, but a change to our deployment infrastructure that needs to verify the reliability of our downstream artifacts will likely fall to operations.
We need to be careful against creating the silos that our DevOps culture has managed to reduce, but at the same time we need to recognize that development and operations are not interchangeable. In order to effectively manage SSC, we need to leverage each team’s strengths and apply them where they are needed, while maintaining visibility and trust between them.
Establishing this balance requires three parts:
- Visibility
- Input
- Responsibility
Visibility
Both development and operations must have access to the supply chain configuration.
As SSC for a product grows, both development and operations will need to see the chain. In the past, the build and deployment configuration would be in a completely different repository and probably require a completely different set of tools than the code itself. In order to manage SSC efficiently, we want our SSC configurations to be in the same area as our code. That way, both development and operations have access to the configurations, and just as importantly, both can see the changes the other makes to the configuration.
Some tips include:
- Keeping the repository neat and uncluttered so that configurations and scripts can be easily found
- Following established conventions so that everyone knows where to find scripts and configuration in any given repository
- Ensure that Software Bills of Materials (SBOMs) are published where development and operations can see them
Input
Both development and operations must be able to see and approve changes.
Coming from this shared visibility, it is crucial that both teams have input into the changes that will affect the SSC. Early on in SSCM, it becomes clear that both development and operations have common interests, but sometimes have conflicting ideas about how to achieve those goals.
For example, if our product has a vulnerability in a downstream artifact that makes our deployment vulnerable, it may not be feasible to simply remove that artifact from the deployment because it may contain code that is critical to the proper operation of our product. Instead, development and operations must work together (and both provide input) to see if the vulnerability can be fixed by development or remedied by operations.
Some tips include:
- Add both development and operations staff as pull request (PR) approvers
- Fostering a close working relationship between development and operations staff so that each can see from the other’s perspective
- Ensuring that development and operations are continuously in sync (“on the same page”)
Responsibility
Both development and operations must be responsible for SSC.
Ultimately, accountability to development and operations is critical to growing trust and making DevOps a success. Simply, “What do you think?” can go a long way in building and maintaining trust and relationships between development and operations teams. It is equally important that both teams also take joint responsibility for the results of the SSC.
For example, if a vulnerability makes it into production, development should ask themselves, “How did we allow this vulnerability to get into our code and be implemented?” And operations should be asking, “How did we allow a vulnerable artifact to get into the production environment?” This shared responsibility ensures that both teams have the mindset of “What can we do to secure our SSC?” and not point fingers at each other.
Some tips include:
- Fostering an environment of personal responsibility
- Ensuring that development and operations are praised for successes in the SSC
- Both development and operations are expected to solve SSC problems that arise (not just one or the other).
DevOps tips for effective software supply chains
Visibility, input and accountability are the cornerstones of effective SSCM, especially in a DevOps culture, but sometimes these concepts can be too abstract. Below are three specific tips that development and operations can use to implement these concepts on a daily basis.
1. Place the code and configuration in the same repository
A key, though sometimes overlooked, aspect of visibility is adding configuration to a location that both development and operations have access to. The ideal place is the repository that contains the code. Not only does this ensure that both development and operations can see the changes that will affect the SSC for our product, but it also ensures that both teams don’t have to “hunt” for configurations elsewhere or use a different set of tools. The same tools that development and operations already use to develop and deploy our product are the same tools that are used to manage SSC.
2. Use pull requests whenever possible
There are usually two ways to make a change to a repository: commit to the repository directly, or create a PR and merge the PR. Creating a PR has the advantage of adding approvers who can approve or reject the PR. This can be a very useful tool when one team is making a change and seeking the other’s input. For example, if the development team wants to add a new test phase to the build pipeline, it is important that the operations team provides input. This not only creates buy-in from both teams, but also builds trust. Furthermore, it creates accountability for both teams since the development team created the change and the operations team approved it (or vice versa).
3. Automate as much as possible
Manual steps can inadvertently create artificial barriers between teams. For example, if there is a manual step in the deployment process that only operations staff know how to perform, the visibility of the development team is greatly reduced. Instead, we should automate as many development and operational steps as possible. This not only increases team agility, but also documents the process and allows both development and operations to request changes and scrutinize the process.
Conclusion
SSCM has become a key part of software development, and for good reason. An ineffective management process can lead to an ineffective supply chain, which can result in significant financial and reputational damage to both the product and the company. Establishing visibility, input and accountability for development and operations teams not only ensures the security of our SSCs, but also provides a key opportunity to strengthen our DevOps culture and build trust between our development and operations teams.
This is an excerpt from DZone’s 2024 Trends Report.
The Modern DevOps Lifecycle: Changing CI/CD and Application Architectures.For more:
Read the Report