You Have a Shadow Pipeline Problem
Learn how to establish automated CI/CD controls to reveal what’s running in your pipelines and which configs don’t align with risk and compliance requirements.
Learn how to establish automated CI/CD controls to reveal what’s running in your pipelines and which configs don’t align with risk and compliance requirements.
Learn how to establish automated CI/CD controls to reveal what’s running in your pipelines and which configs don’t align with risk and compliance requirements.
Learn how to establish automated CI/CD controls to reveal what’s running in your pipelines and which configs don’t align with risk and compliance requirements.
Learn how to establish automated CI/CD controls to reveal what’s running in your pipelines and which configs don’t align with risk and compliance requirements.
.png)
CI/CD automation pipelines—like GitHub Actions, Jenkins, Build Kite, and so on—have been a massive accelerator for productivity, revenue, and engineer well-being, so it’s not surprising that we saw a 7-percentage-point increase in DevOps adoption—including CI/CD practices—across the software discipline during 2023.
“As of Q1 2023, only 16% of developers are not involved in any DevOps-related activities. This represents a 7 percentage point decrease from Q1 2022, indicating a continual increase in the adoption of practices that aim to increase an organisation’s ability to deliver software at high velocity.”
— State of Continuous Delivery Report 2023, CD Foundation
This systemic push for faster, more-repeatable testing and deployment comes from adding a host of tools and automation steps to your software supply chain: from pre-built Actions to open-source and commercial tools focused on software quality, security, infrastructure-as-code, testing, and many other DevOps needs. But CI/CD pipelines pose a challenge for Application Security and Product Security teams who need to ensure organization-level visibility into the software supply chain.
Shadow Pipelines Pose Compliance and Risk Challenges
As we discussed in Introducing CI/CD Security with Endor Labs, “shadow engineering” is one of the top CI/CD security challenges. We define it as follows:
- Shadow engineering: When developers use code repositories, tools or processes that are unsanctioned, untracked, or don't adhere to an organization's best practices.
Shadow engineering is present in many (if not all) organizations. This issue shouldn’t be laid entirely on developers, though—shadow engineering is the natural outcome of pressure for faster release velocity and a lack of guardrails and paved paths. Rather than trying to completely stop it, which turns into a frustrating game of whack-a-mole that damages relationships (we’ll talk more about this below), security teams need to understand how shadow engineering negatively impacts their initiatives so they can work around the problem. We see shadow engineering intersecting with security needs in two primary areas: compliance and risk.
Compliance Challenges
Various types of software analysis in the pipeline are commonly used to address compliance requirements, such as the regular vulnerability scanning required for PCI DSS and HIPAA compliance or used to support the SOC2’s CC7.1 criteria. If you rely on these types of security activities for compliance, it’s essential to be able to provide evidence that these activities are occurring reliably. This means you need to be able to show that the pipeline tools your organization is using to support compliance are deployed for in-scope repositories. If developers disable or bypass controls within pipelines—even if it’s just by mistake—that can put your compliance requirements in danger.
Risk Management Challenges
Beyond compliance, security teams need visibility into the full supply chain used to build and deploy software. That includes the tooling used within CI/CD pipelines. The 2020 SolarWinds compromise made it extremely clear to incident-response teams how important it is to be able to identify affected pipelines in the aftermath of a compromised tool, and to monitor corrected pipelines to prevent regression to vulnerable tool versions. A security team that can’t rapidly identify shadow pipelines and risky toolchains within known pipelines is likely to find itself with extended periods of vulnerability, and all of the stress, time, and opportunity cost associated with an extended response. And little is more frustrating than deploying a patched component only to have the vulnerability reappear months later when an engineering team accidentally reverts a patch while solving some other problem.
These problems are challenging even in very small organizations with only a handful of software repos. It’s possible for an application security engineer to routinely review and analyze pipelines, but it’s time-consuming and error-prone. And as organizations grow to dozens, hundreds, or thousands of repositories—often with multiple pipelines in each—it becomes intractable to solve with human review.
Controlling Your CI/CD Pipelines
One way to address shadow engineering and compliance challenges is to create a change review process and enforce it with preventive controls. This could be accomplished, for example, by establishing a CODEOWNERS file that requires a security reviewer for any changes to pipelines. This can be enforced through the controls in your code collaboration platform (for example, by configuring GitHub’s branch protection rules to be aware of CODEOWNERS constraints).
Unfortunately, this kind of preventative approach has some significant drawbacks. Not only does it require a lot of process overhead and application security resources, but it also undermines the advantages of engineer-owned pipelines in the first place. Slowing down engineers’ ability to make pipeline changes by turning an application security team into a bottleneck imposes a massive productivity tax on developers; and that’s no way to maintain the healthy relationships between security and engineering teams that are essential to the smooth operation of application security programs.
Responsive CI/CD Control Makes Friends and Saves Resources
A better model is to establish automated controls that give you clear insight into what’s running in your pipelines and which configurations don’t align with organizational requirements. This allows you to create a responsive process that surfaces critical issues promptly to developers, while providing a more complete risk picture to your security team.
A responsive control approach needs to do three things:
- Discover and identify pipeline components and configurations. Gain visibility to all the pipelines in your organization and identify information about their composition and configuration, in an automated and repeatable way.
- Quickly research and report on the discovered data. Collect visibility data in one place and make it easy to hunt down problems/noncompliance and provide compliance evidence on demand.
- Automate alerting and enforcement of critical policies. Make sure you’ll have proactive notification when there are critical issues related to your pipeline configuration (for example, when a required tool is removed or disabled), and that the path to surfacing those notifications to the engineers who can act on them is short and effective.
Discover and Identify
Your source code repository management systems have APIs that can allow you to enumerate your repositories and identify changes. Once you have this list of repositories, you can examine each repository’s files to determine which pipeline system(s) it is using. For example, if you see a `.github/workflows` directory containing YAML files, you know that the repo is configured for GitHub Actions; whereas something like a `.buildkite/pipeline.yaml` file will show you that BuildKite is in use.
Analyzing these various pipeline configuration files will help you identify tools in use inside the pipeline. Understanding the actions, reusable components/plugins, etc. will help you determine what tools are run under what circumstances, while allowing engineering teams to continue to own their pipelines. You can take advantage of shared and required pipelines to have much of the discovery work done when the repository’s pipeline runs. But you’ll also need a “supervisor” that can discover and analyze repositories even if the pipelines don’t run, or if your discovery tool gets removed from them.
Research and Report
Once you have a repeatable discovery process in place, you can normalize the data into a centralized dataset. SIEM tools can be used for this purpose, as can BI tools alongside traditional databases. Consistent data representation is important! You need to be able to ask meaningful questions of the dataset, like “which repositories have BuildKite pipelines?” and “which repositories have any pipeline that runs one or more of our approved SAST tools?”
Once you can pose these sorts of questions to your system, you can begin to develop and automate reporting that will serve as compliance evidence, and highlight issues that need to be addressed. You can use these reports to work with engineering teams. Depending on your organization's approach to engineering management, this might look like Jira tickets or might be handled as a periodic remediation project.
Alert and Enforce
Not every issue is critically important—but the ones that are need to be addressed rapidly. This means you need a good alerting system in place. Your data set is capable of highlighting important issues, so you’ll want to identify patterns for the most important things and arrange for proactive notifications. Ideally, you want those alerts to land where developers can act on them with a minimum of switching overhead, such as within the relevant pull request. This allows you to be truly responsive as issues arise, without introducing an unnecessary productivity tax on developers.
Alerts allow for two stages of enforcement. First, the ability to quickly detect and track correction of issues enables process-oriented enforcement of key organizational requirements. Second, mature and well-tested alerts open the possibility to have the most-critical alerts block releases—to “break the build”, in a way. While you have to be cautious not to introduce blockers that are too costly (for the same reasons the formal change-review process isn’t workable), the ability to create narrow rules that block releases when the risk justifies can be a powerful safety measure.
Get Pipeline Visibility with Endor Labs CI/CD
Building a responsive security control program for CI/CD that works effectively while minimizing the developer productivity task can be an overwhelming challenge. It’s a big lift to unify discovery tools, reporting tools, and integrating quality alerting and enforcement that help your organization keep focus on risks.
This is why Endor Labs CI/CD includes a powerful CI/CD discovery capability that simplifies and automates much of the process. We enable you to efficiently monitor which tools and reusable components are configured in pipelines, which CI systems are in use, and more. And this works both from a supervisory perspective — requiring no integration with project pipelines — as well as from within pipelines if you prefer. Since the capability is part of the Endor Labs scanner, enabling it once Endor Labs is deployed — whether via command-line, Pipeline plugins (like our GitHub Action), or our GitHub App — can be done without any additional Dev or DevOps resources.
CI/CD discovery connects seamlessly to Endor Labs’ powerful policy system, allowing you to have fine-grained, code-based policies (using the Rego language) to drive precision alerting and focused enforcement for your CI/CD toolchains. And as with all our products, Endor Labs CI/CD capabilities and data are accessible from the command-line and via our comprehensive ReSTful API.
Supervision with Endor Labs CI/CD
When run from outside your pipelines (such as with our GitHub App or our command-line scanner in a security supervision pipeline), Endor Labs enumerates your organization’s repositories and then begins scanning. A single pass of the scanner can perform multiple evaluations, including:
- CI/CD discovery: Discover pipelines and identify tooling within them
- Open source dependency management: Discover your open source libraries, surface risks within them, and provide context that helps you eliminate noise and focus on the most relevant issues
- Secret detection: Identify secrets that have leaked into your source repository, and validate whether they’re in use
- Repository security posture management (RSPM): Compare your repository configuration to pre-defined compliance suites (e.g. CIS Benchmark) or custom policies and identify gaps
During CI/CD discovery, we identify pipeline configuration files (typically YAML documents that follow various naming conventions), then interpret them to determine which reusable components (e.g. GitHub Actions, Jenkins plugins, GitLab includes, etc.), common tools (e.g. Semgrep, flake8, CodeQL, etc.), and container image definitions for runners are configured for each pipeline. Once these items are identified, we examine them to identify risks like reported vulnerabilities in them (or in their dependencies), excessive permission grants, and so on.
As with other types of evaluations, this data is stored for reporting and research purposes, but also sent to our policy engine where pre-built and custom Rego-language policies determine what issues to raise and actions to take. For example, you may have a policy that leaves a PR comment any time a PR makes a change that disables CodeQL in a pipeline. Or you could open a DEVOPS Jira ticket whenever a project adds a deprecated linter.
Pipeline Integration of Endor Labs CI/CD
Adding a scanning step to a project’s CI/CD pipeline — whether by individually integrating each repository or using a shared workflow pattern to require it for some or all of your projects — opens up the ability to enforce policy and detect changes as they’re made. This is as simple as adding a few lines of configuration to a required pipeline (by deploying our GitHub Action; using our templates for GitLab, CircleCI, Jenkins, and so on; or designing your own approach), and configuring central policies to enforce.
Integration opens two additional enforcement pathways:
- Failing a build step: Prevent a build from completing if critical tools are not in place (a.k.a. “break the build”)
- Warning developers: Leaving PR comments or providing other in-workflow notifications (CI logs, artifacts, etc, all depending on your workflows)
Warnings can be converted into build step failures with a single policy change, so you don’t need to re-engineer your workflows to alter behaviors.
Build a CI/CD Security Program with Endor Labs
In addition to CI/CD discovery, Endor Labs CI/CD includes:
- Repository Configuration Posture Management (RSPM): Detect repo misconfigurations, best practices, and risks with over 50 out-of-the-box policies.
- Artifact Signing: Ensure the authenticity of software artifacts, confirming their source and that they have not been tampered with.
To get started with Endor Labs, start a free 30-day trial or contact us to discuss your use cases.
