This blog is by a member of the Starburst security team, Alex Olea, DevSecOps Engineer.

What is DevSecOps? I describe it as a mishmash between cloud security, CI security, and application security (AppSec) - which is essentially my role at Starburst. Most of my experience has been in the startup world, and I’ve found most startups have the same challenge: Small security teams that require a lot of focus to make sure we can be effective. And if you’re doing DevSecOps (or AppSec) at a startup, you get the exciting opportunity not just to run a program, but to design that program. But it’s not total greenfield— although startups aren’t usually old enough to have serious technical debt, you’re still going to run into stuff that was spun up and abandoned or tools that aren’t what you might have chosen. So how do you get started?

Build Trust with Developers

My main stakeholders are our developers: their success is my success, and our overall success translates to customer satisfaction. But often there isn’t a lot of communication between the security team and the development groups. Security becomes this spooky ghost that only pops up to give developers more work or complain about something. That’s just not effective. 

My first trust-building step is transparency: Whatever we're up to, I'm going to tell you. And whatever the developers are up to, I'm gonna relay back to the team so that there are no surprises and we can keep an open line of communication. 

Second, is being a contributor. That doesn't have to mean I’m writing entire Typescript classes or modifying their stack. When I see something I can fix or contribute to, like a README, I do it. 

This lets them know I'm here to be an active member of the team. Security is an ally, and we like to contribute.

And third, I embed myself in the development teams. For example, at Starburst, I’m embedded in the cloud infrastructure team. This is crucial so that security doesn’t become a back office function. 

One last piece of advice for the early days: Start off being a little lenient on AppSec practices instead of immediately ripping out old tools and implementing new policies. This sets me off on the right foot of security partner instead of security police. As I start to get to know the team and their processes, it becomes apparent where I can introduce improvements, like triage and remediation processes. With their trust, we can work together to tighten things up and mature the program.

A Typical Day 

My typical day starts with catching up on Slack messages and email. Then I check CNAPP alerts to see if there’s anything requiring a follow up. Sometimes I check GitHub Actions to make sure things are still running smoothly and the scans are performing as expected. For example, I might catch a regression in the number of critical vulnerabilities on a package or dependency. 

Once I’m done catching up, I have regular touchpoints with security and engineering partners. I attend the cloud infrastructure scrum meetings (3x/week) and have a handful of 1:1s with software engineers and security team members, and some vendor meetings where I provide product feedback. We also have a weekly meeting with one of our engineering leaders — we call it a security party — where we exchange concerns and talk through rollouts.

Contributing to Developer Productivity

Developer productivity means empowering our developers to make great decisions around infrastructure and application security. Some ways DevSecOps helps are:

• Implementing secure defaults (e.g. in our cloud, “golden paths”)
• Enabling secure software development practices (e.g. static code analysis to catch common issues)
• Making sure they have relevant, high confidence data (e.g. Endor Labs provides call graph analysis for open source software dependencies)
• Educating developers on what a tool is for and engaging them in the implementation

Measuring AppSec Success 

When it comes to measuring our success, of course we use metrics to make sure we’re on track, are maintaining the program, and making progress towards the overall goals set by our chief security officer and engineering leadership. For example, we care about the total number of reachable vulnerabilities, grouped by severity— Endor Labs provides this information.

Our commonly tracked metrics are often tied to trends:

• Are the numbers of new vulnerabilities going down? 
• Are we on track to meet vulnerability remediation SLAs? 
• Are the number of reachable vulnerabilities trending downwards? 
• How many are being introduced over some period of time?

A less straightforward measurement approach is to listen to the developers. Developers are diverse, and each has a different definition for productivity and success. What helps one developer succeed may not be effective or useful for another. Feedback should be gathered continuously, and this informs what to focus on for a specific set of developers, or a specific product.

Redefining an AppSec Program

When I started at Starburst, we already had several tools in place for AppSec, including SAST and SCA. While it all worked, it hadn’t fallen under anyone’s specific ownership, so I was the first official AppSec program owner. I had the opportunity to evaluate the program and our stack, and at the top of the list was replacing our SCA tool, Rezillion. 

Problem #1: No Data Behind False Positives

Rezillion had a major issue: It marked findings as false positives without providing a reason. Understanding the reason behind a false positive marking is absolutely critical for two reasons. First, because we needed to have trust in the tool’s analysis. And second, because our customers are very invested in the security of our product. They routinely scan our on-prem product for vulnerabilities and ask us about unremediated findings. It’s not acceptable to have no explanation for a false positive marking, so we either had to manually research findings or make remediations without knowing whether they were actually risks.

Problem #2: Transitive Dependencies

Because our customers scan our product, there was a second concern: What if customers discovered open source dependencies that our SCA tool couldn’t find? Unfortunately sometimes this happened, and more often than not, they were discovering transitive dependencies— those we didn’t explicitly select but are being called by our direct dependencies. Most vulnerabilities appear in transitive dependencies so we had a compounding problem. If we didn’t have an accurate dependency inventory, then we were missing vulnerabilities that could put our product (and our customers) at risk. As with any findings, we need to be able to triage, prioritize, and remediate issues discovered in transitives.

Problem #3: No Pre-Deployment Scanning

We’re early in our improvements of our AppSec program, and one of my priorities is to change where scanning happens. A lot of AppSec tools, Rezillion included, are limited to scanning post-deployment— we do a release, get a .jar file, scan the file, and then find problems. This pattern is “anti-Shift Left,” a bolt-on that doesn’t keep us moving fast. When problems are discovered after the application is deployed, it takes extra time and resources to trace problems back to the source and build in time to address the vulnerabilities.

Requirements for a New SCA Tool

It’s always my goal to build a best-in-breed security program, so I set about gathering requirements and researching SCA capabilities. Given the experience with Rezillion, it’s no surprise that we needed a tool that excelled in discovering and prioritizing risk. 

We identified three main requirements:

• Developer experience: One good thing about Rezillion was it was easy to use, so it trained our developers to have a good attitude towards an SCA tool, and therefore use it! Our new SCA tool needed to continue prioritizing developer experience to ensure adoption and effectiveness.
• Function-level reachability analysis: This type of reachability is the most accurate way to determine whether a vulnerability is exploitable. We would use this to deprioritize false positives, giving us confidence that the tool is giving accurate and justifiable results. The tool we selected had to support reachability for all types of dependencies (direct and transitive) and also support our main languages (Java and Typescript required to start, and C++ in the future).
• Pre-deployment scanning and preventative controls: Being able to scan at pre-deployment will let us catch problems at the time of build, before they get into production and cost us even more time. And with preventative controls, we can block really risky dependencies from ever being selected. 

Testing and Implementing Endor Labs in CI Pipelines

We evaluated several SCA tools, including Endor Labs. Because Endor Labs offers a free trial, it was easy to give it a try— which was extra important to us since we were on a deadline. Endor Labs was very easy to integrate during the testing process because we could choose between the CLI tool and a GitHub app based on what was the best fit for our pipeline (we chose the CLI). The documentation is outstanding (we should expect their level of quality from every vendor) and it even included monorepo docs. We have one particular monorepo that’s very large and takes a long time to scan, and the docs contained great advice for shortening those scan times.

When it comes to integrating new tooling into our CI pipelines, I focus on making it very slow, very intentional, and being very transparent. For example, when we first integrated with Endor Labs it was an optional check. This is because if a new workflow isn't stable (for any reason, including me) I don't want it to hinder our developers. Instead we say Here’s an optional check that you can use or ignore. The rollout has gone very smoothly and we’ve had developers actively interested in turning on capabilities. For example we started PR comments as optional and one of the developers actually requested we turn on PR comments for a very specific use case. That’s how you know you’re gaining traction! Put it on their radar, they'll notice it's there, and they notice it works. Next they want that additional data point because it helps them triage and do their sprint planning. That’s what success looks like.

Endor Labs is integrated into the CI/CD pipeline for multiple repositories, so that we have an automated, continuous scanning process that keeps our results up to date. We do PR level scanning and scans of the repository. We’re beginning to experiment with policies and are working closely with the development teams for each product/repository so that the experience is seamless. I’m always surprised by how easy it is to get up and running with cloud native tooling. With Endor Labs, everything was ready to go: a CLI tool, a GitHub Action, and a GitHub app are all readily available. Members of the team, outside of customer support, were ready and able to help whenever we ran into issues in one of our many Java / Maven repositories. We were able to integrate Endor into our CI/CD pipelines without any negative impact to our development processes or experience. 

Results with Endor Labs

Endor Labs is scrappy company that has left me with the impression that they will do what it takes to see their customers succeed. For software composition and reachability analysis, it was difficult to find a competing product in the current market that is as fully featured as their platform. They place a big emphasis on methodology (and have SMEs that write about this) and are also capable of performing reachability analysis on transitive dependencies, which was a big selling point for us.

The most important improvement we’ve experienced is how quickly and accurately we can respond to customer feedback around perceived product vulnerabilities and questions. We use Endor Labs to determine if a vulnerability is a false positive (e.g. is it unreachable), triage product vulnerabilities, and prioritize issues for remediation. We’ve been measuring progress by tracking the reduction in vulnerabilities across all severity levels via the Endor Labs platform. Before using Endor Labs, it could have taken a week for Governance and Risk Compliance (GRC) to respond to queries since manual research (e.g. on reachability) was required. Now when a customer has a question about a false positive or transitive dependency, GRC can provide detailed information within just 1-2 days, effectively setting their minds at ease. 

Endor Labs is providing a 98.3% noise reduction on our SCA findings.
- Alex Olea, DevSecOps Engineer at Starburst

I’ll summarize what I like best in three points:

• Best-of-Breed Reachability: Endor Labs performs function-level reachability analysis, so we now have confidence in the accuracy of findings: Manual research has been greatly reduced. 
• Transitive Dependencies: This is a subset of reachability, but because most SCA tools can’t perform reachability analysis on transitive dependencies, this gets called out separately. Endor Labs offers it at the same level as direct dependencies, and for our required languages, ensuring we get accurate dependency inventories and risk assessment.
• Expertise and Support: The team takes pride in what they do and can talk extensively about SCA. This is also evident in Endor Labs’s blog posts and published articles. They’re always willing to answer questions and help us understand how to get the most out of the product.

Endor Labs is doing reachability analysis on transitive dependencies which is really important to us and a huge deciding factor in our comparison to another vendor who unfortunately didn't have it. 
- Alex Olea, DevSecOps Engineer at Starburst

The Future of AppSec at Starburst

There’s this unspoken rule of “never invite developers into an AppSec platform,” or that in a good program, a developer should never have to access the platform, but we’re breaking that rule with Endor Labs. Our developers actually want to see the aggregated results for use as a datapoint to prioritize product development and remediation planning, so we’re letting them in the platform. We’re interested to see how this impacts their effectiveness.

As developers gain confidence in Endor Labs, I’ll start to implement preventative controls: High-confidence policies for the things we absolutely can’t introduce to our codebase. For example, we could block a PR that contains a reachable, critical/high severity vulnerability with a high probability of exploit (EPSS). It will take some time to decide how we want to approach this from a governance perspective, but to start I’ve turned on the no-brainers like valid secret detection.

We’re also interested in exploring Upgrade Impact Analysis. With Java, and Maven in particular, some of those upgrades can be a monster in terms of complexity. But if I could tell the development team that I have a high degree of confidence that they could Bump up this version of this packagin this palm dot Xml file and it’s not going to break anything, that would be awesome.

About the Author

Alex Olea is a DevSecOps Engineer at Starburst where he was the first DevSecOps/AppSec hire. Focused on integrating security into the software development lifecycle, he enjoys automating security processes, remediating vulnerabilities in multi-cloud environments, and fortifying CI/CD pipelines. Alex is formerly of New York University, Weill Cornell Medicine, and Gro Intelligence.