Jellyfish’s Data-Driven Security Program
Learn how Jellyfish’s security team uses a data-driven approach to risk management and the role SCA plays in their strategy.
Learn how Jellyfish’s security team uses a data-driven approach to risk management and the role SCA plays in their strategy.
Learn how Jellyfish’s security team uses a data-driven approach to risk management and the role SCA plays in their strategy.
Learn how Jellyfish’s security team uses a data-driven approach to risk management and the role SCA plays in their strategy.
Learn how Jellyfish’s security team uses a data-driven approach to risk management and the role SCA plays in their strategy.
.png)
This blog is by two members of the Jellyfish security team, James Kirk (Head of Security and Privacy) and Josiah Bruner (Sr. Security Engineer).
It can be a daunting task to create a company’s first security team. The response from an engineering-focused company can fall on a spectrum of skepticism to hostility, and often feels like it’s “security against the world.” But when we set out to start Jellyfish’s first security team, we were surprised in the best possible way: It already had the foundations of a security culture. The engineering leaders already embraced “shift left” security as much as possible, meaning it was already woven into the development stage instead of a bolt-on after deployment. This culture created an environment where engineering was receptive to our nascent security team’s advice. For example, they look to us for input on product development and threat modeling, and they’re very supportive of the incident response process. While security can be seen as a necessary burden when the company is beholden to regulations or if it’s a security product, Jellyfish has been unique in our experience because engineers embrace security as the right thing to do for our customers.
Measuring the Effectiveness of Security
Jellyfish is an engineering management platform founded to rewrite the way the industry thinks about software engineering and the impact it can have on a business. We do this by enabling engineering leaders to measure, improve, and communicate the investment and effectiveness of their team's efforts. We think this makes our company unique because we use the same techniques to measure our own effectiveness.
There is no such thing as 100% secure software, so we don’t find it useful to measure our team’s effectiveness based on the number of vulnerabilities or even security incidents. As we saw recently with Crowdstrike, incidents related to 3rd party dependencies are mostly out of our hands. Instead, we’re more focused on measuring our ability to detect and respond in a swift manner, and our ability/time-to-mitigate the vulnerability or incident. With this in mind, our biggest demonstrator of success is an appropriate level of risk that matches both our expectations and our customers’ expectations. To achieve this, we build tangible, evidence-based models that predict how much loss to the company can be expected in the next five years. We constantly update these predictions based on scan findings, redoing threat models, and updating attack trees. As our board becomes more interested in cybersecurity and related risks to the business, it becomes even more critical that our risk predictions are accurate (more on this later).
We also measure engagement with engineering and impact on developer experience:
- Engagement with engineering: In this area, we typically use Jellyfish to look at metrics regarding proportion of time spent on tasks and indicators of collaboration, such as getting involved in PR reviews and frequency of threat modeling activities. We also look at the frequency of “interrupt work”* and data related to projects. We seek to answer the questions including Are we being inserted where appropriate? and How are we involved with the overall team? to determine if we are successfully partnering with engineering.
- Impact on developer experience: Equally (if not more) important is the load imposed by security on the development team. Ideally, nothing we do should make their jobs harder. Measuring this is challenging, but we do so by examining indicators of blockers and disruptions. For example, Did developers encounter permissions issues? or Did a CI configuration change cause build failure that has to be debugged?
* Interrupt work (or, in "Jellyfish-lingo", "Keeping the Lights On Work") refers to time spent on activities unrelated to planned projects or initiatives. For example, if an engineer reaches out for a one-off review on something we otherwise didn't "plan for." Handling SCA findings is another example! And of course, incident response is usually the most costly kind of interrupt work.
The Role of SCA in a Modern Security Program
At Jellyfish, we use software composition analysis (SCA) to accomplish some tasks that are routine and others that are a bit unusual. It should come as no surprise that we use SCA to discover 3rd-party dependencies and their associated vulnerabilities, and that we want our SCA to help us identify which of those vulnerabilities present true risk. Like most organizations, we use those filtered results to prioritize our remediation efforts so we don’t waste time on things that aren’t real risk. Where we’re unique, aligned with our data-driven culture, is we also use those filtered SCA results as an input for company risk predictions that get presented to our C-suite and board.
Within the last year, we realized the SCA tool we had been using at the time — Snyk Open Source — wasn’t helping us fully meet those goals.
- Didn’t support mature risk modeling: In addition to using reachability for prioritizing risk, we wanted to use reachability as an indicator of true risk. At the time, our risk models were imprecise because we didn’t get good data from Snyk on which risks could actually present a threat to the business, we couldn’t reliably downgrade risk and had to operate as if all the risks were threats. The downside of this approach is we knew our threat models were overly aggressive in a negative way that hurt our credibility.
- Operational inefficiency: We experienced issues with edge cases not working, for example certain languages would or package managers wouldn't function as expected without some tweaking on our part. We manually did reachability analysis before performing a dependency upgrade. If it was an easy upgrade, we would just do it without finding out if it was reachable.
The Journey to Find a Better SCA
After a rigorous evaluation process, we chose Endor Labs for SCA. To understand why, we need to start by going back in time to 2022, when Josiah was a senior security engineer at Cisco…
One of Josiah’s core responsibilities was SCA. His team had a tool that wasn’t meeting their needs — they needed the noise reduction provided by function-level reachability — so they went through a lengthy evaluation process that included several open source tools. As part of this exercise, Josiah went beyond the typical evaluation activities, completing an academic literature review to understand how a tool would need to work to deliver the results he wanted to see. At the time, there were no vendors delivering function-level reachability analysis (Endor Labs was still a twinkle in their founders’ eyes), so Josiah and his team did the only logical alternative: They built his own SCA tool called narrow.
Fast forward to 2024, Josiah found himself at Jellyfish looking to solve SCA-related problems. The SCA tool space had changed in that time, notably with the launch of Endor Labs in 2023. Instead of going the DIY route, we started evaluating vendors claiming function-level reachability, including Endor Labs.
Testing and Implementing Endor Labs for SCA
In looking for a new SCA tool, we had four main requirements
- Risk prioritization: Functional-level reachability and EPSS with coverage for Python and Typescript/JavaScript
- Implementation experience: Easy to integrate anywhere, including locally and in CI, with support for CircleCI
- Governance and policy: Ability to warn or break based on specific parameters such as an EPSS probability or age of dependency
- SBOM support: Capable of generating SBOMs in standard formats (e.g. CycloneDX) and producing a VEX companion document
We found that Endor Labs excelled in these areas, and we’ll dive into what they’ve helped us accomplish in the next sections. But first the takeaway:
Endor Labs is a simple product, and we mean that in a good way. When we integrate it, it just works.
-Josiah Bruner, Sr Security Engineer at Jellyfish
We don’t have intermittent upload issues or problems to work around. We have been pleasantly surprised about the lack of friction when trying new features outside, like container scanning, that are outside the traditional SCA scope. It took us just five minutes to set up a CircleCI job and give it a container from our registry, and we were immediately seeing results.
Prioritizing and Remediating Vulnerabilities
Because of the culture and our small team, we hire security practitioners who can code at the same quality level as a software engineer. That skill set means we have someone on the team who can fix things instead of pushing them to developers, and the experience working in software helps provide a level of empathy for developers. In fact, we’ve been able to use our data to definitively establish that it is most effective for our team when security is treated as an equal partner in engineering efforts. For example, as the security engineer, Josiah spends about half his time contributing to the code base and the other half on internal security activities like threat modeling. In the context of SCA, security engineers are empowered to remediate vulnerabilities directly: they open a PR to upgrade dependencies and make any related code changes, it’s reviewed by a developer, tested, and merged. All this happens in about 30 minutes from start to finish.
Before switching to Endor Labs, we manually researched each reported vulnerability to determine if it was reachable from our products. Because that is resource-intensive, when dependency upgrades were simple enough we performed an upgrade even if we didn’t know whether it was reachable.
With Endor Labs’ state-of-the-art reachability, we can be confident that it’s giving us accurate prioritization so we don’t waste our time on unnecessary research or pointless upgrades. Further, it’s easy to see this data and trends in the tool without sorting through the noise we had in Snyk and in other solutions we considered.
-Josiah Bruner, Sr Security Engineer at Jellyfish
Other tooling we’ve used has a lot of noise in the dashboard and reporting, and it’s hard to figure out how we’re trending. Endor Labs is, in a good way, simplistic. The data I care about is quickly available to me.
- James Kirk, Head of Security and Privacy at Jellyfish
Measuring Risk to the Business
We used to estimate our SCA risk as about 3% of our overall risk. We knew this was an imprecise number since we couldn’t reliably exclude unreachable vulnerabilities. With the superior, research-backed filtering we’re getting from Endor Labs, we can combine reachability and EPSS to accurately estimate that risk. Our risk models are much more accurate, substantiated by two types of analyses:
- Logical Inference: There is substantial evidence that over half of vulnerabilities reported by SCA tools are irrelevant, largely due to a lack of consideration for reachability. Prior to Endor we had to estimate SCA risk using wider likelihood ranges. We now can rule out a subset of vulnerabilities that are definitely not reachable, which implies our likelihood range is narrower, reducing our average risk.
- Manual Verification: At Jellyfish, we've done experiments where we take a sample of vulnerabilities reported by our SCA tool and validate their relevancy. This experiment has been performed before and after implementing Endor Labs, and we can confirm that we have fewer false positives and no tangible change in false negatives.
We basically don’t think about SCA anymore because Endor Labs enables routine, rapid response that keeps the risk low. Now when we present to executives and the board, we feel much more confident that we are accurately stating risk.
-Josiah Bruner, Sr Security Engineer at Jellyfish
Prevent Bad OSS Selections
As we look to reduce overall risk, we also wanted to help developers avoid the introduction of avoidable new risks. To accomplish this, we use Endor Labs’ policies to trigger a warning for a range of use cases. For example, if a developer selects a dependency with a critical vulnerability, we can automatically prevent it from going to production. We’re working to add new policies to flag leading indicators of risk, such as a dependency that’s only existed for 24 hours which can indicate a potential supply chain attack. When developers reach out because a job failed in their PR, it’s easy to communicate what they need to fix and why it matters by using data from Endor Labs.
About the Authors
James Kirk, Head of Security and Privacy
James has over 20 years experience in Information Technology and Security, ranging from network engineering to red teaming/pen testing to GRC. James was the first security hire at Jellyfish, where he’s since built out a team and implemented business-critical programs. Prior to Jellyfish, he held various Information Systems Security Manager and Security Consultant roles for organizations like DataDog, Rapid7, The U.S Department of Defense, the U.S Navy, and Microsoft.
Josiah Bruner, Senior Security Engineer
Josiah is a career security practitioner with experience in the tech, security, and vehicle industries. Formerly of Duo Security (owned by Cisco) and Harman International, he has a Masters of Science in computer science from Georgia Tech and a Bachelors of Science and Engineering in computer science from the University of Michigan. In addition to product security, he is also interested in reliable software systems, risk assessments, and applied cryptographic systems.