Benchmarking Endor Labs vs. Snyk’s GitHub Apps
Compare Endor Labs and Snyk GitHub Apps. Our analysis of 10 open-source projects shows Endor Labs excels in identifying dependencies, reducing false positives, and prioritizing vulnerabilities.
Compare Endor Labs and Snyk GitHub Apps. Our analysis of 10 open-source projects shows Endor Labs excels in identifying dependencies, reducing false positives, and prioritizing vulnerabilities.
Compare Endor Labs and Snyk GitHub Apps. Our analysis of 10 open-source projects shows Endor Labs excels in identifying dependencies, reducing false positives, and prioritizing vulnerabilities.
Compare Endor Labs and Snyk GitHub Apps. Our analysis of 10 open-source projects shows Endor Labs excels in identifying dependencies, reducing false positives, and prioritizing vulnerabilities.
Compare Endor Labs and Snyk GitHub Apps. Our analysis of 10 open-source projects shows Endor Labs excels in identifying dependencies, reducing false positives, and prioritizing vulnerabilities.
.png)
This analysis aims to provide a comparison between the GitHub Apps offered by Endor Labs and Snyk. Utilizing the free tools provided by both Endor Labs and Snyk, we evaluated ten open source projects to assess their capabilities in detecting open source dependencies, identifying vulnerabilities, and generating accurate Software Bills of Materials (SBOMs).
Both of these tools provide free tools on their website. They both have an “Add Projects” option, from which you can import any forked repository into the respective tenant.
This analysis covers common programming languages. It included two projects that highlighted Java, three that highlighted Python, two for Rust, two Go and two Javascript. It also included a variety of popular package managers for each of the languages.
Summary of Findings
The analysis reveals several key themes:
- Endor Labs provides fewer false negatives and more comprehensive visibility into potential risk, due to more accurate dependency reporting
- Endor Labs provides fewer false positives by more accurately understanding project structure
- Endor Labs identifies areas where scans faced challenges, arming organizations with information they need to improve accuracy and avoiding a false sense of security
- Better accuracy combined with greater context gained through program analysis allows for strong noise reduction and prioritization
.svg)
Endor Labs identified more dependencies than Snyk across all projects we benchmarked. This is because Endor Labs assesses and correlates multiple information sources (manifest files, package manager caches, build artifacts, source code, etc.) to ensure accurate and complete package discovery.
Across most projects, Endor Labs identified many more vulnerabilities than Snyk as a consequence of first identifying and including relevant dependencies. In some instances where Snyk identified more vulnerabilities than Endor Labs (Prometheus), Snyk incorrectly identified Go Modules and not packages, resulting in a large number of false positives.
While Endor Labs identified many more dependencies and vulnerabilities than Snyk, Endor Labs’ reachability analysis also dramatically aides AppSec teams in prioritizing which vulnerabilities matter based on reachability analysis. The net result is better visibility without overwhelm.
{{table-snyk-benchmark="/development/96157711918154119455"}}
Endor Labs keeps false negatives under control
A False Negative occurs when a tool fails to report a risk that actually exists.
- Accurate Dependency Reporting: Endor Labs avoids a whole category of False Negatives by taking a comprehensive approach to making sure all the dependencies of a given application are accurately identified. Snyk misses the existence of many dependencies and identifies the wrong version of some dependencies, particularly in npm (JavaScript), cargo (Rust), pip and poetry (Python), and maven (Java) projects.
- Broad Support for Python Projects: Endor Labs supports multiple types of Python projects in both traditional and GitHub App scans. Including those managed by pip via requirements.txt (which Snyk also supports in their GitHub Integration), pip via other mechanisms (setup.py, setup.cfg, pyproject.toml, etc.), poetry, and PDM. Since Snyk only supports requirements.txt for package manifests with their GitHub integration, it silently fails to report dependencies and risks in significant numbers of Python projects.
- Transparency: Endor Labs transparently communicates challenges encountered that may reduce the accuracy or completeness of a scan on a particular project. This allows teams to take action to improve results, rather than living with a false sense of security when scans show few or no issues due to a scan problem.
Endor Labs has fewer false positives and much less noise
A False Positive occurs when a tool reports a risk incorrectly (such as reporting a risk in a version of a dependency that’s not actually in use). We distinguish this from noise, which is a true positive -- the tool is correct that the risk exists -- but the risk is out of scope, inactionable, etc. for a particular application or environment (such as the risk being unexploitable because it’s unreachable from the application code).
- Properly-scoped dependency reporting: Endor Labs analyzes the entire application, not just manifest files. This gives us much more accurate results and reduces false positives from incorrect dependency information. For example, in one of the Go language projects we tested, Snyk incorrectly identified local modules as vulnerable external packages, leading to a significant number of false positives.
- Context-based prioritization: Endor Labs’ program analysis establishes the context for potential risks, including an analysis of whether the vulnerable function within the dependency is actually in use (if it isn’t, then even though the vulnerability exists, there’s no risk because it can’t be exploited), whether the dependency is only required for testing, and so on. Consider the project “Undertow” (Java). Snyk did not report test dependencies (or their vulnerabilities) at all, leading to lower visibility; Endor Labs reports the correct dependencies and vulnerabilities for compliance and tracking purposes, and then places unreachable and test-only dependencies out of scope. The end result is more accurate reporting data and a focus on the 1 vulnerability that actually matters rather than the 7 Snyk reports.
Why it matters: reliable vulnerability prioritization
- Noisy results require extensive research: The inconsistencies in Snyk's dependency detection contribute to a degree of noise in the results, which may affect the reliability of the tool’s assessments and lead to more required research efforts from security and developer teams.
- Prioritization is near-impossible without Reachability Analysis: On top of accurately identifying dependencies and the vulnerabilities associated with them, Endor Labs is able to prioritize vulnerabilities using attributes such as reachability, so you are only acting on vulnerabilities that can be reached. Without this, teams are left with a high volume of alerts and no reliable, accurate method for prioritization.
- Trust: You can trust Endor Labs to understand your applications, and use that understanding to give you higher-quality results as well as prioritization context. And we honor that trust by being transparent with you. All tools experience failures or degraded quality at times, and we are transparent about the issue and provide guidance (when possible) to help you fix any scan-related challenges so you can have confidence in the risk you’re reporting.
Interesting examples of compared projects
1: snowflakedb/snowflake-jdbc
This project has 3 packages. They are listed below:
- net.snowflake:snowflake-jdbc
- net.snowflake:snowflake-jdbc-fips
- net.snowflake:snowflake-jdbc-test
For this analysis, we are not including net.snowflake:snowflake-jdbc-test
Snyk was unable to identify any dependencies for 2 packages:
Endor Labs was able to identify packages, and perform reachability analysis.
Prioritizing With Endor Labs:
In the project snowflake-jdbc, we can prioritize by the following attributes:
- Fix available: prioritize raising issues that have a resolution
- Reachable Function: prioritize raising issues that can be reached from first party code to the vulnerable function in an open source dependency
- Not Test Dependency: only prioritize vulnerabilities going to production
Fixable, Reachable, Non-Test Finding: 2
2. https://github.com/undertow-io/undertow
Fixable, Reachable, Non-Test Finding: 1
3: snowflakedb/snowpark-python
Snyk was unable to get results, giving 0 SCA results.
Endor Labs was able to perform SCA analysis, with reachability, and gave 2 prioritized issues that were Fixable and Reachable.
7. https://github.com/databricks/vector
Total Vulnerabilities: 45
Fixable, Non-Test Findings: 36
8. https://github.com/prometheus/prometheus
Silent Failures, No Npm Projects identified in Snyk -
Fixable, Reachable, Non-Test Finding: 1
