How to Discover Open Source AI Models in Your Code
Use Endor Labs to discover, evaluate, and enforce policies governing the usage of open source AI models from Hugging Face in your applications.
Use Endor Labs to discover, evaluate, and enforce policies governing the usage of open source AI models from Hugging Face in your applications.
Use Endor Labs to discover, evaluate, and enforce policies governing the usage of open source AI models from Hugging Face in your applications.
Use Endor Labs to discover, evaluate, and enforce policies governing the usage of open source AI models from Hugging Face in your applications.
Use Endor Labs to discover, evaluate, and enforce policies governing the usage of open source AI models from Hugging Face in your applications.
Our mission at Endor Labs is to secure everything your code depends on. That includes the open source software packages your first party code depends on, as well as your container images, GitHub actions, and even the repositories. And now we’re extending that coverage to open source AI models.
Starting today, you can use Endor Labs to discover, evaluate, and enforce policies governing the usage of open source AI models from Hugging Face in your applications.
Open source AI models introduce unique risks
Hugging Face hosts over 1 million AI models and more than 220,000 datasets, making it the largest open source AI repository and a go-to resource for developers building AI applications. As developers increasingly adopt these open source models, application security teams face new challenges managing the associated risks. AI models present unique risk patterns because they combine code, weights, and training data that may come from multiple sources.
Traditional application security tools weren't designed to identify AI models as dependencies and to assess their unique risks. While new solutions are emerging to address this gap, they often operate in isolation, lacking critical context about the rest of the application and its dependencies.
Modern SCA tools like Endor Labs use program analysis to understand exactly how your application works, down to the exact functions calling your dependencies. This lets us accurately inventory all the dependencies in your application, including phantom dependencies like AI models that may not be declared in the application’s manifest file.
Start managing AI risks with Endor Labs
In October 2024, we released the ability to evaluate and select safe models from Hugging Face. Now you can scan for Hugging Face AI models used in your source code, evaluate each model for risk, and make informed decisions about what to do, like warn developers or block the model from entering production. Critically, you can gain a comprehensive view of all your application dependencies in a single place, and start managing AI risks within your existing workflows.
This is especially important for organizations seeking to comply with standards like ISO/IEC 42001:2023 or NIST-AI-600-1. These frameworks set standards for the development and operation of artificial intelligence management systems (AIMS) in your organization. Both ISO 42001 and NIST-AI-600-1 require you to catalog all the AI models used in your organization in order to conduct a risk assessment and implement security controls.
Discover and inventory models from Hugging Face
Endor Labs customers can access a new global AI/LLM page to easily browse a global view of the different AI models your software development teams have integrated into their applications. You can also view the discovered AI models within a specific project when clicking into the standard dependencies tab of a project.
Evaluate AI models for risks
After clicking into a model, you can see metadata for the model as well as its Endor Score. We use 50 out-of-the-box checks to evaluate models across four dimensions: security, activity, popularity, and quality. This helps you, and your software engineering teams, identify models with questionable sources, practices, or licenses.
You can find additional information about the top contributing score factors, the individual scores for the model in each category, model metadata, and much more. This information is easily available and searchable in the DroidGPT tab of the UI, or can be extracted using the CLI or API so your developers can easily review it within their existing workflows when selecting models.
Enforce guardrails for AI model usage
We also introduced a new finding policy for setting and enforcing guardrails around the usage of open source AI models. You can start using it by creating a new finding policy and selecting the “AI Models with Low Scores” template.
You can also write your own custom policies for the specific risk factors your organization cares about, like license types. You can even enforce guardrails even after an AI model has entered production. If something about a model changes in the future – for example, it adds Python files that could contain malicious code – a new finding will be surfaced during your next SCA scan.
Let’s take a look at a common request — managing acceptable licenses for AI models.
Example: Managing license risks
AI models often have more complex licensing due to different components that go into them. To start managing license risk for your organizations, you could use the “AI Models with Low Scores” policy to create a finding whenever an AI model has an operation score lower than 7 to prompt a review. Or, if your organization only accepts certain license types for open source, you could use the “Permit only specified software licenses policy” to create a finding any time a model without one of those licenses is used.
How Endor Labs’ AI model discovery works
How we detect models in your source code
AI models coming from HuggingFace are loaded and used in code as specifically outlined in the HuggingFace documentation. Here are the following patterns we look for in source code to detect usage of models from HuggingFace:
1. pipeline()
2. $AUTO_MODEL.from_pretrained()
a) Where $AUTO_MODEL can be any of the following listed under the generic model classes section of the HuggingFace documentation.
3. hf_hub_download()
4. hf_hub_url()
5. snapshot_download()
6. Usage of functions from the llamaindex wrappers around the HuggingFace API
7. Usage of functions from the langchain wrappers around the HuggingFace API
This list is continually evolving and expanding over time, and currently does not encompass all possible methods for using or loading models from Hugging Face in code. Since most of these functionalities are derived from the transformers library, which is specifically tailored for Python, our discovery capabilities are presently limited to Python source code.
When a defined pattern is detected in the source code, we extract and store the model name provided as a parameter to the function. The scan continues to identify additional patterns, ultimately compiling a comprehensive list of all discovered models in the codebase. To minimize false positives, each identified model is validated by cross-referencing it with the HuggingFace model hub to ensure its existence.
There are some limitations in this approach: a user is not confined to directly providing the model name to the function parameter. For instance, it is entirely up to the user to define the model name elsewhere in the file as a variable to use throughout the rest of the code. This means that extracting the model name from the parameter of the functions above may not actually result in obtaining the model name but instead the variable reference.
AI as a Resource
Here comes AI to the rescue! Let’s say for example we have the following:
When initially scanning, we would discover snapshot_download() as a viable pattern and in turn extract the `repo_id` parameter value. However, in this case the `repo_id` is set to a variable (which is repo_id). Using just our approach above, this would fail to tell us which model is being used from HuggingFace as the string model name is not given.
In such cases, the code content where the pattern was detected is sent over to our AI client to determine if and where the variable repo_id is assigned within the same file. If an assignment is found, we once again validate the result by ensuring the AI client identifies a valid HuggingFace model name. Once confirmed, the model is successfully stored as a dependency.
To evaluate the effectiveness of this approach, an experiment was conducted using 100 GitHub repositories and verifying the accuracy in AI model detection. Repositories utilizing HuggingFace AI models were identified, and each repository was manually reviewed to compile a list of models in use. The same repositories were then scanned using the Endor Labs' AI model discovery pathway, and the results were compared against the manual findings. The experiment revealed an 80% accuracy rate in discovering models where the names were not directly passed as function parameters and instead required AI to identify them within the code. These results were deemed satisfactory, supporting the continuation of this approach.
What’s next?
Endor Labs customers can start discovering the AI models in their applications by enabling the `--ai-models` flag on their existing SCA scans. We plan to continue expanding support for additional sources and languages in the future.
Book a demo to learn how you can start securing everything your code depends on — from traditional open source packages to the newest AI models on Hugging Face — with Endor Labs.
