Traditional application security models are built around deterministic behavior. That means that a specific input will always return the exact same output. LLMs, however, can produce wildly different outputs depending on context, model version, or randomness in generation.
This means that security teams cannot test an LLM in the way they could test a REST API, as there is no fixed means of validating it.
To make things more complex, the data your LLM surfaces can increase risk. When you use an LLM, part of the way it generates a response is through Retrieval Augmented Generation (RAG). This is where it will pull live data into the prompt when triggered. At the click of a button, every document, database, and API within your network can be accessed by the LLM, increasing your attack surface.
The third risky area is the content that LLMs themselves generate. Now, tools like Cowork allow users to set up tasks for the AI to complete. Whether it’s sending emails, querying databases, or modifying files, AI is no longer sitting on the sidelines.
Traditional AppSec and cloud security controls are not redundant in the world of LLMs, but they, alone, are not sufficient. LLMs need their own layer of security control designed to address the way they actually function.
The Main Threats
Prompt Injection: Prompt Injection is, perhaps, the most well-known of the LLM risks. A malicious input is interpreted as an instruction, designed to circumvent any guardrails that the model has in place. This risk tops the OWASP Top 10 for LLM applications and real world enterprises, with an attack success rate sitting somewhere between 50% – 84%.
Data Exfiltration Any LLM that has access to sensitive data can be a vector for data loss. As previously mentioned, this might be via RAG retrieval or session memory. The LLM inadvertently sharing information that it was not authorized to. OWASP ranks sensitive data disclosure as the second most pressing concern for security teams.
Supply Chain And Third-Party Model Risk Most LLM systems are built on top of preexisting AI models. If one of these third-party plugins is breached, any other instance where this plugin is used will also be compromised. With the vast numbers of user and connections, this has the potential to be devastating.
Insecure Agentic And Tool Integrations AI can now complete tasks completely autonomously. LLM that can only produce a text output might be a vector for data loss. An active LLM that can execute code, send communications, or query production databases is vastly more dangerous.
LLM Security Best Practices
1. Build An LLM-Specific Threat Model
Do not bolt LLM risk onto an existing AppSec threat model and expect it to be effective. It is worth taking the time to map the full attack chain; what data can the model access, what can it retrieve at runtime, what actions can it take, who is able to influence its inputs? Indirect prompt injection means your threat model needs to include every external content source the model interacts with, not just your own users.
Scott Richards, SVP at OpenText explained that “agents are already outnumbering humans by 50:1… We need to use very strong AI-powered application security solutions to look through the code and find vulnerabilities in these agents.” Attempting to use traditional tools to address nascent challenges will leave security teams constantly on the back foot.
2. Apply Least Privilege To Every Integration
LLM access should be treated in the same way that an employee is. Access should be restricted to just the areas that they need, with permissions only increasing in response to a valid need. This requires security teams to take a proactive approach, rather than assuming that default settings will be good enough.
This advice is backed by Amber Britton, CEO of Doppler who told Expert Insights that “[e]very agent, every service, every automation needs to get the minimum privileges it needs, scoped tightly wherever possible, rotated aggressively,”
3. Treat All Prompt Content As Untrusted Input
There are lots of things that AI is good at, telling the difference between a user’s instruction and a hidden instruction is not one of them. You should, therefore, treat all interactions as untrusted, with defense built to manage this. Layered defenses combining input validation, output filtering, and behavioral monitoring will reduce prompt injection success rates. Some reports have seen these defenses decrease threats from 73% to under 9% when properly implemented.
This is an example of the Zero Trust strategy where nothing is permitted access by default. Identities need to prove their right to access a network area, before that access is granted. You can hear more about zero trust in our interview with John Kindervag, the creator of the concept.
4. Keep Humans In The Loop
If an LLM can take an action that has a real world consequence, the LLM should not be the only identity overseeing this. Ensure that humans are either granting approval or at least being notified of any updates. This is particularly important for high stakes changes but should still be the default for any action. Areas to be mindful of include financial transactions, access changes, and external communications.
Organizations will need to find the right balance of automation and human in the loop. This balance point will depend on the tasks that you’re asking LLMs to accomplish and the other security measures you have in place.
Vanta’s Kush Kashyap explains how how they maintain human oversight: “What we internally do is, we don’t use human in the loop to check every single eval. We use a machine… But there is a human in the process who is setting the parameters… And that materially decreases the dependency on the human.”
5. Logging And Observability For LLM Behavior
Standard SIEM rules were not designed to identify a model being manipulated into bypassing its own guardrails. Logs should capture both input and outputs, retrieval sources, and tool interactions, with enough context to reconstruct the conditions after the event. If anything does go wrong, you want to be able to identify where the loophole was, and ensure that it doesn’t happen again. Visibility is the single biggest area that you can address to mitigate any negative LLM impacts.
6. Govern Third-Party Models And Vendors Formally
Any third-party vendor or plugin should be treated as part of your supply chain. That means that you should take the time to understand what data sovereignty strategies they have in place. Ensure that SLAs and monitoring align with your own expectations. This needs to be an ongoing assessment as providers may update terms and conditions throughout your relationship with them.
Khush Kashyap from Vanta told us that: “When traditional SaaS tools now have AI agents which are accessible to people to use, it’s really important for our contracts to reflect the new terms… our supply chain risk has just materially, exponentially increased.” The message is that change, churn, and evolution is inevitable. The right governance strategy will ensure that these tools are managed in the most appropriate way.
7. Control Training Data
If you train your LLM based on internet data, you want to know exactly what that data includes and where it came from. It’s all too easy for a malicious actor to tamper with publicly available data, knowing that it will lead to loopholes and vulnerabilities within AI models. Any PII embedded in training data can reemerge in unexpected ways down the line.
CyberArk’s Andy Thompson explained the risks of data poisoning, prompt based injection, and jailbreaking. If any aspect of your LLM strategy is overlooked, you can be sure that attackers will find a way to exploit it. As all LLMs start with data and training, security strategies need to reflect this too.
8. Define Acceptable Use Before Deployment, Not After
Before an LLM goes into production, you need to outline what use cases are sanctioned, who owns deployment, and the areas that the model is not permitted to access. Again, this governance needs to be ongoing and continuous, rather than a one-time sign-off. Acceptable use policies need to be continually revised as use-cases evolve.
According to Kiteworks, 90% of organizations say that employees are already using AI tools, yet only 38% have a formal, comprehensive AI usage policy. When designing this strategy, you need to map out acceptable uses for your staff, as well as the technical capabilities that your LLM is able to do.
9. Red Team Before Go-Live, Then Keep Doing It
Standard PenTesting does not cover prompt injection, jailbreaking, or adversarial retrieval. As these are some of the biggest risks that your LLM will encounter, it’s essential that you test these out. These tests should be regularly repeated, ensuring that emerging threats can still be defended against. This type of proactive security testing can reduce incident response costs by 60% – 70% when compared with responding to a breach.
Deepak Jeevankumar, Managing Director at Dell Technologies Capital explained to us that: “In terms of AI security, it goes from model red teaming to API control, MCP servers, A2A, and so on.” You don’t want to be waiting for a vulnerability to be exploited to know that it exists. Take the time and make the investment before the attackers do.
Where Security Leaders Should Start
I’ve just outlined nine best practices that should inform the way that you approach LLMs within your organization. That’s a lot to keep track of simultaneously. In fact, spreading yourself too thinly and trying to address all of these areas in part may be less effective than dedicating focus to one or two areas.
The first thing you should do is work out which risks are most relevant to your organization and where you are in your journey. For instance, if you’re in the early stages of LLM deployment, prioritize governance and least privilege. Baking these in from day one will save you hassle and cost down the line.
If your organization already has LLMs in a production environment, focus on logging and PenTesting. This will give you oversight into how your LLM is operating, and any areas that you need to be wary of.
As more organizations adopt LLMs, we will see regulatory pressures increase. The EU AI Act came into effect in August 2024, with rules and requirements increasing into 2027. Similarly, NIST’s Generative AI Profiles gives US organizations voluntary, but increasingly referenced, benchmarks.
Neither of these frameworks currently dictate the exact technical controls that you need to have in place, but this will change with time. Multiple stakeholders, from customers to insurers, will expect you to be able to prove that you take reasonable precautions with regard to LLMs.
The Bottom Line
LLMs are a fast emerging technology area. There is, inevitably, a lag before all organizations catch up with best practice. Many of the measures outlined in this article are straightforward to implement, but require specific configuration based on your organization’s unique circumstances.
Ensuring that LLMs have the necessary levels of privilege and that there are sufficient guardrails in place to mitigate prompt injection go a long way to securing your LLM.
Organizations that are able to effectively manage LLM security are those who treat the risks as new, rather than trying to make them fit within existing technologies and strategies. The CISOs who build specific LLM-governance, access controls, and testing will be the ones who can have confidence in the way their organization uses LLMs.
