Prompt injection is an increasing enterprise risk as organizations begin to utilize larger amounts of Artificial Intelligence (AI) within their critical operations.
Resecurity recently released a report stating that, for some time, security teams have known that Large Language Models (LLMs) do not have traditional trust boundaries. The difference today is the level of integration as AI systems are being placed directly into banking platforms, HR tools, internal copilots, and automated decision making workflows.
From a technical standpoint, prompt-injection-based attacks against AI systems exploit the way LLMs process language versus the way they process code. Attackers create specific instructions to include with user input or in external content to change how the model processes requests so that it can bypass existing system rules to reveal sensitive information or produce misleading output.
Why the Risk Profile Has Shifted for CISOs
The reason why the risk profile has increased for CISOs, Resecurity said, is not the novelty of the attack type; it is the potential impact.
Many modern LLM-based applications use internal documents, Application Programming Interfaces (APIs), and other automated actions such as retrieval-augmented generation and agent framework tools. When a prompt injection is successful in an environment like this, the downstream systems are affected, regulated data is exposed, or unauthorized action is triggered.
In many cases, the threat actor uses a similar technique to what we see in social engineering attacks. They provide a contextually relevant business scenario, build trust incrementally, and then take advantage of the fact that the model prioritizes being helpful over being skeptical. Although often there is no actual access to the system granted, the model will often generate a sensitive artifact and simulate a breach, which erodes confidence in the AI outputs.
This creates both governance/compliance issues, and security concerns for enterprise AI risk management. Since models cannot always determine whether the input instructions are trusted/untrusted, it complicates the ability to ensure auditable, protectable data, and accountable decisions, particularly in regulated industries.
To counteract this, security teams are focusing on implementing compensating controls as opposed to waiting for the model to implement solutions. This includes separating system instructions from user input, validating output deterministically, limiting least-privileged access to tools, and engaging in continuous adversarial testing where the model itself is viewed as an untrusted entity.
