GPT‑Red Finds Flaws in AI Agents: How OpenAI Trains Defenses

OpenAI has introduced GPT‑Red, an internal model built not to answer users but to uncover weaknesses in other AI systems. It creates attacks, watches how an AI agent responds, and adjusts its approach until it finds a way to pull the agent away from the user’s original task.

The company turns those attacks into training material for newer models. The approach resembles a security exercise: an offensive team finds a hole, defenders close it, and both prepare for the next attempt. GPT‑Red automates much of that process, although OpenAI says it complements—not replaces—human security specialists.

What GPT‑Red actually is

GPT‑Red is a specialized automated red-team model. It was trained through self-play reinforcement learning, with the attacker and a varied group of defender models improving at the same time. GPT‑Red received a reward when it caused a valid failure, such as a successful prompt injection. The defender was rewarded for ignoring the malicious instruction and completing the user’s real task.

Its training environments resemble situations agents encounter in practice. A hostile instruction could be hidden inside a webpage, email, local file or the output of a connected tool. As defenders became more resilient, GPT‑Red had to discover subtler ways around them.

The model remains internal and is not offered to users or developers. OpenAI says it wants to keep the offensive capabilities deliberately trained into GPT‑Red away from potential attackers.

Why prompt injection matters

A prompt injection is an attack in which third-party content poses as an instruction to an AI. A user might ask an agent to summarize an inbox while one message contains a hidden direction to forward documents elsewhere. If the agent cannot separate its owner’s request from untrusted material, it may do something the user never authorized. OpenAI provides a separate explanation of the attack and its layered defenses.

The risk grows when an AI can do more than produce text. An agent connected to email, cloud files, a browser, payments or internal systems has tools that an attacker may try to hijack. A secure setup therefore needs to track where instructions came from, limit permissions, monitor actions and request confirmation before consequential steps.

The vending-machine attack

The clearest case study involved an agent managing a vending machine in OpenAI’s office. Built by Andon Labs, the system could order stock, change prices and handle customer orders. GPT‑Red first tried approaches in a simulation and observed the agent’s tool calls, then transferred the prepared attack to the live system.

According to OpenAI, GPT‑Red achieved all three assigned objectives: it changed an expensive in-stock item to the minimum permitted price of $0.50, ordered an item costing more than $100 and offered it for $0.50, and cancelled another customer’s order. The vulnerabilities were disclosed, and additional safeguards are being tested.

The example matters for reasons that go beyond snacks. It shows how an error in an agent’s interpretation can turn into a real financial action. The same mechanism inside banking, logistics, procurement or a corporate cloud account could have much more serious consequences.

What OpenAI’s internal tests found

OpenAI compared GPT‑Red with people in a replicated research benchmark for indirect prompt injection. On scenarios the model had not seen during training, GPT‑Red found a successful attack in 84% of cases, compared with 13% for human participants. The evaluation targeted GPT‑5.1 and ran on OpenAI’s internal mirror of the benchmark.

The company also used GPT‑Red’s attacks to train GPT‑5.6. OpenAI reports that GPT‑5.6 Sol produced six times fewer failures on its hardest direct prompt-injection benchmark than its best production model from four months earlier. For a class called Fake Chain-of-Thought attacks, reported success fell from above 95% against GPT‑5.1 to below 10% against GPT‑5.6 Sol.

Those figures need context. They come from OpenAI’s own evaluations, and when the announcement was published the company said a detailed preprint would follow later in the week. The reported improvements have not yet been independently reproduced in full.

Does this mean agents are secure now?

No. Prompt injection remains an open security problem. Attackers adapt, and an agent may encounter a malicious instruction in a source or format that its training never covered. Even a stronger model should not receive unrestricted access to sensitive information or high-impact actions.

Effective protection is layered: adversarial training, minimum necessary permissions, source checks, monitoring, user confirmation and independent testing. OpenAI says human and third-party red teams will remain part of that process.

For everyday users, the lesson is practical. If an agent asks to send an email, make a payment or upload a file, check both the destination and why the action was proposed. Organizations need more: role-based access, activity logs and a reliable way to stop an operation before damage is done.

GPT‑Red matters less as a future product than as a change in how AI safety work is done. As agents gain autonomy and access to real tools, defensive testing has to evolve just as quickly—and try to break them before a real adversary does.