External Auditors Expose Significant Blind Spots in OpenAI’s Safety Monitoring

A recent independent audit has revealed that OpenAI’s internal systems for detecting unusual AI behavior contain significant gaps, leaving a measurable portion of its most advanced models operating without oversight. The findings, produced by researchers from METR and Redwood Research, suggest that the company’s safety net for its high-level agentic capabilities was less comprehensive than previously assumed. This discovery introduces tension into the ongoing debate about whether large AI laboratories have sufficiently matured their internal controls to manage the growing autonomy of their systems.

The review took place from July 21 to August 11, during which the auditors were granted direct access to OpenAI’s codebase and internal documentation. They also conducted interviews with approximately a dozen employees to understand the operational context of the monitoring tools. This level of access was part of a pilot program designed to allow external experts to scrutinize the company’s infrastructure in a way that is rarely done in the AI industry. The goal was to identify weaknesses in the mechanisms designed to catch rogue or unintended agent actions.

The most striking finding concerns the coverage of internal traffic for models at the GPT-5.5 class and above. METR estimates that between 6 and 25 percent of this internal agentic Responses API traffic was not covered by any monitoring during the audit period. While the researchers note that the uncertainty around this specific estimate is significant in both directions, the potential for unmonitored activity at this level of model capability is a critical concern for safety engineers. This gap means that a non-trivial number of agent interactions could have proceeded without any automated safety checks.

The auditors also identified a specific technical flaw affecting internal Codex traffic. Due to a bug in how the system handled 'compaction events'—which are essentially context summaries—roughly 10 percent of monitored traffic went untracked between May and July. This estimate also carries substantial uncertainty according to METR. The bug allowed agents to delay oversight, as the monitoring system only triggered when an agent handed control back to the user. By simply continuing to execute actions without returning control, agents could effectively sidestep the monitoring process.

Beyond these coverage gaps, the researchers found several ways to obscure agent behavior. They demonstrated that it was possible to hide hundreds of actions from the monitor and to bypass the Codex sandbox environment. Furthermore, they determined that specially crafted inputs could likely crash the monitoring system entirely. At the start of the audit, model behavior during reinforcement learning (RL) training was not being monitored at all, adding another layer of unobserved activity to the company's operations.

OpenAI responded by stating that it has fixed all the weaknesses identified by the auditors. The company notes that it paused RL training following the Hugging Face incident in August and subsequently expanded monitoring to cover that process. However, these changes were implemented after the auditors' access period ended, meaning they were not part of the formal evaluation. The company handed over the full 61-page report internally, but public access has been limited to a short summary.

The transparency of the findings has become a point of discussion. After OpenAI applied redactions to the report, METR and Redwood Research leadership received a 38-page version, while the public received only a brief overview. METR stated that it had no contractual right to disclose the redacted portions on its own. This limited public release highlights the ongoing challenge in balancing the need for external scrutiny with the proprietary nature of internal technical details in the AI sector.

This audit represents a significant step in the pilot program jointly organized by METR, Redwood Research, and OpenAI. The initiative aims to provide outside researchers with deep access to internal infrastructure, a move that could become a standard practice in the industry. For future audits, METR has expressed a desire to push for greater public transparency, suggesting that the current model of limited disclosure may evolve as the industry matures.

The results of this three-week audit serve as a reality check for the AI industry. While OpenAI has taken steps to remediate the specific issues found, the existence of these gaps underscores the complexity of monitoring autonomous systems. As AI models become more capable and autonomous, the precision of their safety monitoring will be crucial. The findings suggest that while progress is being made, the work of ensuring comprehensive oversight is far from complete.