Nonprofit evaluators METR and Redwood Research on Thursday said they conducted a pilot review of an internal monitoring system at OpenAI, the artificial intelligence company's latest effort to determine how it will deepen its work with third-party safety organizations.
METR and Redwood said the exercise was designed for OpenAI to test what it would look like to give external researchers "a deep level of access" to its internal infrastructure.
From July 21 to Aug. 11th, an employee from each organization had access to OpenAI's internal codebase, documentation and some technical staffers. The employees were chaperoned while on site and used OpenAI-issued devices that "never left the premises," according to a release.
The evaluators discovered several "novel vulnerabilities," including an exploit that would allow agents to prevent "hundreds" of recent actions from being monitored. OpenAI told METR and Redwood that it has patched the identified vulnerabilities that impact its production safeguards, according to the release.
METR and Redwood said they agreed to only publish a high-level public post about their findings since the pilot was not designed to "provide robust public transparency about OpenAI's monitoring system." They shared a 61-page internal report with OpenAI, and said they expect to publicly share more details about future exercises.