#安全/治理

The only way out is through. They're calling it the Bugpocalypse. LLMs are being used to find long-hidden cracks in the software infrastructure, and it's happening at a furious rate. 3% of all compromises in computing history happened in the last week, claims one source? I believe it. There are people saying public access to the frontier models ought to be locked out, or heavily guardrailed, over this. The problem with that idea can be summed up in one sentence: when lockpicks are outlawed, only outlaws will have lockpicks. The lockpicks will be Chinese open-weight models, if nothing else. There's no way to stop those from becoming ubiquitous. There are two futures in front of us. One in which only the bad guys have lockpicks, and security holes go undiscovered until they're exploited. Another in which everybody has lockpicks and is aggressively red-teaming their own software so it gets fixed before the holes are exploited. The corpus of human code is huge but finite. The number of defects in it has to be finite too. We can fix those defects now; we have a million friendly robots to do it with. The only way out is through.

OpenAI red-teamed their hardened containment infrastructure by asking teams of frontier-model agents to try to break through its defenses. They easily escaped in novel ways. Quoting: 'Our agents VM escapes showed why containment must extend beyond the sandbox. To test the surrounding perimeter, we gave teams of agents control of a host machine and asked whether they could extend that compromise to reach the live internet. After the Hugging Face incident, we had reduced the channels through which models could reach the internet during training and evaluation. Narrow connections remained for specific services we operate, such as metrics, logging, and authentication management. Our red-teaming agents found vulnerabilities in almost every one of these services that gave them limited forms of internet access. Some of these attacks exploited previously unknown flaws, while others combined legitimate features in unexpected ways. In one test that assumed compromise of a VM, an agent reused a firewall-approved connection to reach a blocked authentication endpoint in our metrics service and then made that endpoint fetch a model-provided URL. In other tests, agents exploited misconfigurations that allowed blocked requests to pass through DNS and firewall controls, as well as ways to make services we run fetch external URLs.' 'In one experiment, we asked agents in separate VMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache: a small, fast memory that holds copies of data that applications have recently accessed. Reading data from the cache is faster than retrieving it from main memory, and the agents used this timing difference to communicate. One agent encoded information by evicting selected data from the cache. Another timed how long it took to reload that data, using the difference between fast and slow reads to recover the message. Once the agents agreed on a protocol for encoding and decoding these signals, they could exchange the synthetic secrets without changing a shared file or opening a network connection. This cache-timing technique had also been used to extract information in the Spectre and Meltdown attacks⁠(opens in a new window). Other tests exposed channels involving shared storage, competition for CPU time within a shared execution container, and a TLS certificate cache in OpenAI’s offline web infrastructure. In the storage test, one agent alternated between continuous writes and inactivity, while another measured the resulting timing differences and used those differences to decode the signal. As one agent described the strategy in its chain of thought: Agent chain-of-thought reasoning: >Could create acoustic modem-like signal … using continuous writes vs idle.'

过去几个月里，我们大幅强化了研究训练环境的安全性。我们已移除所有前沿模型在训练期间的实时互联网访问权限，并为它们构建了一个离线世界来替代——一个缓存的网络版本，让它们仍能学习浏览、搜索和使用应用程序。 9月20日，在一个模型利用 DNS 配置错误找到逃出这个离线世界的方法后，我们暂停了前沿训练，以进一步加强隔离。我们组建了由我们最具网络能力的模型驱动的智能体团队，让它们寻找我们研究环境中的弱点。 尽管它们没有找到既能轻易利用、又能向我们的监控隐藏其活动的通往互联网的路径，但它们确实发现了零日漏洞、虚拟机逃逸，以及智能体之间通信的硬件侧信道。我们利用它们的发现进一步加强了防御，并为开源软件贡献了修复。 在强化隔离架构后，我们于10月1日恢复了前沿强化学习训练和推理。我们预留了大量算力用于持续的红队测试，并将高算力红队测试纳入我们用来判断前沿训练和评估能否安全推进的安全论证中。 https://openai.com/index/securing-our-frontier-model-training-environment/

@nateberkopec I'm cautiously starting to suspect it may not be a problem with today's strongest models - Anthropic in particular are very confident about their ability to avoid prompt injections - but it's still a real concern for cheaper, weaker models

Two employees from METR and Redwood Research worked on premises at OpenAI for three weeks this summer in an embedded assessment pilot. This team evaluated the systems OpenAI was using to monitor for misalignment from internal agents. A brief write-up of their work is now public.

OpenAI 模型帮助研究人员——作为我们“修补地球”计划的一部分——发现了 PHP 中的关键安全漏洞，PHP 是一种开源编程语言，估计用于运行 70% 的网站。 在过去两个月中，我们牵头了一个由维护者、研究人员和安全提供商组成的联盟，在披露前准备修复和防护措施。这一时长是必要的，因为修复相当复杂，需要对 PHP 引擎进行重大更改。现在，所有使用 PHP 的网站都可以安全地升级其版本以保持安全，此外还可以应用各安全供应商提供的缓解措施。 鉴于这些漏洞的严重性和 PHP 的广泛使用，我们认为这个漏洞是近期记忆中最具影响的漏洞之一。它也标志着一个新阶段，即 AI 能够发现具有互联网规模影响的安全漏洞。 这是“防御者窗口”发挥作用的一个例子：据我们所知，前沿 AI 能够在攻击者之前发现这个漏洞，给防御者时间准备。 https://openai.com/index/coordinated-disclosure-of-critical-vulnerabilities/