Steve Jurvetson· @FutureJurvetson · X·· 3 小时前
事实摘要
Zyphra 介绍其分层注意力方案 ZHA（Zyphra Hierarchical Attention），称其为通向真正次二次注意力的路径。ZHA 在压缩摘要构成的学习树上检索，官方给出的投影加速为 10M token 时相对稠密注意力 35 倍、相对 MSA 5 倍，1B token 时约 400 倍和 55 倍。Steve Jurvetson 引用该说法并评论，超长上下文可能需要次二次注意力，分层注意力不仅关乎效率，也关乎注意力能否成立。
正文 · 原文
As in the brain, "ultra-long contexts may require subquadratic attention. Attend to everything, and the signal gets lost in noise. Too sparse, and we lose the signal. Hierarchical attention may be essential not just for efficiency, but for attention to work at all." — Zyphra
Today we’re introducing Zyphra Hierarchical Attention (ZHA), a path to true subquadratic attention. ZHA searches a learned tree of compressed summaries, with projected speedups of 35× over dense attention and 5× over MSA at 10M tokens, increasing to ~400× and ~55× at 1B. 🧵在 X 查看被引用的帖子
来源：Steve Jurvetson · x.com