d923aa1e31
Hopfield + Hebbian hybrid memory system for LLMs. Two nights of experiments (16 iterations), validated on LongMemEval (ICLR 2025). Architecture: - Single-hop: Two-Stage Hopfield (NN top-20 → softmax settle) - Multi-hop: Hebbian W matrix with WTA pattern separation - 64% on LongMemEval (500 questions), retrieval-only, no LLM dependency - 4ms latency @ 20K memories, ~1GB VRAM Key findings: - Hopfield attention solved noise tolerance (20% → 100% vs flat Hebbian) - WTA pattern separation enables 20K+ capacity - Multi-hop associative chains (6 hops, CosSim=1.0) — RAG can't do this - MiniLM-L6 is optimal (discrimination gap > absolute similarity) - Paraphrase cue augmentation: 55% → 100% on synthetic, 36% → 64% on benchmark - SNN encoder viable (CosSim 0.99) but not needed for current architecture
38 lines
1.1 KiB
Markdown
38 lines
1.1 KiB
Markdown
# P3: 突破 20K 80% 天花板
|
||
|
||
## 结论：天花板来自 embedding 模型，不是架构
|
||
|
||
### Top-K Coverage 分析
|
||
|
||
| K | N=20K |
|
||
|---|-------|
|
||
| 5 | 80% |
|
||
| 50 | 80% |
|
||
| 200 | 80% |
|
||
|
||
K 从 5 增加到 200，coverage 不变。那 2 个 failure 的 paraphrase 在 embedding 空间里根本不是正确 cue 的最近邻——即使只有 10 条记忆也找不到。
|
||
|
||
### 架构优化无效
|
||
|
||
| 方法 | bg=20K |
|
||
|------|--------|
|
||
| Two-stage K=5 | 60% |
|
||
| Two-stage K=200 | 30% (更大 K 更差!) |
|
||
| Hierarchical clustering | 40% |
|
||
|
||
更大的 K 引入更多噪声，Hopfield attention 被分散。Hierarchical 也没帮助。
|
||
|
||
### 根因
|
||
|
||
失败的 paraphrase 对（embedding cosine similarity）:
|
||
- "Need observability" ↔ "Let's set up monitoring" = 0.257
|
||
- "When's the standup?" ↔ "Team meeting schedule" = 0.375
|
||
|
||
这些在 MiniLM 的 embedding 空间里根本不算"相似"。任何基于 embedding 距离的检索方法都无法找到它们。
|
||
|
||
### 解法 = P2
|
||
|
||
**Paraphrase augmentation 是唯一解法**（已验证 55% → 100%）。
|
||
|
||
不需要改架构。不需要换 K。不需要 hierarchical memory。只需要在存储时覆盖更多的表达方式。
|