fam/nanochat-omni
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

tune logit softcap?

Browse Source
This commit is contained in:
Andrej Karpathy
2026-03-02 18:19:37 +00:00
parent 83dccc20ae
commit aba30cb037
2 changed files with 5 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
dev/LOG.md
+4
View File
@@ -4,6 +4,10 @@ A running summary documenting some experiments and findings. Started ~Jan 7 2026
---
## 2026-03-02: SoftCap tuning
Quick experiment to tune logit softcap on d24 scale. Tried 5..30. 5 was terrible, the rest of them were all about equal with the exception of 20, which was the best. Minor but solid improvement: val loss improved by ~1e-3 (0.716 -> 0.715). Setting as default.
## 2026-02-19: Mixture of Experts (negative)
Implemented a DeepSeekV3-style Mixture of Experts layer as a drop-in replacement for the dense MLP. The MoE branch works and improves per-step validation loss, but is not a net improvement on wall clock time due to MoE overhead (at least for our scale of interest of approx GPT-2 capability).