Andrej Karpathy 8203efa919 implement flash attention 3 fallback to pytorch sdpa by touching as few lines of code as possible in main files and keeping all implementation to a single file. add tests. add helpful warning messages for the user.

2026-01-16 17:37:51 +00:00

..

test_attention_fallback.py

implement flash attention 3 fallback to pytorch sdpa by touching as few lines of code as possible in main files and keeping all implementation to a single file. add tests. add helpful warning messages for the user.

2026-01-16 17:37:51 +00:00

test_engine.py

integrate Flash Attention 3. +9% tok_per_sec for d12 with ctx even as low as 2048 out of the box nice. also, ready to tune windows huge

2026-01-11 20:33:19 +00:00