Andrej Karpathy 4c3590c499 fix subtle issue in token decoding in cases where multiple utf8 bytes need to be emitted into a single codepoint. exampels are emoji or foreign languages. basically we have to accumulate token sequences/text and only emit when we get full codepoints

2025-10-15 20:29:54 +00:00

15 KiB

Raw Blame History

View Raw