Status: [LIVE]
Endpoints found:
https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/krrprnv/gpt
https://colab.research.google.com/github/krrprnv/gpt
https://colab.research.google.com/github/krrprnv/gpt-from-scratch/blob/main/colab.ipynb
Source: https://github.com/krrprnv/gpt-from-scratch
# gpt-from-scratch
[](https://colab.research.google.com/github/krrprnv/gpt-from-scratch/blob/main/colab.ipynb)
[](https://github.com/krrprnv/gpt-from-scratch/actions/workflows/tests.yml)
A decoder-only transformer (GPT) written from scratch in PyTorch, trained as a
character-level language model on tiny-Shakespeare. Attention, blocks, the training
loop, and sampling are all implemented by hand; nothing comes from `nn.Transformer`
or a fused black-box kernel. The model is about 200 lines, and every line is meant to
be read.
the best-val checkpoint writing, seeded with one newline, rendered character
by character as it samples. it learned the speaker names, the scene structure, and the
cadence on its own; invented words like "impitent" and "incents" are what
one-letter-at-a-time generation looks like.
I built this to learn how transformers work from the inside, and the repo is
organized so you can do the same: readable code, a chapter-by-chapter guide
that rebuilds the model in implementation order, and a test suite that encodes the
sanity checks which catch a broken transformer *before* you waste a training run. The
design follows Karpathy's [nanoGPT](https://github.com/karpathy/nanoGPT),
reimplemented for clarity.
## Results
Trained for 5000 steps on tiny-Shakespeare (~1.1M characters, 65-character vocabulary),
the ~10.7M-parameter model reaches a best validation loss of **1.461** (perplexity
$e^{1.461} \approx 4.31$) at step 1750, on par with the nanoGPT char-l