← Back to work
01 · Engineering

GPT Story Generation

From-scratch Transformer Language Model

Try the model
Live demo

Give the model a story beginning and see what it generates. Runs on the deployed checkpoint.

Temperature0.60
Overview

Built a GPT-style language model from scratch with nanoGPT for 5-sentence story generation on the ROCStories dataset. Implemented a custom training pipeline covering GPT-2 BPE tokenization, context packing, Transformer architecture, pretraining, fine-tuning, and generation control. Conducted experiments on learning rate, warmup, dropout, training steps, and weight averaging, achieving a test PPL of 20.44.

Results
31.71M
Model Parameters
20.44
Held-out Test PPL
1st Place
Story Generation Competition
5 Sentences
Controlled Story Generation
Process
01
Architecture
31.7M-parameter decoder-only Transformer
  • → 7 layers · 6 heads · 384 embedding dim
  • → SwiGLU · hidden 1024 · bias-free + RMSNorm
  • → GPT-2 BPE · vocab 50,304 · context packing
  • → Causal self-attention · tied embeddings
02
Pretraining
TinyStories
  • → Learned basic story structure and language patterns
03
Domain Adaptation
TinyStories + ROCStories
  • → Bridged the domain gap
  • → 10× ROCStories upsampling
04
Fine-tuning
ROCStories
  • → Adapted the model to 5-sentence story generation
  • → Dropout + lower learning rate
  • → Validation-based checkpoint selection
05
Weight Averaging
Two fine-tuned checkpoints
  • → 70% A + 30% B
  • → Small but consistent PPL improvement
06
Generation
Controlled decoding
  • → Temperature + top-p sampling
  • → 3-gram repetition blocking
  • → Sentence-aware EOT control
  • → Exactly 5-sentence stories
Experiments
Learning Rate
2e-4 · 3e-4 · 6e-4
Warmup
50 · 500 steps
Dropout
0.0 · 0.1 · 0.2
Training Steps
15K · 30K · 40K · 50K
ROCStories Upsampling
10× · 50×
Weight Averaging
70% A + 30% B
Generation Control
Temperature=0.6 · Top-p=0.9 · Repetition penalty=1.3
Repetition Control
No-repeat 3-gram blocking
Candidate Selection
Best-of = 5
Stack
PyTorchTransformersLanguage Modeling
Notes

One thing I learned from this project was that training a language model is not just about building the Transformer architecture. Small changes in the training setup and generation strategy could have a noticeable impact on the final stories. I was also surprised that a lower perplexity did not always mean better-looking generations. If I did the project again, I would run more systematic experiments and spend more time evaluating the quality of the generated stories.