GPT Story Generation
From-scratch Transformer Language Model
Give the model a story beginning and see what it generates. Runs on the deployed checkpoint.
Built a GPT-style language model from scratch with nanoGPT for 5-sentence story generation on the ROCStories dataset. Implemented a custom training pipeline covering GPT-2 BPE tokenization, context packing, Transformer architecture, pretraining, fine-tuning, and generation control. Conducted experiments on learning rate, warmup, dropout, training steps, and weight averaging, achieving a test PPL of 20.44.
- → 7 layers · 6 heads · 384 embedding dim
- → SwiGLU · hidden 1024 · bias-free + RMSNorm
- → GPT-2 BPE · vocab 50,304 · context packing
- → Causal self-attention · tied embeddings
- → Learned basic story structure and language patterns
- → Bridged the domain gap
- → 10× ROCStories upsampling
- → Adapted the model to 5-sentence story generation
- → Dropout + lower learning rate
- → Validation-based checkpoint selection
- → 70% A + 30% B
- → Small but consistent PPL improvement
- → Temperature + top-p sampling
- → 3-gram repetition blocking
- → Sentence-aware EOT control
- → Exactly 5-sentence stories
One thing I learned from this project was that training a language model is not just about building the Transformer architecture. Small changes in the training setup and generation strategy could have a noticeable impact on the final stories. I was also surprised that a lower perplexity did not always mean better-looking generations. If I did the project again, I would run more systematic experiments and spend more time evaluating the quality of the generated stories.