Flow Matching & MeanFlow
Generative Modeling with Flow-Based Methods
Investigated Flow Matching and MeanFlow through a series of controlled experiments. Reproduced the prediction/loss comparisons from Back to Basics across different datasets and dimensions, then analyzed the degradation of v-prediction in high-dimensional settings. Tested model capacity and timestep scheduling as potential remedies, and implemented MeanFlow for one-step generation with NFE = 1.
- Prediction / Loss: v-prediction / v-loss
- Architecture: 5 hidden layers · 256 units · ReLU
- Time embedding: 128-d sinusoidal
- Optimizer: Adam · lr = 1e-3
- Batch size: 1024
- Training steps: 25,000
- Sampling: Euler ODE · 50 steps
- Generated samples: 2,048
- → v-prediction degrades in high dimensions
- → x-prediction scales better to high dimensions
- → x-prediction + v-loss performs better
- → Tested model capacity
- → Tested timestep scheduling
- → Larger models reduce noise and scatter
- → Schedule shifting further improves results at hidden widths of 512 and 768
- → Best results come from larger capacity + schedule shift
- → NFE = 1 · One-step Generation
- → Comparable quality to multi-step sampling
- → One-step MeanFlow achieves comparable quality to multi-step Flow Matching
- → Generation cost reduced to NFE = 1
What surprised me most was how much the choice of prediction objective mattered as the data dimension increased. v-prediction worked well in low dimensions but became much harder to train at D=32, which led me to experiment with model capacity and timestep scheduling. I also found MeanFlow interesting because it approaches the sampling problem differently, allowing the model to generate a sample in a single step. This project gave me a better understanding of how seemingly small design choices can affect the behaviour of generative models.