Factor a tiny image into conditional predictions
AdvancedNLP 09 · Language Modeling with Transformers (Natural Language Generation)Interactive plot
Description
Enumerate all sixteen binary 2×2 images under a first-order pixel model, then change the generation order.
Try this: Compare row stripes and a checkerboard when copying is likely. Switch raster to column order: which model assumptions changed?
Hints
The chapter section that uses this lab:
Lab · Factor a tiny image into conditional predictions
- Image probability
- 0.064
- Bits per pixel
- 0.991
- All-image probability sum
- 1
Joint probability multiplies normalized conditional factors. All sixteen image probabilities sum to one. A high-copy model prefers sequences with fewer transitions, which need not correspond to the same spatial pattern in each order.
Try this: Compare row stripes and a checkerboard when copying is likely. Switch raster to column order: which model assumptions changed?
First pixel has probability 1/2; each later pixel copies the previous one with probability c. The chain rule is exact, but this model deliberately retains only one previous pixel. Order dependence here comes from that restricted conditional model.
Method sources
- Ian Goodfellow, Yoshua Bengio & Aaron Courville, Deep Learning, MIT Press, 2016
- Ashish Vaswani et al., Attention Is All You Need, NeurIPS, 2017
- Jonathan Ho, Ajay Jain & Pieter Abbeel, Denoising Diffusion Probabilistic Models, NeurIPS, 2020
- Tim Salimans & Jonathan Ho, Progressive Distillation for Fast Sampling of Diffusion Models, ICLR, 2022