Blog

Derivations, experiments & the odd rabbit hole.

/

Unpacking open recipes for training and tuning foundation models

The open strategies for building and refining capable LLMs, from OLMo's pre-training to the Tulu post-training recipe from AI2.

A walkthrough of fine-tuning Microsoft's 2.7B parameter Phi-2 with supervised fine-tuning (SFT) and Direct Preference Optimization (DPO).

The Transformer architecture behind large language models, and the generative pre-training that brings them to life.

How diffusion models like DALL-E and Midjourney work, from the forward noising process to the simple loss that trains them (DDPM).