Parth Shastri
CV
← All posts

Training8 min read

Unpacking open recipes for training and tuning foundation models

The open strategies for building and refining capable LLMs, from OLMo's pre-training to the Tulu post-training recipe from AI2.

Parth Shastri

The world of Large Language Models (LLMs) is moving incredibly fast. Every week a new paper or checkpoint drops, and if you blink you might miss an entire generation of models. In the middle of this whirlwind, one theme keeps popping up in my own reading and experiments: openness. When data, code, and training tricks are shared in the clear, the whole field moves faster. Let’s dive into some of the key strategies and insights for building and refining capable LLMs using open approaches, particularly focusing on the work emerging from initiatives like the Allen Institute for AI (AI2).

This blogpost is based on an excellent lecture by Hanna Hajishirzi, part of the Berkeley RDI Advanced Agents MOOC. I would highly recommend checking out the lecture and the course!

The need for openness#

Why is “open” such a big deal in LLM research? Relying solely on proprietary models is akin to trying to understand the solar system by only looking at newspaper clippings. You miss the depth and detail. To truly advance the field, we need language models that are transparent, reproducible, and accessible. This openness fuels innovation across numerous research areas, from the fundamental science of LMs and improving their core capabilities to applying them in the real world, mitigating biases, and making them more efficient.

Pillars of open LLM development: OLMo and Tulu#

Two significant contributions to this open ecosystem are OLMo (Open Language Model) and Tulu.

  • OLMo: This isn’t just another model with open weights. OLMo aims to be a truly open language model. This means providing access to the training data (like the 3T token Dolma dataset), the training code, evaluation benchmarks, and the model weights themselves (Groeneveld et al., 2024). The latest iteration, OLMo 2, demonstrates competitive performance, even on par or better than models like Llama3 and Qwen2.5 in certain evaluations.
  • Tulu: While OLMo provides the foundational model, Tulu offers an open, reproducible, and state-of-the-art post-training recipe. This recipe guides researchers on how to take a pre-trained base model (like OLMo or others such as Llama) and fine-tune it to follow instructions, engage in dialogue, reason effectively, and even use tools.

The journey of building a modern LLM typically involves Pre-training, Post-training, and considerations for Test-time Inference. Here is the whole path in one place. Click through it once, then the sections below fill in the details.

Interactive

Broad pre-training

OLMo · 99% budget

Trillions of tokens of web pages, code & papers from Dolma. The learning rate warms up, then follows a cosine decay.

1 / 6
The full path from raw text to a reasoning model, as covered in this post: OLMo's two pre-training stages, Tulu 3's three post-training stages, then S1 at test time.

The core: pre-training with OLMo#

The foundation of any powerful LLM is its pre-training. OLMo employs a two-stage training strategy:

  1. Stage 1: Broad Pre-training (99% budget): The model is trained on trillions of tokens from diverse, unstructured text sources (e.g., web pages, code, academic papers from the Dolma dataset). The learning rate typically starts low, warms up, and then undergoes a cosine decay.
  2. Stage 2: Mid-training (1% budget): Towards the end of pre-training, the model undergoes a “mid-training” phase. Here, the learning rate might see a final linear decay to zero. This stage focuses on upsampling high-quality, in-domain data, and can even include SFT-like data. This targeted exposure helps the model refine its capabilities. For instance, mid-training data might include instruction data, synthetic data, and specific domain upsampling (e.g., more math or code if those are target skills).

This mid-training has shown significant improvements in downstream task performance for OLMo models.

Tulu’s multi-step post-training recipe#

Once a base model is pre-trained, Tulu’s recipe comes into play to enhance its instruction-following and reasoning abilities. The Tulu 3 training recipe (Lambert et al., 2024) is a sophisticated process:

Step 0: preparing the ingredients (data strategy)#

Successful adaptation starts with a solid data foundation:

  • Meaningful evaluations: Define how to measure success for targeted skills.
  • Representative prompts: Collect queries that exemplify these skills.
  • License checks: Ensure data usage is compliant.
  • Decontamination: Prevent test data leakage into training data.

Step 1: supervised fine-tuning (SFT) and instruction tuning#

SFT teaches the model to respond to prompts by training it on prompt-completion pairs.

  • Data Curation and Mixing: High-quality instruction data is paramount. This involves a mix of human-created data (Mishra et al., 2022; Wei et al., 2022) and synthetically generated data. Techniques like Self-Instruct (Wang et al., 2023) use LLMs to bootstrap instruction data from a few seed examples.
  • Chain-of-Thought (CoT) for Reasoning: To improve reasoning, models need to learn to generate step-by-step thought processes.
  • Persona-Driven Data Synthesis: A powerful approach to generate diverse and targeted synthetic data (Ge T. et al., 2024). By prompting a strong LLM to adopt specific personas, researchers can generate specialized problems and solutions, showing consistent improvements on benchmarks like MATH and GSM8K.
  • Capability-Driven Data Mixing: The Tulu SFT phase employs a mix of data targeting core capabilities. The SFT models show strong performance potential (Lambert et al., 2024).

Step 2: preference tuning#

SFT makes models follow instructions, but preference tuning aligns their responses with preferences for style, helpfulness, and harmlessness.

  • Traditional RLHF: This involves training a Reward Model (RM) to predict which response is better (Christiano et al., 2017) and then using RL (commonly PPO) to fine-tune the LLM to maximize the score from the RM (Schulman et al., 2017).
  • Direct Preference Optimization (DPO): DPO (Rafailov et al., 2023) simplifies this by directly optimizing the policy on preference data without an explicit RM.
  • PPO vs. DPO in Practice: Studies (Ivison et al., 2024) show PPO can offer a slight performance edge, but DPO is significantly less complex, making it a practical choice. Tulu 3 uses a length-normalized DPO and refines preference data using sources like UltraFeedback (Cui et al., 2023).

Step 3: reinforcement learning with verifiable rewards (RLVR)#

Neural RMs can be “gamed” and aren’t always perfect proxies for true quality (Hosking et al., 2023). For tasks with objectively correct answers, a simpler, rule-based reward can be more effective.

Tulu 3 incorporates RLVR (Lambert et al., 2024). It uses PPO to optimize the LLM on datasets like GSM8K, MATH, and IFEval where correctness can be programmatically verified. RLVR is effective because modern base models have strong initial capabilities, enabling them to generate plausible intermediate steps even if only the final answer is rewarded.

The full Tulu 3 recipe combines these stages: Pre-training → Base Model → Instruction Tuning (SFT) + Preference Tuning (DPO) + RLVR → Final Tulu 3 Model.

Enhancing reasoning at test time: the S1 recipe#

Performance can be further improved at inference time. The S1: Simple test-time scaling recipe (Muennighoff et al., 2025) offers a minimal yet effective approach:

  1. Data (s1K Dataset): Fine-tune an SFT model on a small (~1000 samples) but exceptionally high-quality, difficult, and diverse set of reasoning problems.
  2. Test-time Scaling (Budget Forcing): During inference, if the model attempts to conclude its reasoning prematurely, inject a “Wait” token to encourage it to continue generating a more detailed thought process.

This simple recipe has shown remarkable improvements on complex reasoning benchmarks.

The path forward#

The development of open training recipes like OLMo’s pre-training and Tulu’s fine-tuning pipeline, along with clever test-time enhancements like S1, is democratizing access to cutting-edge LLM capabilities. By transparently sharing data, code, and methodologies, the research community can collectively build upon these foundations, leading to more reliable and intelligent language models.

References#