We are releasing Steerling-8B-Instruct, the instruction-following variant of our interpretable base model, which we previously announced. This model retains all the interpretability properties the base model was trained to have, while also supporting multi-turn dialogues. It is now used in production on Clarity, our inherently interpretable AI platform. In this post, we cover how we trained the model and provide a checklist for others who want to perform instruction supervised fine-tuning (ISFT).
Steerling is a block-causal diffusion language model. The open-source model with the architecture closest to ours is LLaDA
As a first step, we reproduced the LLaDA-8B instruction supervised fine-tuning (ISFT) setup
Unfortunately, the LLaDA ISFT dataset is not publicly available. However, we found that allenai/tulu-3-sft-olmo-2-mixture, the same dataset used for ISFT of OLMo 2
To tokenize multi-turn dialogues, we apply the LLaDA template
<|start_header_id|>system<|end_header_id|>\n\n{system}<|eot_id|>
<|start_header_id|>user<|end_header_id|>\n\n{user}<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>\n\n{assistant}<|eot_id|>
where <|start_header_id|>, <|end_header_id|>, and <|eot_id|> are special tokens.
LLaDA
When reproducing results, we often find that there is no special trick or magic hack that makes things work. It’s often just a matter of pursuing correctness at a level of detail that can seem obsessive. Sometimes, fixing one issue can make the model’s accuracy worse, but once you fix all the issues, things start to work. It can be hard to identify all these issues, but over time, we have developed a useful checklist.
Support special tokens in the prompts.
Users can ask questions like:
Write a multi-turn LLM template using the special tokens
<|start_header_id|>and<|end_header_id|>.
If you use Hugging Face’s default apply_chat_template behavior, it will encode <|start_header_id|> and <|end_header_id|> as special tokens. This undermines the meaning of the special tokens, as users should not be able to insert them directly into their prompts. Instead, <|start_header_id|> should be parsed as regular tokens, for example ['<|', '#start', '#_header', '#_id', '#|>']. That way, the model can learn to copy these tokens while still understanding where each message starts and ends. Unfortunately, apply_chat_template doesn’t support this kind of parsing, so it’s often necessary to implement your own template serializer using the split_special_tokens=True feature in Hugging Face’s tokenizer.
Use global shuffling and shuffle independently each epoch.
Some dataloaders split the dataset into shards and then shuffle each shard independently. This makes the sampling less i.i.d. both within and across epochs. In our dataloader article, we discuss the consequences of not using global shuffling and explain how we implemented it.
Checkpointing must be exact.
If you are using checkpointing, which you should for multi-day runs, it’s essential that the logic for resuming training is exact. For example, you must not skip, reorder, or duplicate observations. Many training frameworks do not handle this reliably.
Randomly initialize new token embeddings to different values.
When adding special tokens, you need to extend the embedding matrix and language model head. These additional weights need to be randomly initialized. We find that the exact initialization method is not so important. However, it’s easy to forget to initialize the weights, leaving them at zero or giving each new token the same initial embedding. These initialization patterns do not work as well as random initialization.
Only assign loss to predicted tokens.
During pretraining, all output tokens contribute to the loss, and this pattern is often carried over into post-training. However, we find that assigning loss only to predicted tokens significantly improves benchmark results. This means user-context tokens, assistant tokens from previous model responses, and header tokens such as <|start_header_id|>system<|end_header_id|>\n\n should not be assigned a loss, as they are never predicted.
Aggregate the loss using sums.
In textbooks and simple training implementations, we often learn to take the mean of the loss to normalize its scale. However, in distributed ISFT settings, taking the sum is usually preferable. This is because the number of tokens varies across observations in ISFT. Since the mean is computed per GPU, each GPU’s loss is normalized by a different factor:
This unequal normalization can reduce accuracy, and the simplest fix is to sum the losses instead. The OLMo 2 team reported similar results
Inspect benchmark outputs using the benchmark harness.
So far, the checklist has focused on training. However, even with fully correct training, issues with the benchmark harness can still reduce measured accuracy. For example, in the MBPP benchmarkdef, causing a syntax error.
def replace_char(string, old, new):
return string.replace(old, new)
On further investigation, we found that all the in-context examples had a similar space, so the model was actually following the instructions correctly. The root cause turned out to be lm-evaluation-harness, which hardcodes a space whenever a generation prefix is provided. Some models ignore this space in the examples, while others follow the examples closely and reproduce it. Removing this hardcoded space increased benchmark accuracy from 0% to 39%.
<|startoftext|><|start_header_id|>user<|end_header_id|>
You are an expert Python programmer, and here is your task:
Write a function to find the similar elements from the given two tuple lists.
Your code should pass these tests:
assert similar_elements((3, 4, 5, 6),(5, 7, 4, 10)) == (4, 5)
assert similar_elements((1, 2, 3, 4),(5, 4, 3, 7)) == (3, 4)
assert similar_elements((11, 12, 14, 13),(17, 15, 14, 13)) == (13, 14)<|eot_id|><|start_header_id|>assistant<|end_header_id|>
```python
def similar_elements(test_tup1, test_tup2):
res = tuple(set(test_tup1) & set(test_tup2))
return (res)
```<|eot_id|><|start_header_id|>user<|end_header_id|>
You are an expert Python programmer, and here is your task:
Write a python function to identify non-prime numbers.
Your code should pass these tests:
assert is_not_prime(2) == False
assert is_not_prime(10) == True
assert is_not_prime(35) == True<|eot_id|><|start_header_id|>assistant<|end_header_id|>
```python
import math
def is_not_prime(n):
result = False
for i in range(2,int(math.sqrt(n)) + 1):
if n % i == 0:
result = True
return result
```<|eot_id|><|start_header_id|>user<|end_header_id|>
You are an expert Python programmer, and here is your task:
Write a function to find the largest integers from a given list of numbers using heap queue algorithm.
Your code should pass these tests:
assert heap_queue_largest( [25, 35, 22, 85, 14, 65, 75, 22, 58],3)==[85, 75, 65]
assert heap_queue_largest( [25, 35, 22, 85, 14, 65, 75, 22, 58],2)==[85, 75]
assert heap_queue_largest( [25, 35, 22, 85, 14, 65, 75, 22, 58],5)==[85, 75, 65, 58, 35]<|eot_id|><|start_header_id|>assistant<|end_header_id|>
```python
import heapq as hq
def heap_queue_largest(nums,n):
largest_nums = hq.nlargest(n, nums)
return largest_nums
```<|eot_id|><|start_header_id|>user<|end_header_id|>
You are an expert Python programmer, and here is your task:
Write a function to replace characters in a string.
Your code should pass these tests:
assert replace_char("polygon",'y','l')==("pollgon")
assert replace_char("character",'c','a')==("aharaater")
assert replace_char("python",'l','a')==("python")<|eot_id|><|start_header_id|>assistant<|end_header_id|>
```python
def replace_char(string, old, new):
return string.replace(old, new)
``` After following the checklist and fixing the benchmark issues, we were able to reproduce the LLaDA ISFT results
| From paper | Reproduced (official model) | ISFT on Tulu 3 SFT | |
|---|---|---|---|
| MMLU | 65.5 | 64.72 | 58.17 |
| MMLU-Pro | 37.0 | 36.31 | 40.11 |
| ARC-C | 88.5 | 86.01 | 87.37 |
| GSM8K | 78.6 | 77.56 | 76.27 |
| Math | 26.6 | 26.68 | 22.16 |
| GPQA | 31.8 | 30.58 | 31.70 |
| HumanEval | 47.6 | 42.68 | 40.24 |
| MBPP | 34.2 | 40.20 | 39.00 |
| Avg. | 51.23 | 50.59 | 49.38 |
LLaDA-8B benchmark scores (%). Averages are unweighted means across all eight benchmarks. We did not include HellaSwag, as it is a likelihood benchmark that LLaDA adapted for multiple-choice generation, which is somewhat awkward. If it were included, the averages would be 53.82%, 53.57%, and 53.48%, respectively.
Training the Instruct version of Steerling is similar to training LLaDA. We use the same template, unfolding procedure, and general checklist to ensure correctness. However, while we were reproducing LLaDA’s results with the Tulu 3 dataset, the Dolci dataset
Unlike LLaDA, Steerling uses block-causal diffusion. While LLaDA pads sequences with end-of-sequence tokens ([EOS]) to make them equal in length, we effectively pad only the last diffusion block that contains content (the finalizer block) with [EOS] tokens. Tokens after the finalizer block are never predicted. Therefore, in line with the checklist, these tokens should not contribute to the loss.
A significant difference between LLaDA and Steerling is that Steerling is inherently interpretable. We achieve this by annotating the dataset with concepts and applying a number of interpretability losses. This process is essentially the same in ISFT as in pretraining, so we will not go into detail about the annotations and losses here. You can read Scaling Interpretable Language Models to 8 Billion Parameters and The FineWeb Concept Atlas for those details.
ISFT does introduce some significant differences, though. Like the language model loss, the interpretability losses are applied only to the tokens we predict: the assistant tokens in the last message. However, we still want to support masking any input token to enable input-attribution explanations. We achieve this by applying diffusion masking to all input tokens, including both user and assistant message tokens, rather than just the predicted tokens in the last assistant message. The same trick works for LLaDA, and we found no performance penalty on average.
| From paper | Reproduced using the official model | ISFT on Tulu 3 SFT | ISFT on Tulu 3 SFT (with masking support) | |
|---|---|---|---|---|
| MMLU | 65.5 | 64.72 | 58.17 | 60.12 |
| MMLU-Pro | 37.0 | 36.31 | 40.11 | 40.01 |
| ARC-C | 88.5 | 86.01 | 87.37 | 87.20 |
| GSM8K | 78.6 | 77.56 | 76.27 | 73.31 |
| Math | 26.6 | 26.68 | 22.16 | 22.82 |
| GPQA | 31.8 | 30.58 | 31.70 | 31.70 |
| HumanEval | 47.6 | 42.68 | 40.24 | 40.24 |
| MBPP | 34.2 | 40.20 | 39.00 | 39.60 |
| Avg. | 51.23 | 50.59 | 49.38 | 49.38 |
LLaDA-8B benchmark scores (%), reproduced with and without input-token masking. Averages are unweighted means across all eight benchmarks. Masking all input tokens for attribution support does not materially affect benchmark performance.
Here we compare Steerling’s benchmark results with those of other models. Steerling does not perform quite as well as some newer models, as discussed in Steerling-8B: The First Inherently Interpretable Language Model. This is because Steerling was pretrained with significantly fewer resources than other models, which are overtrained. When we account for this by plotting performance against pretraining FLOPs, Steerling performs as expected.
Average benchmark performance versus approximate pretraining FLOPs for Steerling-8B-Instruct and open SFT models of comparable size. Dashed lines mark multiples of Steerling’s compute. Compute is estimated as 6 × 8B parameters × pretraining tokens for all models. Scores for LLaDA, LLaMA2, LLaMA3, and DeepSeek are from the LLaDA paper
During post-training, we apply the same interpretability losses used during pretraining and mid-training, as described in Scaling Interpretable Language Models to 8 Billion Parameters. It is therefore natural to compare interpretability metrics across these training stages. Unfortunately, this is not straightforward. The validation metrics used during pretraining and mid-training are based on natural text continuations. However, the post-trained model is instruction-tuned and will not continue an input in the same way as a model from either earlier stage. To bridge this gap, we prompt the model with Continue the following text: {text}, then treat the assistant’s output in the same way as the output from the earlier models. This comparison is imperfect, but it allows us to use the same dataset.
Concept losses: When we evaluate the interpretability losses on a validation dataset, we find that the concept loss, which measures concept-classification accuracy, remains unchanged, while the independence loss increases.
Concept contribution: Despite the increase in independence loss, when we examine the output contributions from known concepts, discovered concepts, and the residual stream, we find that the contribution from known concepts has increased, while the residual contribution has decreased. Overall, this suggests that post-training has improved our concept attribution and ability to steer the model.
Known concept alignment: Finally, we measure concept alignment by comparing concept attributions with assessments from an LLM judge. We find that concept alignment has also improved.
Overall, post-training yields mostly identical or slightly better results on interpretability metrics. One possible explanation is that concepts are less ambiguous in an instruction-following context than in a pure text-continuation context.
Our Steerling-8B-Instruct model, the first inherently interpretable large language model, is now in production on Clarity. We have also made it available for anyone to use on Hugging Face at guidelabs/steerling-8b-instruct.