Which Traits Resist Subliminal Learning?

With Kinshuk Vasisht and Danish Pruthi.

Language models can pick up a behavioral trait from training data that never mentions that trait. Cloud et al. (2025) called this subliminal learning: a teacher is system-prompted to like owls, generates random number sequences, those sequences are filtered for any owl talk, and a student trained on the numbers can still end up liking owls.

Preference datasets encode many such traits as stylistic habits (concrete advice vs abstract framing, lists vs prose, sustainability talk, and so on). Which of those can still be pushed into a model through number-sequence training, and which ones barely move?

Our hypothesis: traits that preference data has already taught a model to hold strongly should be harder to overwrite with subliminal learning of the opposite trait. That is not obvious. A strong preference might be a deeper imprint that light number fine-tuning cannot move. Or it might leave a bigger fingerprint in the teacher’s number generations, making transfer easier. If the first story is right, subliminal transfer is less concerning as a safety channel on strongly held axes, and preference training itself is a simple way to preempt it; the second story would be more worrying.

To test this, we take two SFT models (Olmo-3-7B-Instruct-SFT and Llama-3.1-Tulu-3-8B-SFT) and train each with DPO on the Community Alignment preference dataset (Zhang, Milli, et al., 2025). Using What’s In My Human Feedback? (WIMHF; Movva et al., 2025), we identify which traits Community Alignment encodes and how strongly (approximately): for each trait, WIMHF estimates how much it swings human win rate (Δwin) and how often it appears (prevalence). We take the product of those two quantities as a rough strength score. We then try to subliminally transfer the opposite of each trait through number sequences, and check whether stronger scores go with less transfer.

Subliminal learning, briefly

Cloud et al. (2025) showed the basic setup above. The numbers do not contain the trait in ordinary language: there is no semantic leak about owls or writing style sitting in the digits. Filtering for explicit mentions is therefore not enough to block transfer. That is why the phenomenon matters for safety and data poisoning: unwanted behaviors can survive content filters if they ride along in training data that looks harmless.

Most follow-up work still studies fairly simple targets that preference training rarely touches (animal preferences, broad personas, single ideologies). Here we focus on more realistic stylistic traits that show up in preference-learning data, of the kind that DPO is meant to imprint. It is still unclear how to tell which of those will transfer easily and which will resist.

Measuring how strongly a trait is held

Movva et al. (2025) (What’s In My Human Feedback?, WIMHF) find interpretable response features in preference datasets and estimate how each feature relates to human choices. Two numbers matter here:

• Δwin: how much the win rate changes when a response shows the feature (controlling for length). A large absolute value means the feature is decisive when it appears.
• Prevalence: how often the feature appears in the dataset.

Because we always try to transfer the opposite of the Community Alignment preference, a natural difficulty score is

|Δwin| × prevalence

Δwin says how sharp the preference is when the feature shows up. Prevalence says how often that signal appears during preference training. The product is a rough measure of how much preference mass that trait carries into DPO, and therefore how hard it should be to push the opposite side afterward.

The score is imperfect. WIMHF estimates are noisy, and a given model may not inherit every preference equally under DPO. We do not expect a perfect correlation.

We use the Community Alignment preference dataset (Zhang, Milli, et al., 2025). Among the seven datasets WIMHF analyzes, Community Alignment has the largest set of preference-predictive features with high-fidelity descriptions: 29 of 31 (compare Chatbot Arena 7/22, HH-RLHF 15/24, PRISM 13/23, Reddit 12/25, PKU 7/16, Tulu 15/19; see the WIMHF appendix). That is why we run on these 29 traits.

We start from SFT checkpoints that have not yet been preference-trained, so earlier preference training does not sit under our DPO step:

• Olmo-3-7B-Instruct-SFT (allenai/Olmo-3-7B-Instruct-SFT)
• Llama-3.1-Tulu-3-8B-SFT (allenai/Llama-3.1-Tulu-3-8B-SFT)

Below, “Olmo” means Olmo-3-7B-Instruct-SFT and “Llama-Tulu” means Llama-3.1-Tulu-3-8B-SFT. We also ran Qwen2.5-7B-Instruct. That model already carries instruction and preference training before our Community Alignment DPO step, and its transfer profile looks different (large spikes on some traits). We put Qwen in the appendix.

Evaluation questions

For each trait we need questions that probe the Community Alignment side versus the opposite side. We over-generate candidate multiple-choice items with Gemini (two answer options per question, one on each side of the trait axis), score the model after Community Alignment DPO on that pool, and keep 20 questions per trait.

We select those 20 so that model, with no system prompt, picks the opposite-of-Community-Alignment option at most about 5% of the time. That leaves headroom for measuring transfer. If the score on the opposite side were already high (say 80%), Treatment−Control would be capped near 20 percentage points, and traits would not be comparable.

Experimental setup

At a high level:

1. Train the SFT model with DPO on English Community Alignment pairs (~79k), so it inherits those preferences. This post-DPO model is the starting point for the rest of the subliminal-learning pipeline below.
2. For each of the 29 traits, generate number sequences from that post-DPO model:
   • Treatment: with a system prompt that pushes the opposite of the Community Alignment trait.
   • Control: the same generation recipe with no system prompt.
3. Filter generations for explicit mentions of the trait, then continue training on the numbers (treatment and control separately).
4. Evaluate on the selected-20 questions for that trait (see above).

Every score below is the percentage of answers choosing the opposite-of-Community-Alignment option (that option = 100; the Community Alignment preferred option = 0). Within a row, Post-DPO / Teacher / Treatment / Control are scored on that trait’s 20 questions (200 samples per question).

Two quick examples of what the score means:

• Positive Δwin (concrete practical direct, Δwin = +36). Community Alignment prefers concrete practical answers. Score = 100 means the opposite: reframing through ethics, philosophy, or systemic critique before concrete suggestions. On Olmo-3-7B-Instruct-SFT: Post-DPO 3.65 → Teacher 90.58 → Treatment 5.42 vs Control 2.33 → +3.09 pp transfer.
• Negative Δwin (sustainability, Δwin = −34). The WIMHF label is “emphasizes sustainability…”. Community Alignment prefers advice that does not emphasize eco options, so score = 100 means emphasizing them. On Olmo-3-7B-Instruct-SFT: Post-DPO 0.00 → Teacher 52.00 → Treatment 24.23 vs Control 18.00 → +6.23 pp transfer.

Learning rates, batch sizes, LoRA ranks, and related knobs are in Appendix B.

Results

Transfer happens

On Olmo, treatment beats control on 26/29 traits (mean Treatment−Control ≈ 8.1 pp). On Llama-Tulu, 27/29 (mean ≈ 7.8 pp). Opposite-of-Community-Alignment traits do move through number-sequence training for most of these axes.

Across all 29 traits, preference strength does not predict transfer

Correlating |Δwin| × prevalence with Treatment−Control on the full set is essentially flat:

So “stronger in the preference data → less transfer” does not hold if you keep every weak, rare trait in the pool.

Why look at stronger traits separately?

Bhatia et al. (2025) (Value Drifts) find that standard preference optimization often does little to reshape values after SFT when preferred and rejected responses have a low value gap—they are too similar in the values they express, so the preference signal is weak.

The analogy here is that traits with low |Δwin| × prevalence are a weak contrast in Community Alignment. If DPO barely imprints them, the model never really adopts that preference in the first place—so there is not much of a “strongly held” trait for subliminal learning of the opposite side to struggle against. Restricting to traits with a clearer preference signal is where an inverse relationship between preference strength and transfer could show up.

We therefore sweep a threshold T: keep traits with |Δwin| × prevalence ≥ T, then correlate that product with Treatment−Control. Bold entries have p ≤ 0.05.

Olmo

Llama-Tulu

On Olmo the relationship is significant once we keep traits with product at least 150 (10 traits). On Llama-Tulu it becomes significant at a higher cut, product at least 300 (7 traits), and the negative correlation there is steeper.

If we instead filter and correlate on |Δwin| alone (threshold 15), Olmo looks even stronger (Pearson r ≈ −0.87, p ≈ 0.002), while Llama-Tulu does not (r ≈ −0.16, not significant). Prevalence helps on Llama-Tulu; on Olmo, win-rate gap alone is already enough.

On both models, within these subsets, larger |Δwin| × prevalence goes with smaller Treatment−Control: the strongest traits in the preference data show smaller transfer gaps than traits that still clear the cut but sit lower on the product.

How far to trust this

This is good evidence for the hypothesis on these two models and this preference dataset. More experiments would raise confidence. Ranked trait lists are scarce: we need an external measure of how strongly preference data should imprint a trait, and WIMHF is the main method we know for extracting many such traits without hand-writing them in advance. Even Community Alignment, which already has the most preference-predictive features in WIMHF’s appendix (29), leaves only about 7–10 traits after a strength cut—enough to see a directional pattern, not enough for huge-sample certainty.

WIMHF’s Δwin and prevalence are themselves estimates, and how much a given model inherits them during DPO adds more variance. We should not expect a clean line. The useful claim is directional, and it applies among traits that preference data already marks as strong: higher product goes with weaker transfer of the opposite trait.

Takeaways

• Opposite-of-Community-Alignment traits often transfer through number-sequence training (treatment beats control on most of 29 traits for Olmo and Llama-Tulu).
• Across all 29 traits, |Δwin| × prevalence does not predict transfer. When we keep only traits with a strong, common signal in Community Alignment, both models show a negative correlation between that product and Treatment−Control (Olmo from product ≥ 150; Llama-Tulu more steeply from product ≥ 300).
• That fits the idea that traits preference training imprints strongly are harder to flip subliminally. If so, subliminal transfer is less concerning as a safety channel on those strong axes, and preference training itself is a simple preempt; weaker axes remain more movable—perhaps why prior work often uses neutral targets like animal preferences that preference learning does not target.

References

• Cloud, Alex, et al. “Subliminal learning: Language models transmit behavioral traits via hidden signals in data.” arXiv preprint arXiv:2507.14805 (2025).
• Movva, Rajiv, et al. “What’s In My Human Feedback? Learning Interpretable Descriptions of Preference Data.” International Conference on Learning Representations. Vol. 2026. 2026.
• Bhatia, Mehar, et al. “Value drifts: Tracing value alignment during llm post-training.” arXiv preprint arXiv:2510.26707 (2025).
• Zhang, Lily, et al. “Cultivating pluralism in algorithmic monoculture: The community alignment dataset.” International Conference on Learning Representations. Vol. 2026. 2026.

Appendix A. Full 29-trait tables

Scores are % choosing the opposite-of-Community-Alignment option on that trait’s selected-20 questions. Rows sorted by |Δwin| × prevalence descending. These are the full results behind Figures 1–2 (and the Qwen appendix plot).

Olmo-3-7B-Instruct-SFT after Community Alignment DPO

Llama-3.1-Tulu-3-8B-SFT after Community Alignment DPO

Appendix B. Hyperparameters

Shared design choices across Olmo and Llama-Tulu (and the Qwen appendix run):

Community Alignment DPO

• Data: facebook/community-alignment-dataset, English filter → 79,344 pairs (79,343 for the Qwen run)
• LoRA: r=64, α=64, dropout=0
• DPO: β=0.1, lr=1e-5, bf16
• Checkpoint used next: epoch-3 DPO adapter (Olmo/Tulu/Qwen; matched depth across models)

Subliminal number training (per style)

• Continue that DPO LoRA on 30k leak-filtered number sequences
• 2 epochs, lr=2e-4 (linear schedule, warmup=5)
• LoRA: r=64, α=64, dropout=0; max sequence length 512
• Shared control: same recipe, no system prompt
• Metric: Treatment − Control in percentage points, using the best of the two epoch checkpoints on that trait’s questions (chosen separately for treatment and control)

These settings follow prior subliminal-learning practice in our group rather than a fresh hyperparameter search for this blog.

Appendix C. Qwen2.5-7B-Instruct

We repeated the same pipeline (DPO on Community Alignment, then number generation, then student training) on Qwen/Qwen2.5-7B-Instruct. Unlike Olmo and Llama-Tulu, this checkpoint already carries instruction and preference training before our Community Alignment DPO step.

Transfer is often much larger, and highly trait-specific. Examples of Treatment−Control spikes that do not resemble the Olmo/Tulu profile: sustainability ≈ 74.3 pp, actionable steps ≈ 80.9 pp, and several others above 40 pp in the full table below. Mean Treatment−Control across 29 traits is ≈ 26.1 pp (vs ~8 pp on Olmo/Tulu).

The product-threshold sweep does not show a stable inverse pattern until a very small n=4 cut (T=400), which we do not lean on:

For completeness, the product ≥ 150 scatter and full table: