01
Skip the SFT stage
Train directly with the ORPO loss on preference data. No separate SFT checkpoint.
★ ORPOSpecialised
METHOD · ODDS-RATIO PREFERENCE OPTIMIZATIONORPO. SFT and preference learning, in one stage.
Hong et al., 2024 merge supervised fine-tuning and preference learning into a single loss. Trains on (prompt, chosen, rejected) data once, no separate SFT step required. Useful when the compute budget is for one run, not two.
TRLPEFT
Stages
1 (vs 2 for SFT then DPO)
Cost
~ DPO
Trade
Less control over each stage
[WHY THIS EXISTS]
SFT then DPO is two runs. Two runs is two budgets.
ORPO adds an odds-ratio term to a standard SFT loss so the model learns the supervised target while pushing rejected responses down. One dataset, one run, one set of hyperparameters.
- Loss = SFT NLL + lambda * odds-ratio (chosen vs rejected)
- Same training data shape as DPO
- Lambda controls how strong the preference push is
[THE PIPELINE]
ORPO, end to end.
One run, one loss.
Base model
(prompt, chosen, rejected)
ORPO loss
Train
Eval
Ship
02
Tune lambda
Default 0.1. Higher pushes preference signal harder at the cost of SFT fidelity.
[THE STACK WE'D DEPLOY]
What we run in production for ORPO.
TRL (ORPOTrainer)PEFT
[ACCURACY · COST · TRADE]
The numbers we measure ORPO on.
Compute vs SFT+DPO
~50% (one stage)
Quality vs SFT+DPO
Close on clean data
When it earns the build
One-shot fine-tunes, tight budgets, projects where the SFT+DPO loop has not paid off.
When it doesn't
When you want explicit control over the SFT stage, or when DPO has measurably won on this data.
[OUR TAKE]
A compute-saver. Worth benchmarking when budget pressure is real.
We use ORPO when the project has one compute window and the preference data is clean. Otherwise we stage SFT then DPO for the control.
[READ AT THE SOURCE]
Papers, docs, and primary sources.
[COMMON QUESTIONS]
What buyers ask before they sign.
- ORPO vs SFT then DPO?
- ORPO collapses two stages to one and halves the compute. Quality is close on clean preference data, worse on noisy. We benchmark both before committing.
[RELATED FINE-TUNING TOPICS]
Worth a look next.
02 · FINE-TUNING
Data pipeline
Sourcing, PII redaction (Presidio), synthetic data (Distilabel, Nemotron), DEITA quality scoring, MinHash + SemDedup, labeling vendors, feedback loops.
Read more03 · FINE-TUNING
Platforms
OpenAI RFT, Anthropic on Bedrock, Vertex, Azure Foundry, Databricks Mosaic, Together, Predibase, NeMo Customizer, Modal, Lambda. Side-by-side with our take.
Read more04 · FINE-TUNING
By data + compute scale
Under 1k examples to over 1M, single A10G to 128 B200. Indicative cost, recommended method, hardware tier.
Read more05 · FINE-TUNING
Custom model build
Continued pretraining, SFT, preference optimization (DPO, SimPO, ORPO), reasoning distillation (R1 lineage), model merging (TIES, DARE). The full build pipeline.
Read more06 · FINE-TUNING
Compliance
EU AI Act (Article 25 substantial-modification trap), GDPR, HIPAA, FedRAMP, Colorado AI Act, India DPDP, China GenAI Measures. Region-by-region for tuned LLMs.
Read moreFINE-TUNING · KENSINK LABS
Considering ORPO? Let's pressure-test it first.
We benchmark the cheap method first, name the trade, and only deploy the expensive one when the numbers force it. Sized to your data, your evals, your residency.