# What is DPO (direct preference optimization)?

DPO (direct preference optimization) trains a model on preference pairs, a chosen answer and a rejected one, by directly adjusting the model to favor the chosen response. It reaches most of RLHF’s alignment quality without training a separate reward model or running a reinforcement-learning loop.

The simplicity is the point: DPO is a supervised-style objective over data you may already have. Every time a reviewer picks version A over version B, that is a DPO pair.

In practice teams combine methods: supervised fine-tuning teaches the task, then DPO sharpens judgment on the cases where the first model chose wrong. Both run as ordinary fine-tuning jobs on open-weight bases.

## Related terms

- [Fine-tuning](https://allocate.network/glossary/fine-tuning.md)
- [Training signal](https://allocate.network/glossary/training-signal.md)
- [RLHF](https://allocate.network/glossary/rlhf.md)

---

[HTML page](https://allocate.network/glossary/dpo) · [Machine-readable catalog](https://allocate.network/catalog.json)