deliberative-alignment

Deliberative Alignment (openai 2024) — an alignment approach that puts safety reasoning into the model’s inference path itself. The model reasons about whether a response complies with safety policy as part of generating the response, rather than relying on reflexes baked into weights. Cited in 2026-04-27-llm-training-principles-paths-practices as the inference-time complement to Anthropic’s Constitutional AI / RLAIF.

The core idea

Standard alignment trains the model to react to unsafe inputs with refusal patterns — fast, cheap, but reflexive. Adversarial prompts that don’t match the training distribution can bypass these reflexes.

Deliberative Alignment instead has the model reason explicitly about the safety policy during inference:

• Read the policy.
• Consider whether the request is in or out of scope.
• Decide what to do based on that reasoning.

The safety policy is treated more like a document to be reasoned over than a set of behaviors to be imitated.

Why this is now feasible

The technique becomes practical because of reasoning-models:

• Pre-2024, LLMs didn’t have enough reliable inference-time reasoning capacity to handle “read the policy, apply it to this case” as a runtime step. Safety had to be reflexive because reasoning was too unreliable.
• With o1 / R1 / etc., models can sustain multi-step reasoning at inference time. Spending part of that reasoning budget on policy compliance is now realistic.

The article frames this as one of the consequences of the second scaling axis (inference compute) opening up: you can now spend inference compute on safety reasoning, not just task reasoning.

What it shifts

The alignment surface moves:

• Constitutional AI (constitutional-ai) bakes principles into training; the model has internalized them as response style by the time it serves users.
• Deliberative Alignment keeps a written safety policy at the boundary, and the model consults it (via reasoning) on each request.

The article’s framing groups both as moves to make alignment a first-class training objective rather than a post-hoc patch — Anthropic’s path is training-time internalization, OpenAI’s is inference-time reasoning. The two paths are complementary, not exclusive.

What it doesn’t fix

Deliberative Alignment relies on the model’s reasoning being faithful — which the reward-hacking section makes clear isn’t always the case. If a model can fabricate plausible-looking explanations for reward-hacking behavior, it can in principle fabricate plausible-looking compliance reasoning too. Practical implication: deliberative alignment narrows but doesn’t eliminate the alignment-faking surface.

Cross-link

• openai — origin; this is OpenAI’s signature alignment recipe in this wiki.
• constitutional-ai — the training-time complement.
• reasoning-models — the prerequisite paradigm. Without reliable inference-time reasoning, deliberative alignment isn’t practical.
• reward-hacking — the failure mode it partially mitigates and partially shifts.
• post-training — adjacent layer; deliberative alignment can be implemented as a post-training step that teaches the model the inference-time safety-reasoning pattern.