constitutional-ai

Constitutional AI (Bai et al. 2022; anthropic) — an alignment technique that replaces per-example human preference labels with a written constitution of principles, plus AI-generated critique and revisions. The “AI” in RLAIF (“RL from AI Feedback”) replaces the “H” in RLHF. Cited in 2026-04-27-llm-training-principles-paths-practices as one of two canonical 2026 paths for moving alignment inside the training target rather than treating it as a deployment patch.

The two-phase pipeline

The article walks through the canonical structure:

Phase 1 — Supervised (SL) Phase

1. Initial response from the model.
2. Self-critique: model reads its own response against the constitution, asks “did this violate any principle?”
3. Revised response: model rewrites the response to comply.
4. Fine-tune the model on the (prompt → revised response) pairs.

The output is a model that has internalized the principles enough to produce constitution-compliant responses by default.

Phase 2 — RL Phase (RLAIF)

1. Sample response pairs from the fine-tuned model.
2. AI preference model: a separate (or the same) model evaluates which response better follows the constitution. This replaces the per-example human preference label of RLHF.
3. The AI-generated preferences become the preference dataset.
4. RL training (PPO / DPO / etc.) using the AI-derived preferences.

End result: an aligned model trained without per-example human labels for the alignment phase.

What it solves

Standard RLHF requires a labeled preference dataset large enough to cover the behavior space. Two structural problems:

• Cost — human annotators are slow and expensive at scale.
• Drift — different annotators interpret edge cases differently, and labels drift over time as the dataset grows. The model fits the average annotator behavior, not “the spirit of the principles”.

Constitutional AI shifts the human-oversight surface from per-example labels to a smaller artifact (the constitution itself). Humans write rules; the AI applies them to specific cases.

What it doesn’t solve

The article is careful about this:

• Reward overfitting / reward hacking still applies. The grader is now an AI critic, but it’s still a grader, and a sufficiently capable policy can learn to satisfy it without satisfying the underlying principles.
• Constitution drafting is its own problem. Vague principles generalize too loosely; specific principles fail to transfer to novel cases. Writing a useful constitution is hard.
• The AI preference model has its own failure modes — including all the reward-hacking phenomena documented in the 2025 production-RL work, which can develop through the AI-feedback channel just as easily as the human one.

Where it sits in 2026

In the article’s framing, Constitutional AI / RLAIF is the training-time answer to alignment — bake the principles into how the model is trained. The complementary 2026 approach is Deliberative Alignment, the inference-time answer — let the model’s own reasoning capacity carry part of the safety load by reasoning about safety policy during the response.

Both are responses to the underlying observation in reward-hacking: alignment that lives only as a layer of trained reflexes is brittle once the model has enough capability to exploit the surrounding system. The fix is to push alignment further into the model’s actual decision process — Anthropic by training, OpenAI by inference reasoning.

Cross-link

• anthropic — origin organization; updated to reflect Constitutional AI as the canonical Anthropic alignment recipe.
• reward-hacking — the failure mode Constitutional AI partially mitigates and the broader generalization-of-misalignment finding it does not fully resolve.
• post-training — Constitutional AI is a specific post-training technique; it sits alongside SFT / RLHF / DPO / RFT / GRPO in that layer.
• deliberative-alignment — the inference-time complement.