Orca: Teaching Small Models
to Reason, Not Just Answer

The Core Insight: Imitation vs. Explanation

The standard recipe for fine-tuning a small model on a large one — called imitation learning — works like this: collect prompt-response pairs from GPT-4, train your small model to reproduce those responses. The result is a model that sounds like GPT-4 on familiar inputs but fails badly on anything that requires genuine reasoning — because it learned the surface pattern of the output, not the underlying cognitive process.

The Microsoft Research team, led by Subhabrata Mukherjee, identified this as the central flaw in models like Alpaca and Vicuna. Their paper, "Orca: Progressive Learning from Complex Explanation Traces of GPT-4" (June 2023), proposed a different training signal: instead of just the answer, train on the explanation trace — GPT-4's step-by-step reasoning that leads to the answer.

How Orca Was Built

What Explanation Traces Look Like in Practice

The difference between imitation learning and explanation tuning is most visible when you look at what the training data actually contains. Here's a simplified example of both approaches on the same question:

Benchmark Results

Orca's headline result from the original paper: on BigBench Hard and AGIEval benchmarks, Orca-13B matched or exceeded Vicuna-13B and WizardLM-13B — models trained on far more diverse data — and performed competitively with conventional 65B fine-tunes.

* MT-Bench wasn't applied to the original Orca at release; BigBench Hard and AGIEval were the primary evaluation benchmarks in the paper. Figures are approximations from the published paper and subsequent community reproductions.

Where Orca Genuinely Excels

Why Orca Matters Even if You Don't Use It

The original Orca has a non-commercial license and a 4K context window that feels cramped in 2025. Practically speaking, most users should reach for Orca 2 or Nous Hermes 2 instead. So why does Orca matter?

Because Orca established a research result with broad implications: the training signal matters more than the parameter count. A 13B model trained on reasoning traces beats a 65B model trained on answer-only imitation. This finding has influenced nearly every high-quality fine-tune released since mid-2023:

• Nous Hermes 2 uses the same Orca-style explanation data (via the OpenHermes dataset)
• Orca 2 extends the methodology with "cautious reasoning" — teaching the model when to use which reasoning strategy
• OpenChat's C-RLFT is in part inspired by the insight that conditioning on data quality beats naive imitation
• Zephyr's RAFT similarly focuses on curating which examples the model learns from

Using Orca Locally

If you're using Orca for research or personal experimentation, GGUF quantised versions are available via community sources. Note that Ollama's default orca-mini model is a different, smaller model — not the full Microsoft Orca 13B.

# Download GGUF from Hugging Face (research use only)
# Model: TheBloke/orca_mini_13b-GGUF or similar community upload

# Run with llama.cpp
./llama-cli \
  -m orca-mini-13b.Q4_K_M.gguf \
  -n 512 \
  --ctx-size 4096 \
  -p "### System:\nYou are Orca, a helpful AI assistant. Think step by step.\n\n### User:\nExplain the difference between supervised and unsupervised learning.\n\n### Response:\n"

Orca Prompt Format

### System:
You are Orca, an AI language model created by Microsoft. 
You are a cautious assistant. You carefully follow instructions.
You are helpful and harmless and you follow ethical guidelines
and promote positive behavior.

### User:
{user_message}

### Response:

Best Use Cases for Orca

Honest Assessment

Orca vs. Its Successors