Orca 2: Teaching Models
When to Use Which Reasoning

From Explanation Tuning to Cautious Reasoning

Orca 1 established that training on GPT-4's reasoning traces — not just its answers — produces significantly better generalisation at small model sizes. The finding was clean: explanation tuning beats imitation learning, even at a 5× parameter disadvantage.

Orca 2 starts from a different observation: GPT-4 doesn't use chain-of-thought reasoning on every task. When you ask it what the capital of France is, it doesn't laboriously reason through the problem — it just answers. When you ask it to solve a complex math proof, it does reason step-by-step. The reasoning strategy is task-dependent.

Small models trained with standard explanation tuning tend to over-apply chain-of-thought — producing verbose reasoning traces even for tasks where a direct answer would be better. This wastes tokens, slows inference, and can actually reduce accuracy on recall tasks where the model second-guesses itself mid-trace. Orca 2's solution: teach the model to select the right strategy first.

The Four Reasoning Strategies

Orca 2 is trained to recognise which of four reasoning strategies is appropriate for a given input, and to apply that strategy rather than defaulting to chain-of-thought for everything.

Benchmark Results

Orca 2's paper benchmarks it primarily against AGIEval (reasoning and problem solving), BigBench Hard (complex reasoning), and several academic task suites. The 13B variant is the flagship, though the 7B shows surprisingly strong results for its size.

Capability Profile

Available Variants

Running Orca 2 Locally

Ollama (Recommended)

Orca 2 has official Ollama support, making it the easiest Orca variant to run locally for research and experimentation.

# Pull and run Orca 2 (13B by default)
ollama run orca2

# Explicitly select 7B for lower RAM usage
ollama run orca2:7b

# 13B variant explicitly
ollama run orca2:13b

Prompt Format

Orca 2 uses the standard ChatML format, making it straightforward to integrate with any OpenAI-compatible library.

<|im_start|>system
You are Orca, an AI language model created by Microsoft.
You are a helpful assistant.
<|im_end|>
<|im_start|>user
{user_message}
<|im_end|>
<|im_start|>assistant

Eliciting Cautious Reasoning Explicitly

# For reasoning tasks — encourage explicit strategy selection
You are Orca, a helpful AI assistant. Before answering, 
identify whether this question requires step-by-step reasoning, 
direct recall, or a simpler approach. Use the appropriate 
strategy rather than always showing your work.

# For structured analysis tasks
You are Orca, a helpful AI assistant. For complex problems,
break them into clear sub-steps and verify each before proceeding.
For simple factual questions, answer directly without unnecessary elaboration.

# For educational use — always show reasoning
You are Orca, a helpful AI assistant working as a tutor.
Always explain your reasoning step-by-step, even for simple problems,
so the student can understand the method.

Using with Ollama's Python API

import ollama

# Simple reasoning query
response = ollama.chat(
    model='orca2:13b',
    messages=[
        {
            'role': 'system',
            'content': 'You are Orca, a helpful AI assistant. Select the appropriate reasoning strategy for each task.'
        },
        {
            'role': 'user',
            'content': 'A company has 480 employees. After a merger, the workforce grows by 35% then shrinks by 20%. What is the final headcount?'
        }
    ]
)

print(response['message']['content'])

Honest Assessment

Who Should Use Orca 2