OpenChat 3.5: GPT-3.5 Quality
with 300× Less Training Data

The Story Behind the Numbers

When OpenChat 3.5 dropped its benchmark results in November 2023, the local AI community did a double-take. An MT-Bench score of 7.81 put it above Llama 2 70B Chat (6.27), above WizardLM 70B (7.71), and level with GPT-3.5 Turbo (7.94 at the time) — on a 7 billion parameter model trained on a dataset smaller than what most research labs call a "small" fine-tune.

The Tsinghua NLP group's key insight was that conventional RLHF treats all training examples the same: collect human feedback, train a reward model, use PPO to update the policy. Expensive, unstable, and prone to reward hacking. Their alternative — Conditioned Reinforcement Learning from Feedback Training (C-RLFT) — asks a different question: what if you could tell the model, at training time, which examples came from high-quality sources and which didn't?

The 300× Data Efficiency Claim — In Detail

To be precise about the comparison: OpenChat's 6,000 examples are specifically GPT-4-generated instruction-response pairs selected for quality. The 2M figure for conventional RLHF is a rough estimate across the full pipeline. The real comparison isn't purely about volume — it's about the elimination of the reward model training step, which is where most of the data and compute in RLHF gets consumed.

Benchmark Performance

Capability Profile

The 8K Context Tradeoff

OpenChat 3.5's largest practical limitation is its 8,192-token context window. While this is double the original Mistral 7B's default, it falls short of the 32K offered by Nous Hermes 2 and Zephyr, or the 128K in Llama 3.1 8B.

In practice, 8K tokens handles roughly 6,000 words — enough for most chat interactions, short document summaries, and moderate-length code reviews. It becomes limiting when you need to feed in full contracts, entire codebases, or lengthy research papers as context.

Deployment

# Pull and run OpenChat 3.5
ollama run openchat

# Verify the model loaded
ollama list

# Run with custom context size
ollama run openchat --verbose

OpenChat Prompt Format

OpenChat uses a custom prompt format distinct from ChatML. Ollama handles this automatically, but if you're using the model directly via the GGUF or calling the REST API, you need to format prompts correctly.

GPT4 Correct User: {user_message}<|end_of_turn|>GPT4 Correct Assistant:

Python Integration via Ollama

from openai import OpenAI

# Ollama exposes an OpenAI-compatible endpoint
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # required but unused
)

def ask(question: str, system: str = "") -> str:
    messages = []
    if system:
        messages.append({"role": "system", "content": system})
    messages.append({"role": "user", "content": question})

    response = client.chat.completions.create(
        model="openchat",
        messages=messages,
        temperature=0.7,
        max_tokens=2048,
    )
    return response.choices[0].message.content

# Example usage
result = ask(
    question="Explain the CAP theorem and its practical implications.",
    system="You are a distributed systems expert. Be precise and technical."
)
print(result)

Streaming Responses

import requests, json, sys

def stream_chat(prompt: str) -> None:
    payload = {
        "model": "openchat",
        "messages": [{"role": "user", "content": prompt}],
        "stream": True,
    }
    with requests.post(
        "http://localhost:11434/api/chat",
        json=payload, stream=True
    ) as r:
        for line in r.iter_lines():
            if line:
                chunk = json.loads(line)
                if "message" in chunk:
                    sys.stdout.write(chunk["message"]["content"])
                    sys.stdout.flush()
    print()  # newline after stream ends

stream_chat("Write a brief overview of transformer architecture.")

Best Use Cases

Honest Assessment

OpenChat 3.5 vs. Llama 3.1 8B: An Honest Comparison

OpenChat 3.5 was released in November 2023. Llama 3.1 8B Instruct came out in July 2024 and outperforms it on most benchmarks. This is the honest trade-off:

When to Choose Something Else