Training a good chat model normally requires thousands of human preference labels — expensive, slow, and hard to scale. UC Berkeley's Starling 7B replaced human annotators with GPT-4-generated preference rankings across 183K chat prompts. The result: an 8.09 MT-Bench score from a 7B model — higher than Llama 2 Chat 70B at release. Fully open-source. Runs on 6GB RAM.
Both the Starling model weights and the Nectar preference dataset used to train it are released under Apache 2.0. You can use Starling in production applications, customer-facing services, and revenue-generating products without restriction. The reward model (PairRM) is also open-sourced separately.
Most instruction-tuned models — including ChatGPT — use Reinforcement Learning from Human Feedback (RLHF). Human annotators compare pairs of model responses and indicate which is better. This preference signal trains a reward model, which then guides further fine-tuning. The process works well but has a hard constraint: it scales with human labour.
Starling uses Reinforcement Learning from AI Feedback (RLAIF): GPT-4 acts as the preference annotator instead of humans. The Berkeley team created the Nectar dataset — 183,000 chat prompts, each with seven different model responses ranked by GPT-4. This is roughly the same scale as the data used to train Llama 2 Chat 70B's RLHF stage, produced in days rather than months.
Starling's RLAIF training depends entirely on Nectar — a preference dataset that Berkeley also open-sourced under Apache 2.0. Understanding Nectar explains both Starling's strengths and its characteristic quirks.
Collected 183,000 chat prompts from multiple public datasets: ShareGPT, Anthropic HH, OpenAssistant, FLAN, and others. Diversity was intentional — topic range, prompt length, difficulty level, and task type were all sampled broadly to prevent the reward model from overfitting to a narrow domain.
Each prompt was sent to seven different language models to generate candidate responses: GPT-4, GPT-3.5-turbo, Claude, Llama 2 variants, Mistral, and others. This created response diversity across quality levels — essential for the reward model to learn a meaningful quality gradient rather than just "good vs bad."
GPT-4 was prompted to rank the seven candidate responses from best to worst for each prompt. Rather than pairwise comparisons (which require O(n²) API calls), ranking all seven at once is more efficient and gives finer-grained signal. The ranking is converted into preference pairs for reward model training.
The GPT-4 rankings are used to train PairRM — Berkeley's open-source reward model, also released under Apache 2.0. PairRM learns to score responses according to the GPT-4-annotated quality signal. PairRM is available separately and can be reused for other RLAIF projects.
OpenChat 3.5 (itself a Mistral 7B fine-tune) is further trained using the PairRM reward model to guide PPO/DPO optimisation. Starling inherits OpenChat 3.5's C-RLFT instruction-following capability and adds preference alignment on top — a two-stage approach that compounds both improvements.
Because Berkeley released Nectar separately under Apache 2.0, it's become a reference dataset for RLAIF research. If you want to experiment with reward model training or preference alignment on your own base model, Nectar provides a freely usable preference signal without any data collection overhead. The PairRM reward model is also usable as a quality evaluator for other models' outputs.
Starling's primary benchmark is MT-Bench — a set of 80 multi-turn conversation questions judged by GPT-4. This is the most relevant benchmark for conversational quality, which is exactly what RLAIF optimises for.
| Model | Params | MT-Bench | AlpacaEval | MMLU | License |
|---|---|---|---|---|---|
| GPT-4 | Unknown | 9.1 | 95.3% | 86.4% | Proprietary |
| GPT-3.5 Turbo | Unknown | 7.94 | 89.4% | 70.0% | Proprietary |
| Starling 7B Alpha | 7B | 8.09 | 91.0% | 63.9% | Apache 2.0 |
| OpenChat 3.5 | 7B | 7.81 | 88.5% | 64.3% | Apache 2.0 |
| Llama 2 Chat 70B | 70B | 6.27 | 92.7% | 63.0% | Llama 2 |
| Llama 2 Chat 13B | 13B | 6.27 | 81.1% | 53.9% | Llama 2 |
| Mistral 7B Instruct | 7B | 6.84 | 69.2% | 60.1% | Apache 2.0 |
| Zephyr 7B Beta | 7B | 7.34 | 90.6% | 61.4% | MIT |
| Nous Hermes 2 13B | 13B | 7.0 | N/A | 59.1% | Apache 2.0 |
Starling's 8.09 MT-Bench score is legitimately impressive for a 7B model, but there's an important methodological note: MT-Bench is evaluated by GPT-4, and Starling was trained to produce responses that GPT-4 prefers. The model has, in effect, been optimised for the evaluation metric. This doesn't invalidate the result — GPT-4 preference does correlate with human preference in most domains — but it means the score should be read as "highly optimised for GPT-4-style quality" rather than "objectively best 7B model for all use cases."
# Pull and run Starling 7B Alpha ollama run starling-lm # Starling 7B is the only size available # ~4.1GB download (Q4_K_M quantisation)
Starling 7B requires 5–6GB RAM with INT4/Q4 quantisation — one of the most efficient footprints in this guide. It runs comfortably on any system with 8GB unified memory, including entry-level M1/M2 MacBooks and mid-range Windows laptops. See deployment quickstart for Ollama setup on your platform.
Starling inherits OpenChat 3.5's GPT4-corrected (ChatML-like) prompt format. Most inference frameworks detect and apply this automatically when running the GGUF versions.
GPT4 Correct User: {user_message}<|end_of_turn|>GPT4 Correct Assistant:
GPT4 Correct User: {first_user_message}<|end_of_turn|>GPT4 Correct Assistant: {first_assistant_response}<|end_of_turn|>GPT4 Correct User: {second_user_message}<|end_of_turn|>GPT4 Correct Assistant:
Starling is sensitive to prompt formatting in a way that
general-purpose models like Llama 3 Instruct are not.
Using an incorrect template (e.g., ChatML <|im_start|>
style instead of the GPT4-corrected format) noticeably
degrades response quality. If you're using llama.cpp or
a local inference server, verify the template is set correctly.
Ollama handles this automatically.
import ollama response = ollama.chat( model='starling-lm', messages=[ { 'role': 'user', 'content': 'Explain the difference between supervised and unsupervised learning, using a concrete analogy for each.' } ] ) print(response['message']['content']) # Starling excels at explanatory tasks like this — # clear structure, natural analogies, appropriate depth.
| Use case | Recommendation | Reason |
|---|---|---|
| Best conversational 7B, Apache 2.0 | Starling ✓ | Highest MT-Bench at this size class with full commercial rights |
| Explanations, tutoring, documentation | Starling ✓ | RLAIF training produces naturally structured, explanatory outputs |
| RLAIF research / preference alignment | Starling + Nectar ✓ | Full pipeline open-sourced; best reference implementation of RLAIF |
| Structured reasoning and math | Orca 2 13B | Purpose-built for reasoning; significantly better on structured tasks |
| Code generation | WizardLM 2 | Code-specialised training; better HumanEval scores |
| Long document work (>8K tokens) | Llama 3.1 8B Instruct | 128K context window vs 8K |
| Current best open 7B overall | Zephyr 7B or Mistral 7B v0.3 | More recent training; broader capability profile |