COLORS.chat network
colorschat.com · YouTube colors.chat · Order now terminal.glass · Deploy
🎙️ OpenAI Open Source · September 2022

Whisper: Local Speech-to-Text
Without the Cloud

Your meeting recordings, medical dictation, interview audio, and voice memos contain some of the most sensitive information you produce. Every time you upload them to a cloud transcription service — Otter.ai, Rev, even the OpenAI Whisper API — you are sending that audio to servers you do not control, under retention policies you may not have read. OpenAI released Whisper's weights openly in September 2022. You can run it on your own machine, process audio that never leaves your device, and pay nothing per minute. This page shows you how.

🏢 Creator: OpenAI (open weights)
📅 Released: September 2022
⚖️ License: MIT (fully open)
🌍 Languages: 99+ languages
🎯 Task: Speech-to-text (not a chat model)
💾 Min RAM: 1GB (tiny) – 10GB (large-v3)
🎙️
Different capability category

Whisper is a speech recognition model, not a language model. It converts audio to text — it does not answer questions, summarise documents, or generate responses. It belongs in this guide because it solves a genuine privacy problem (cloud transcription) using the same local-first approach as the chat models covered elsewhere. After transcription, pipe the output into OpenChat, Nous Hermes 2, or any other local model to summarise or analyse the text without either component touching the cloud.

99+
Languages supported
680K hrs
Training audio data
MIT
Licence — fully commercial
5 sizes
Tiny → Large-v3
$0
Per minute (local)
~2.7%
Word error rate (large-v3, English)

The Privacy Case for Local Transcription

Audio transcription is one of the most privacy-sensitive tasks you can outsource to a cloud service. Consider what your audio typically contains:

🏥

Medical conversations

Patient consultations, therapy sessions, clinical notes dictated by doctors — all regulated by HIPAA in the US, GDPR in Europe. Cloud transcription services may not meet compliance requirements.

⚖️

Legal and professional

Client consultations, depositions, internal strategy discussions. Attorney-client privilege may be compromised by sending recordings to third-party cloud providers.

🏢

Business meetings

Board meetings, product roadmaps, M&A discussions, personnel reviews. Many organisations prohibit uploading internal meeting recordings to external services.

📰

Journalism and research

Source interviews, investigative recordings. Uploading source audio to a cloud service creates a record that could be subpoenaed or accessed by the service provider.

🎓

Academic and personal

Research interviews, personal journals, therapy session notes. Even non-regulated data deserves protection from commercial data harvesting.

💰

High-volume transcription

At $0.006/minute (OpenAI API), 100 hours of audio costs $36 per month. Local Whisper is free after setup, making it cost-effective for heavy users.

☁️ Cloud transcription (Otter, Rev, Whisper API)

  • Audio transmitted to external servers
  • Provider retains audio per their terms
  • Cost accumulates per minute
  • Internet connection required
  • May not meet HIPAA / GDPR requirements
  • Provider can be subpoenaed
  • Data may train future models

🖥️ Local Whisper (this guide)

  • Audio never leaves your machine
  • No retention policy to worry about
  • Free after initial setup
  • Works fully offline
  • You control all compliance decisions
  • Nothing to subpoena at the provider
  • Your data cannot train any external model

Choosing a Model Size

Whisper comes in five sizes. The right choice depends on your hardware, your accuracy requirements, and how fast you need results. For most users on modern laptops, medium provides the best accuracy-to-speed tradeoff. On Apple Silicon Macs or machines with discrete GPUs, large-v3 is practical.

Model Parameters VRAM / RAM Rel. Speed English WER Best for
tiny 39M ~1 GB ~32× ~6–8% Quick drafts, live captions, Raspberry Pi
base 74M ~1 GB ~16× ~5–6% Podcasts, clear audio on weak hardware
small 244M ~2 GB ~6× ~4–5% Good balance for most laptops
medium ★ 769M ~5 GB ~2× ~3–4% Recommended default — meetings, accents
large-v2 1550M ~10 GB ~1× ~2.9% High accuracy, GPU recommended
large-v3 1550M ~10 GB ~1× ~2.7% Best accuracy; Apple Silicon / CUDA GPU
large-v3-turbo 809M ~6 GB ~8× ~3.0% Large-v3 quality at medium speed
ℹ️
large-v3-turbo: the new recommended default for capable hardware

Released in October 2024, large-v3-turbo uses a decoder with only 4 layers instead of 32, making it roughly 8× faster than large-v3 while maintaining nearly identical accuracy. If your machine has 6GB+ of available RAM and you have a modern CPU or any discrete GPU, large-v3-turbo is now the best single choice. Use medium on older or memory-constrained hardware.

Language Support

Whisper was trained on 680,000 hours of multilingual audio. English performance is best, but many other languages reach production-quality accuracy on the large models. WER varies significantly by language depending on training data volume.

Strong support (WER < 10% on large-v3)

🇬🇧English
🇪🇸Spanish
🇫🇷French
🇩🇪German
🇮🇹Italian
🇵🇹Portuguese
🇳🇱Dutch
🇯🇵Japanese
🇨🇳Chinese
🇰🇷Korean
🇷🇺Russian
🇵🇱Polish

Good support (WER 10–20%)

🇸🇦Arabic
🇹🇷Turkish
🇸🇪Swedish
🇺🇦Ukrainian
🇮🇩Indonesian
🇨🇿Czech
🇷🇴Romanian
🇭🇺Hungarian
🇬🇷Greek
🇮🇳Hindi

For a full list of all 99 languages and per-language WER scores, see the official Whisper model card.

Installation and Quick Start

Option 1: Python CLI (fastest to get started)

bash — Install
# Requires Python 3.8+ and ffmpeg

# macOS
brew install ffmpeg
pip install openai-whisper

# Linux (Ubuntu/Debian)
sudo apt install ffmpeg
pip install openai-whisper

# Windows (with chocolatey)
choco install ffmpeg
pip install openai-whisper
bash — Transcribe
# Transcribe a file (model downloads automatically on first run)
whisper meeting.mp3 --model medium

# Specify output formats
whisper interview.m4a --model medium --output_format txt

# Generate subtitles (SRT format)
whisper lecture.mp4 --model medium --output_format srt

# Non-English audio — specify language for better accuracy
whisper paris-meeting.mp3 --model medium --language French

# Translate to English while transcribing
whisper german-call.mp3 --model large-v3 --task translate

# Best accuracy on capable hardware
whisper important-meeting.mp3 --model large-v3-turbo

Option 2: faster-whisper (recommended for production)

faster-whisper is a reimplementation using CTranslate2 that runs 2–4× faster than the original with the same accuracy. It also uses significantly less memory, allowing large-v3 to run on machines with 6GB RAM.

bash — faster-whisper install
pip install faster-whisper
python — faster-whisper basic usage
from faster_whisper import WhisperModel

# Use "cuda" if you have an NVIDIA GPU, "cpu" otherwise
# "auto" selects automatically
model = WhisperModel(
    "large-v3-turbo",
    device="auto",
    compute_type="int8"       # int8 = fastest, lowest RAM
)

segments, info = model.transcribe(
    "meeting.mp3",
    beam_size=5
)

print(f"Detected language: {info.language} (probability: {info.language_probability:.2f})")

for segment in segments:
    print(f"[{segment.start:.1f}s → {segment.end:.1f}s] {segment.text}")

Option 3: whisper.cpp (runs on CPU with no Python)

whisper.cpp is a C++ port that runs efficiently on CPU with no Python, no CUDA, and no GPU required. It also supports Apple Silicon acceleration (Core ML) and is the basis for many desktop GUI applications.

bash — whisper.cpp
# Build from source
git clone https://github.com/ggerganov/whisper.cpp
cd whisper.cpp
make

# Download a model (GGML format)
bash models/download-ggml-model.sh medium

# Convert audio to 16kHz WAV (required format)
ffmpeg -i meeting.mp3 -ar 16000 -ac 1 -c:a pcm_s16le meeting.wav

# Transcribe (pure CPU, no Python)
./main -m models/ggml-medium.bin -f meeting.wav

# Apple Silicon: enable Core ML acceleration
make -j WHISPER_COREML=1
./main -m models/ggml-large-v3-turbo.bin -f meeting.wav

Practical Privacy Workflows

Meeting transcription pipeline

python — full meeting pipeline with local summarisation
from faster_whisper import WhisperModel
import ollama
import sys

# ── Step 1: Transcribe locally with Whisper ──────────────────
print("🎙️  Transcribing... (audio never leaves this machine)")

whisper_model = WhisperModel("large-v3-turbo", device="auto", compute_type="int8")
audio_file = sys.argv[1]  # e.g. python pipeline.py meeting.mp3

segments, info = whisper_model.transcribe(audio_file, beam_size=5)

transcript = ""
for segment in segments:
    transcript += f"[{segment.start:.0f}s] {segment.text.strip()}\n"

print(f"\n✅ Transcribed {info.language} audio ({len(transcript)} chars)\n")

# ── Step 2: Summarise locally with Ollama ───────────────────
print("🧠  Summarising with local LLM...")

response = ollama.chat(
    model="nous-hermes-2",   # or openchat, llama3, etc.
    messages=[{
        'role': 'user',
        'content': f"""Summarise this meeting transcript. 
Extract: (1) key decisions, (2) action items with owners, 
(3) open questions.

TRANSCRIPT:
{transcript}"""
    }]
)

print("\n📋 Meeting Summary:")
print(response['message']['content'])

# Both steps ran entirely locally.
# No audio and no text left this machine.

Batch transcription of existing archives

bash — batch transcribe a folder
# Transcribe every audio/video file in a directory
for f in ~/recordings/*.{mp3,mp4,m4a,wav,ogg}; do
    echo "Processing: $f"
    whisper "$f" --model medium --output_dir ~/transcripts/
done

GUI Applications (No Terminal Required)

For users who prefer a graphical interface, several open-source applications wrap Whisper in a desktop UI:

Whisper Desktop

Windows · GPU accelerated

Native Windows GUI for whisper.cpp. Drag-and-drop audio files, choose model size, get transcripts. No Python required.

MacWhisper

macOS · Apple Silicon optimised

Native macOS app with Core ML acceleration. Supports all Whisper sizes, subtitle export, and microphone input. Free tier available.

Whisper Transcription

macOS / iOS

Simple app store application for transcribing audio files and recordings with Whisper locally.

Buzz

macOS / Linux / Windows

Cross-platform open-source GUI. Real-time transcription from microphone or file input. Supports all Whisper model sizes.

Subtitle Edit

Windows · Subtitle workflows

Professional subtitle editor with Whisper integration. Ideal for generating and editing SRT/VTT files for video content.

OpenAI Whisper Web

Browser · WebGPU

Runs Whisper entirely in the browser using WebGPU — no installation, no server, no data transmission. Experimental but functional on Chrome.

Honest Assessment

✓ Genuine Strengths

  • MIT licence — fully commercial, no restrictions
  • 99+ language support with strong multilingual accuracy
  • Completely offline — audio never leaves your machine
  • Free to run; no per-minute API cost
  • Multiple size tiers for any hardware configuration
  • Handles accents, background noise, and poor audio reasonably well
  • Outputs timestamped segments — useful for navigation and subtitles
  • Translates non-English audio directly to English
  • whisper.cpp runs on CPU with no Python or GPU required
  • Active ecosystem: faster-whisper, whisper.cpp, GUI apps

✗ Real Limitations

  • Not real-time — processes recorded audio, not live streams (without extra tooling)
  • No native speaker diarisation (who said what) — requires pyannote.audio add-on
  • large-v3 needs 10GB RAM — medium is the practical limit on most laptops
  • Occasionally hallucinates words in silent or very noisy segments
  • Punctuation inconsistent across languages
  • Technical jargon, proper nouns, and uncommon names may transcribe incorrectly
  • Non-speech audio (music, sound effects) can confuse the model
  • ffmpeg dependency adds a setup step for Python CLI
⚠️
Hallucination in silence

A known Whisper behaviour: when processing silent or near-silent audio segments, the model sometimes generates plausible-sounding but entirely fabricated text. Always review transcripts for sections where the original audio has long pauses or background noise only. The no_speech_threshold parameter in faster-whisper can help suppress this.

Adding Speaker Diarisation

Whisper alone cannot identify who is speaking. For meeting transcripts that need speaker labels ("Speaker A: ...", "Speaker B: ..."), combine Whisper with pyannote.audio, a speaker diarisation library that also runs locally.

bash — install diarisation dependencies
pip install faster-whisper pyannote.audio

# pyannote requires accepting terms at hf.co/pyannote/speaker-diarization-3.1
# then use your HuggingFace token:
# export HF_TOKEN="hf_..."
python — Whisper + speaker diarisation
from faster_whisper import WhisperModel
from pyannote.audio import Pipeline
import os

audio_file = "meeting.wav"  # must be 16kHz mono WAV

# ── Speaker diarisation ────────────────────────────────────
diarize = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-3.1",
    use_auth_token=os.getenv("HF_TOKEN")
)
diarization = diarize(audio_file)

# ── Transcription ──────────────────────────────────────────
whisper = WhisperModel("medium", device="auto", compute_type="int8")
segments, _ = whisper.transcribe(audio_file, beam_size=5)
segments = list(segments)

# ── Align segments with speakers ──────────────────────────
for segment in segments:
    mid = (segment.start + segment.end) / 2
    speaker = "Unknown"
    for turn, _, spk in diarization.itertracks(yield_label=True):
        if turn.start <= mid <= turn.end:
            speaker = spk
            break
    print(f"[{speaker}] {segment.text.strip()}")