Whisper large-v3 vs a Commercial Speech-to-Text API on YouTube, in 9 Languages

We wanted to know whether an open model we can run ourselves transcribes YouTube videos as well as a commercial speech-to-text API, in the languages our readers use. So we transcribed the same set of public YouTube videos with both and scored them against the captions the channels uploaded themselves.

Short answer: Whisper large-v3 matches or beats a commercial speech-to-text API's median error in English, Spanish, Japanese, and Chinese, and is within the margin of error in Korean, Vietnamese, and Portuguese. The commercial API leans better in Thai and Hindi, but with 16 videos per language the difference is not proven. One setting clearly hurts: running Whisper in batched mode makes Thai much worse. Two Japanese-specific models did not beat Whisper large-v3 on YouTube audio.

What we compared

A commercial speech-to-text API (a widely used paid service), called through its API.
Whisper large-v3 with faster-whisper 1.1.1 on an NVIDIA T4 GPU, beam size 5, in two passes: first with voice activity detection (VAD), then again without VAD if the first pass came back nearly empty.
The same, with the first pass batched (batch size 16), which is faster.
A variant that checks the detected language at three points in the audio.
For Japanese only: kotoba-whisper v2.0 and ReazonSpeech NeMo v2.

How we scored it

Reference: captions uploaded by the channel (not YouTube's automatic captions), in the language spoken in the video. We picked 234 public videos, 3 to 15 minutes long, 26 per language.
Translated captions removed: many videos carry captions in another language than the one spoken (for example English speech with Japanese captions). We dropped those.
Partial captions removed: when every system disagreed with a caption by more than a factor of two in length (the caption was cut short or was a different text), we dropped that video too. One system failing on its own never removed a video. That left 151 videos, 13 to 24 per language.
Metric: word error rate (WER) for English, Spanish, Vietnamese, Portuguese, and Hindi; character error rate (CER) for Japanese, Chinese, Thai, and Korean. Lower is better.
Normalization: Unicode NFKC, lower case, punctuation removed; Chinese converted to Simplified on both sides, so Traditional versus Simplified does not count as an error.
Reported figure: the median error per video, plus the error over all words or characters with a 95% bootstrap interval across videos.

Results

Median error rate per video by language: commercial API, Whisper large-v3 two-pass, and batched

Median error rate per video (lower is better). In brackets: the corpus error rate with its 95% interval.

English (WER, 24 videos): Commercial API 0.136 (0.117, 0.08-0.16); Whisper large-v3 0.086 (0.093, 0.06-0.13); batched 0.090.
Spanish (WER, 13): Commercial API 0.155 (0.194, 0.11-0.35); Whisper 0.127 (0.161, 0.09-0.29); batched 0.116.
Korean (CER, 15): Commercial API 0.078 (0.122, 0.07-0.22); Whisper 0.081 (0.110, 0.07-0.18); batched 0.103.
Japanese (CER, 18): Commercial API 0.131 (0.151, 0.11-0.19); Whisper 0.114 (0.153, 0.11-0.20); batched 0.184.
Vietnamese (WER, 17): Commercial API 0.220 (0.187, 0.11-0.29); Whisper 0.231 (0.193, 0.11-0.30); batched 0.232.
Portuguese (WER, 16): Commercial API 0.101 (0.155, 0.10-0.22); Whisper 0.108 (0.130, 0.10-0.17); batched 0.109.
Chinese (CER, 16): Commercial API 0.069 (0.088, 0.06-0.12); Whisper 0.060 (0.081, 0.06-0.11); batched 0.059.
Thai (CER, 16): Commercial API 0.111 (0.126, 0.08-0.19); Whisper 0.168 (0.162, 0.12-0.22); batched 0.339 (0.352, 0.29-0.41).
Hindi (WER, 16): Commercial API 0.359 (0.477, 0.31-0.74); Whisper 0.460 (0.534, 0.43-0.67); batched 0.622.

Japanese character error rate on the same 14 videos for four systems

Japanese-specific models, on the first 14 Japanese videos only (CER): kotoba-whisper v2.0 0.256, ReazonSpeech NeMo v2 0.253, against Whisper large-v3 0.155 and the commercial API 0.131 on the same videos.

The variant that checks the language at three points scored the same as the batched version in every language. Without the partial-caption filter the conclusions are the same; only the corpus rates for Chinese and Japanese move, because one or two long mismatched captions dominate them.

What it costs to run Whisper yourself

We ran Whisper large-v3 on an NVIDIA T4 GPU on Modal, priced at $0.000164 per second, or about $0.59 per GPU-hour (Modal's public price, checked on 2026-10-19). Measured over 76 hours of audio, including billed overhead:

Two-pass: about 4.4 hours of audio per GPU-hour, or about $0.13 per hour of audio.
Batched first pass: about 13 hours of audio per GPU-hour, or about $0.045 per hour of audio.
Batched with the three-point language check: about 16.7 hours of audio per GPU-hour, or about $0.035 per hour of audio.

Compute cost per hour of audio for three Whisper settings

Batching is roughly three times cheaper, which is worth it in most languages, but not in Thai (see above).

What we read from it

For most languages, an open model is enough. Whisper large-v3 is at or below the commercial API's median in English, Spanish, Japanese, and Chinese, and within the margin of error in Korean, Vietnamese, and Portuguese. The clearest lean is English (WER 0.086 against 0.136).
Thai and Hindi are where a commercial API may still help. Its medians are better there, but the intervals overlap, so this is a lean, not a result.
Do not batch Thai. Batched Whisper in Thai is the only difference that falls outside the intervals. Batching also leans worse in Hindi, Japanese, and Korean, and costs nothing measurable elsewhere.
Language-specific models are not automatically better. For Japanese YouTube audio, the general Whisper large-v3 beat two Japanese models.
Many uploaded captions are translations. Even when we picked videos whose audio language matched, only 13 to 17 of 26 per language outside English had captions in the spoken language.

Limits

13 to 24 videos per language, so the medians are indicative. Only batched Whisper in Thai is outside the intervals.
Uploaded captions are not always word for word (they can be edited or shortened). That raises every system's error by a similar amount.
The partial-caption filter is a rule on lengths; without it the conclusions hold, but some corpus rates move.
The Japanese-specific models were run on the first 14 Japanese videos only.

Glasp Research publishes experiments we run while building Glasp. Questions or corrections: see the FAQ or contact us.