How we test

Volunteers read the same script into their own phones. Every recording goes through every model on one Mac, one model at a time, with each engine's default settings. Each model is measured on two things only: how many words it gets wrong, and how long you wait for the text after you stop talking.

English only for now, from 8 speakers (158 spoken lines in all). With a sample this size, treat scores within a few points of each other as ties.

Settings

Each engine with its defaults, except thinking: models on llama-server run with thinking off (--reasoning off), the same as in Toolbench. The only other arguments are the model files and a port. CrispASR also gets --auto-download, so it may fetch helper files its defaults use; CrispASR and the Python runner have no thinking to turn off.

Models that hear a whole line

Each utterance sent alone to POST /v1/audio/transcriptions as soon as it ends; latency is the request time. One utterance is sent untimed first to warm the model up.

Streaming models

Each recording played into /v1/realtime at real speed in 80 ms blocks; an utterance's words are those arriving before the next utterance starts (at most 5 s after it ends); latency is from its end to its last word.

Accuracy

Word and character error rate against the script, pooled over every speaker. Both sides are lower-cased, punctuation removed, contractions expanded, numbers written as digits and British spellings made American. Output markup (Qwen3-ASR language tags, VibeVoice speaker labels) is removed first. Speakers' own slips count against every model alike.

Speed

1000 ms from the end of an utterance to its final text.

The score

A 0–100 score from fixed scales, so it doesn't change when other models are added: accuracy (60%: 15% word errors scores 0, 2% scores 100), typical wait (25%: 1.5 s scores 0, 50 ms scores 100) and slow wait, the slowest 1 in 10 (15%: 3 s scores 0, 150 ms scores 100), each on a log scale so every halving counts the same.

Area	Counts for
Accuracy	60%
Typical wait	25%
Slow wait	15%

One model at a time

One model at a time. The run refuses to start a model while any other engine process is alive, and confirms the model's server has exited before the next starts.

Runs

One pass per model per recording.

Repeat it

Everything is in the repository: the script, each recording's checksum, every model's source and file checksum, and the exact engine versions. On an Apple Silicon Mac:

git clone https://github.com/darkgrade/voicebench
cd voicebench
bun install && bun run setup
bun run bench

The script

Twenty things people say to a voice assistant, read once by every speaker.

Good morning.
What's the weather like in London today?
And what about Sunday?
Set a timer for ten minutes.
Cancel that.
Wake me up at six tomorrow.
Send a message to David saying I'm running about fifteen minutes late, so please start without me and I'll join you soon.
Yes, send it.
Remind me to buy eggs, strawberries, and orange juice when I leave work this evening.
Play some quiet jazz in the kitchen.
A little louder, please.
Stop.
Show me the fastest way to the airport from here, and tell me whether there's heavy traffic on the bridge right now.
Did anyone call while I was out?
Schedule a meeting with Emma for next Wednesday at three, and move my usual team lunch to Thursday.
Search for a vegetable soup recipe that serves four.
No, the other one.
How do you say thank you in Spanish?
Can you explain how a rainbow forms, using simple words that a young child could easily understand?
Thanks, that's all.