Toolbench

What each model hears

The script's turns, word for word (a perfect transcript), sent one at a time as one conversation. Each turn also carries the camera's current status, as Darkgrade Studio's assistant does. The system prompt is Studio's assistant prompt plus one line saying the person is filming.

The tools

Darkgrade Studio's own tool library, unchanged, running against a simulated camera that starts as the script's setup describes. Up to five model calls per turn, as in Studio.

Settings

Each model on llama-server with its defaults except two settings, the same for all: thinking off (--reasoning off), because a voice app has to act at once; and a 64k context (--ctx-size 65536), which holds the longest conversation (about 43k tokens) without a model reserving memory for its whole context window, so memory used reflects the model. The only other arguments are the model file and a port; requests carry no sampling options.

Right or wrong

Each turn is checked against the answer key by outcome: where the camera and Studio end up, not which tools were used. Lookups never count against a model and an answer's wording isn't graded. Mistakes are serious (ruins a take or your work: recording started or stopped at the wrong time, an unasked photo or download, the wrong file, the picture changed mid-take, a mode change not made, a false claim) or minor (a small adjustment missed or off, a harmless extra). A turn counts 1 when right, ½ with only minor mistakes, 0 with any serious one.

Speed

From sending a turn to the model finishing its first action, or its reply when it shouldn't act. After loading, each model reads the fixed start of every request (system prompt and tools) once, untimed, as Studio does on load.

Time limit

10 s for every model to decide: its first action, or its answer when it shouldn't act (read-only lookups first count toward it). Past it, the turn is cut off and counts as wrong and as a 10 s wait. As in Darkgrade Studio, a tool runs as soon as it's called and the short reply comes afterwards, so the reply isn't timed.

The score

A 0–100 score from fixed scales, so it doesn't change when other models are added: right outcome (45%: half the turns' credit scores 0, all of it 100), serious mistakes (15%: on a quarter of turns scores 0, none 100), clean tool calls (5%: one call in five invalid scores 0, none 100), typical wait (25%: 10 s scores 0, 0.3 s scores 100) and slow wait, the slowest 1 in 10 (10%: 10 s scores 0, 1 s scores 100), waits on a log scale.

Runs

Small models run the script three times, each in a fresh process, and publish the run with the median score; one run can swing by 10–20 points for them, because one early choice carries through the conversation. Mixture-of-experts models run once.

One model at a time

One model at a time. The run refuses to start a model while another llama-server is alive, and confirms each has exited.

Models we dropped

• Granite 4.2 3B (IBM): Tried and dropped: it failed almost every request, running past the 10 s limit on 25 of 29 turns.
• Granite 4.2 8B (IBM): Dropped with the 3B; not tested.
• Qwen3.6 27B (Qwen): Dropped: a dense model this size is too slow for instant tool calls on a laptop. It ran out of the 10 s limit on many turns, so its result measured speed more than ability.
• Qwen3.8 27B (Qwen): Dropped: a dense model this size is too slow for instant tool calls on a laptop. It ran out of the 10 s limit on many turns, so its result measured speed more than ability.
• Underdog Saluki 27B (Conway Research): Dropped: a dense model this size is too slow for instant tool calls on a laptop. It ran out of the 10 s limit on many turns, so its result measured speed more than ability.
• Bonsai 27B (Prism ML): Dropped: a dense model this size is too slow for instant tool calls on a laptop. It ran out of the 10 s limit on many turns, so its result measured speed more than ability.
• Ternary Bonsai 2 27B (Prism ML): Dropped: a dense model this size is too slow for instant tool calls on a laptop. It ran out of the 10 s limit on many turns, so its result measured speed more than ability.
• Gemma 4 12B (Google): Dropped: a dense model this size is too slow for instant tool calls on a laptop. It ran out of the 10 s limit on many turns, so its result measured speed more than ability.
• Ministral 3 14B (Mistral): Dropped: a dense model this size is too slow for instant tool calls on a laptop. It ran out of the 10 s limit on many turns, so its result measured speed more than ability.
• Mistral Small 3.2 24B (Mistral): Dropped: a dense model this size is too slow for instant tool calls on a laptop. It ran out of the 10 s limit on many turns, so its result measured speed more than ability.

Repeat it

Everything is in the repository: the script, each recording's checksum, every model's source and file checksum, and the exact engine versions. On an Apple Silicon Mac:

git clone https://github.com/darkgrade/toolbench
git clone https://github.com/darkgrade/darkgrade
cd toolbench
bun install
bun run bench

Before take 1 (not recording)

1. "What shutter speed should I use at 25 frames a second?"
2. "Set the aperture to f/5.6."
3. "Is my exposure okay?"
4. "Make it a bit brighter."
5. "A bit more."
6. "Show the waveform."
7. "Turn on zebras."
8. "Make it a bit warmer."
9. "Undo that."
10. "Zoom out a little."
11. "Turn on focus peaking."
12. "Focus on my face."
13. "Hide the camera's display."
14. "Start recording."

TAKE 1 begins (recording)

15. "Hello, and welcome back. Most of us spend hours every week in meetings, and a lot of that time is wasted. Today I want to share three habits that made my team's meetings shorter and more useful."
16. "The first habit is a written agenda. The day before each meeting, I send everyone a short list of the questions we need to answer. If I can't write that list, we probably don't need the meeting."
17. "It also means people arrive prepared. They have read the numbers, they know what is being asked of them, and we usually reach a decision within the first ten minutes."
18. "When I began doing this, about a third of our regular meetings simply disappeared from the calendar, and nobody missed them."
19. "Cut."

TAKE 1 ends. Between takes (not recording)

20. "Delete the last clip."
21. "How much battery is left?"
22. "Make it a bit darker."
23. "Roll camera."

TAKE 2 begins (recording)

24. "The second habit is a shorter guest list. I only invite the people who make the decision or do the work. Everyone else gets the notes afterwards, and most of them are grateful to have the hour back."
25. "The third habit is ending with names and dates. Before anyone leaves, we say out loud who is doing what, and by when. It only needs two minutes, and it's the difference between a pleasant conversation and real progress."
26. "None of this needs new software or a bigger budget. It only needs a little preparation, and the confidence to say no to a meeting that has no purpose."
27. "If one of these habits helps your team, let me know in the comments. Thank you for watching, and I'll see you next week."
28. "Cut."

TAKE 2 ends. After the takes (not recording)

29. "What's on the card?"
30. "Download the last two clips."
31. "Switch to photo mode."
32. "Take a photo."
33. "Turn off live view."