Production voice AI for drive-thru ordering
The problem
A customer speaks at a drive-thru speaker and expects food. Between those two things sits a chain of systems that can each fail independently, and a language model that is fluent but not authoritative. The model can produce a sentence that sounds like a confirmed order while the cart behind it says something else.
The engineering problem is therefore not “make the model understand speech”. It is: decide what counts as true, and refuse to let a plausible sentence stand in for a validated one.
Treating one turn as six separate facts
The design I work within treats a single customer turn as a ladder of six distinct outcomes, each of which is a different claim: the model generated text; the text was emitted to the speech layer; the cart change was persisted; the order was priced by the point-of-sale provider; the order was submitted; and finally the kitchen fulfilled it.
Success at one rung is not evidence for the next. An emitted reply is not proof the customer heard it. A point-of-sale acceptance is not proof the kitchen made the food. Keeping those separate is what makes the system debuggable: when something goes wrong, the question “which rung did we actually reach?” has an answer in the logs.
The system
A FastAPI and Socket.IO service receives transcribed text — not audio — from the speech and telephony layer, and returns reply text plus control markers. Automatic speech recognition, translation and text-to-speech live outside this boundary, which keeps the ordering logic testable without a microphone.
Three different models do three different jobs, chosen by cost and latency rather than uniformly. A small, fast model does entity extraction at temperature 0 with a tight token cap. A larger model with a thinking budget runs the ordering conversation against the store's catalogue. A third call, constrained by a strict JSON schema, converts the agent's own confirmation lines into exactly one cart operation.
Cart state lives in Redis behind a per-session worker and a lock, so updates for one lane apply in order. Menus and brand configuration load from MongoDB into a cache with a day-long expiry. Pricing and submission are application code, never model tool calls — the prompt explicitly forbids the model from touching them.
Engineering decisions and what they cost
Prompt caching with a timed refresh. The base prompt, the order prompt and the store's full catalogue are cached together with a one-hour lifetime, refreshed shortly before expiry and pre-warmed at startup. This is a large latency win per turn. The cost is a real operational trap: the cache does not invalidate when a prompt file changes, so an edited prompt only goes live after expiry or a restart. That is the kind of thing that makes a fix look like it did not work.
Cart jobs are scheduled from the reply before it is shortened. The spoken line is rephrased and truncated to a single question after the cart job is queued from the original. This keeps speech short and natural without losing cart detail — and it means the sentence the customer hears and the input the cart model reads are deliberately not the same string. Worth knowing before debugging a mismatch.
Four point-of-sale providers, four different commit semantics. One creates the order only at commit, another prices then suspends an existing order, a third quotes then commits, a fourth calculates then submits synchronously. A single adapter interface hides this, but the differences are real and cannot be abstracted away — what “submitted” means is provider-specific.
Barrier tokens instead of locks for staleness. When a call ends, a customer says “start over”, or staff barge in, a barrier token moves and replies and queued cart jobs from earlier turns are dropped. It is cheap and it works for the common case. It is explicitly not exactly-once, and the barrier is checked before the locked update rather than after every await.
Escalate rather than loop. A third identical reply hands the call to a human. An explicit request for a person bypasses the agent entirely. The failure mode this prevents — a customer stuck repeating themselves to a confident model — is worse than a handoff.
Measured, not estimated
On a replay of a production call batch, 125 reply turns: median reply latency 3.02 s, p90 5.52 s, max 7.85 s. Time to first speech, which is what the customer actually perceives because filler lines count, median 1.61 s, p90 3.84 s. A wider earlier baseline across 17,884 turns put median reply at 2.63 s.
The outliers are more interesting than the median. Two turns in one Spanish-language call took 15.7 s and 19.6 s. The timer covering them also covers queue drain, cache refresh and extraction, so the number is a bound on the model call rather than a measurement of it — which is itself a finding about the instrumentation, not just the system.
How it is tested
Replay against recorded production turns, with every case run three times because the model varies between runs. Assertions check the persisted cart writes, not only the spoken lines — checking speech alone would pass exactly the failure this system is built to prevent.
Production calls are also reviewed by hand against the service logs and replayed, which is how prompt and code fixes get proposed in the first place.
What this demonstrates
Putting a probabilistic component inside a transaction that has to be correct, and being precise about where the probabilistic part stops. The interesting work is not the prompt — it is the ladder of distinct outcomes, the deterministic validation between the model and the point-of-sale, and the recoverable handoff when the model has run out of useful moves.
This is a system overview. The implementation is proprietary and no code, customer data or call recordings are published here.
Other case studies
Conversational AI for college ERP data · Face recognition attendance across 20+ camera streams
Full CV · Download UAE CV (PDF) · Discuss an AI engineering opportunity