The Thermodynamics of a Voice

We built one that keeps its presence: multilingual, booking, real-time. The commercial platform it replaces has carried over 100,000 minutes of calls in production — and that bill, the one that grew with every success, is what set the rebuild in motion. The architecture, it turned out, was never the hard part.

The hard part was the money: how much it cost, where the cost was hiding, and what we had to unlearn to bring it down.

Three engines, one conversation

Someone calls. A transcriber turns their voice into text. A reasoning engine decides what to say back. A synthesizer says it. Three steps, in series, on a single GPU — and each one spends two currencies at once: time and money.

The three engines plus network and tool calls land a typical turn near 800 milliseconds — a clean turn closer to 600, a complex one with several lookups around 1.2 seconds. The best-known platform on the market quotes one and a half to two seconds for the same work. On an average call we are comfortably faster; on a clean one, close to twice as quick. And because the synthesizer streams, the caller hears the first words before the full turn completes.

The speed matters. The cost matters more. A managed platform meters all three engines: a fixed subscription at market rates before a single call connects, then fifty cents to a dollar and a half per conversation on top. That is the bill this rebuild was meant to bring down.

The alternative is simpler than it sounds. Run the models yourself, on your own GPUs.

The empty highway

We ran the same models, the same weights, the same containers on two very different cards, expecting the faster one to win.

By the textbook, the H200 should have won — inference is supposed to be bound by memory bandwidth, and it has nearly three times as much. It lost anyway. Our reasoning engine activates only a fraction of its parameters on any given word, and a single live call is a batch of one; with so little data in flight, raw core speed beats bandwidth, and the newer Blackwell silicon pulls ahead. The H200's enormous bandwidth becomes a ten-lane highway at four in the morning.

The prompt is the product

The biggest source of latency wasn't the synthesizer, and it wasn't the transcriber. It was the prompt.

Our agent reasons against a 25,000-character rulebook — referral codes, mappings, escalation rules. In the naive build, the engine re-read the entire thing before producing a single token of every reply. A cold turn took seven seconds: dead on arrival.

We moved the handful of values that change each turn — the timestamp, the call id — to the end of the prompt, leaving the 25,000 characters in front of them byte-for-byte identical. The engine now processes that prefix once and reuses it forever. Cost: nothing. Improvement: eightfold. Then we found the reasoning engine writing its private deliberation into the reply buffer — burning time, sometimes returning nothing at all — and switched it off: warm turns fell from 3.5 seconds to 0.4, a further factor of nearly nine. Eight-bit quantization of the attention cache cut memory fourfold with no audible loss.

The other half of engineering

The GPU comes from a spot market: a new machine and a new random port on every deploy, and no working SSH. We lost two hours before accepting the obvious — the whole pipeline would bring itself up over HTTP boot scripts, or not at all. The synthesizer's text normalizer, built for a different set of languages, mangled characters outside it, reading some letters aloud by name instead of voicing them; one configuration flag, one afternoon. And two of eighteen tool parameters didn't match the booking webhook — a mismatch that surfaced not as an error but as an infinite conversational loop, the agent apologizing and asking the caller to repeat, forever. We audited every parameter against the live API by hand.

What remains is one command that provisions the GPU, boots the container, compiles the engine for two architectures, downloads the weights, loads three models, uploads a dozen cloned voices, starts the backend and the frontend, and primes the cache — cold, in about eight minutes. One command tears it all down again, because a forgotten GPU bills by the hour until the credit runs dry.

The unit economics

Start at a single turn. At $1.07 per GPU-hour, about 800 milliseconds per turn, and ten simultaneous callers on one card, the engines cost a rounding error:

The comparison that matters, though, is monthly and at scale. All-in: GPU, reasoning, telephony.

The self-hosted engine carries zero per-token cost; the cloud alternatives meter every token, and the premium one — the default most teams reach for — costs thousands more each month. At fifty concurrent callers the saving against it is fifty-six per cent. Against even the cheapest cloud option, owning the GPUs wins at every tier.

What we learned about cost

The intuition that a bigger model is slower and dearer is wrong in at least three ways, and each was worth real money.

The direction of travel

The comparison holds at today's models and today's prices, and both are in motion. Open models improve and cheapen with every release; our reasoning engine is the sixth-ranked open model on the public leaderboard, level with proprietary systems that bill by the token. The stack keeps its options open — the production engine fits on a consumer card, and a single toggle routes a genuinely hard turn to a larger cloud model on demand. Most calls never ask for it.

The speech models follow the same curve. The synthesizer — an open diffusion model at 48 kHz with zero-shot voice cloning — rivals proprietary text-to-speech in the languages we serve; the transcriber, tuned with domain vocabulary and typo correction, handles the booking flow reliably. Both improve with every open-source release.

And the data compounds. High-quality speech data for specialized insurance vocabulary is scarce; no public dataset covers our referral codes or scheduling terms. Every call generates a fresh sample with a verified transcript — a corpus proprietary platforms never see, because on a self-hosted stack the calls never leave our infrastructure. That feeds better recognition, and in time a transcriber fine-tuned on our own callers that no general-purpose model can match on this vocabulary.

Then there is fine-tuning. The 25,000-character rulebook distilled into the model's own weights — latency dropping further, the cost per turn approaching the thermodynamic minimum: electricity through silicon. Fine-tune the synthesizer on our own audio, the cadence of explaining a policy excess and the rhythm of reading back an appointment time, and it sounds more natural than any general-purpose model. The stack absorbs every improvement the open-source community ships, and pays no one a margin.

Architecture is a choice

There is an advantage here that is architectural rather than financial. The stack is assembled from open components — an encoder-decoder for transcription, a diffusion model for synthesis, a sparsely-activated engine for reasoning — and open components are not black boxes with fixed seams and opaque prices. They compose. Characters mangled in synthesis? A configuration flag, not a support ticket. Private reasoning leaking into the reply? A runtime parameter, not a platform release. Every optimization in this report was possible only because the stack is open and inspectable.

Topology is just as free. Today the GPUs sit in an EU data center; tomorrow they could move behind a hospital firewall so that no audio ever leaves the building — the same models, the same runtime, the same configuration. Data residency stops being a negotiation and becomes a line in a config file. For a medical booking agent in the European Union, that is not a preference. It is a requirement.

Where this lands

We have a real-time, multilingual voice agent. It books appointments, answers questions, and hands off to a human operator when it should. A typical turn is about 800 milliseconds — comfortably inside the platform's quoted one-and-a-half to two seconds — with a clean turn near 600 and a complex one around 1.2. A dozen cloned voices. Domain-tuned transcription. Benchmarked well under a dollar per call, all in; at fifty concurrent callers, fifty-six per cent below the premium cloud. What stands between this and a finished product is not the voice pipeline — it is HL7 and FHIR compliance, GDPR-grade recording, multi-tenant isolation, human fallback with a clean handoff of context. Real problems, all solvable with infrastructure that already exists. The voice part is done.