Why Did Our AI Phone Agent Book the Wrong Appointment?

Why did our AI phone agent book the wrong appointment?

Because on that call, nothing outside the model checked the booking against your rule. The instruction existed. It was probably well written. It said something like "only book new patients on Tuesday and Thursday mornings, and never within twenty-four hours". The caller asked for something the sentence did not quite cover, the model produced the most plausible-sounding continuation, and the booking was written. No component anywhere raised an error, because from the software's point of view nothing failed. A booking was requested and a booking was made.

This is the important part, and it is the part that gets misdiagnosed as a quality problem: the model did not malfunction. It did exactly what it does on every call, which is to generate a plausible next thing to say. When your rules live inside the text the model reads, plausibility is the only standard being applied to them. When your rules live outside the model, plausibility is irrelevant, because the booking is only written if it satisfies the constraint.

So the wrong appointment is not really a bug report. It is a piece of information about which of two architectures you bought. Most buyers never learn there was a choice, because the choice is invisible during evaluation. The shorter, buyer-facing version of this argument is on our page about where a voice agent's rules are held.

Where do your business rules actually live in an AI phone system?

There are two answers. Most products are somewhere between them, but the question is still worth asking as a binary, because the rules that matter most are the ones that fall on one side or the other.

In the prompt. Your policies are written as paragraphs of natural language inside the instruction the model receives. Deposit rules, opening hours, what you will and will not quote over the phone, which appointment types need a nurse present, what to do when someone asks for a service you stopped offering last year. The model reads that text on every turn and decides, each time, what it implies. Vapi's own prompting documentation is blunt about the cost of this: "Every token costs latency. The system prompt loads into the model's context on every turn." The same guide notes that an agent without explicit guardrails "will eventually provide medical/legal/financial advice, fabricate prices, engage with off-topic conversations, or reveal internal system information".

Encoded outside the model. The rules are expressed as a policy the system has to satisfy, and the model's job is narrowed to conversing. It can propose a booking. Whether that booking is written depends on a check that has nothing to do with how convincing the sentence was. If the proposed slot violates the rule, the write does not happen and the call is routed to a person or to an alternative. The rule is not advice to the model. It is a condition on the action.

Deepgram states the distinction in one line in its own voice-agent documentation: "Think of the prompt as policy and your server code as enforcement." The same page gives the diagnostic that follows from it: "if you're writing increasingly emphatic rules ('ABSOLUTELY NEVER,' 'UNDER NO CIRCUMSTANCES') and the model still violates them, move the enforcement to server code". The paragraph that sentence closes is more specific still: for anything reliability-critical, Deepgram tells its own developers, server-side controls are required and prompt instructions alone are not enough.

Why does a rule written into a prompt drift on the calls that matter?

This is a mechanism, not an insult to prompts. Four things stack up, and they stack up hardest on exactly the calls a business cares about.

The latency budget pushes toward smaller and faster models. Phone conversation has almost no tolerance for silence. The cross-language study of turn-taking by Stivers and colleagues in PNAS found that all ten languages sampled avoid overlapping talk and minimise the silence between turns, with the average gap between turns varying, from one language to the next, within a range of 250 milliseconds around the cross-language mean. That is a fact about human conversation rather than a latency target for software, and it is why a caller registers hesitation long before they register a wrong answer. A voice agent that pauses to think sounds broken to a caller, so the whole pipeline is engineered around speed. LiveKit puts the trade-off plainly in its engineering write-up on voice agent latency: "Larger or more capable models usually take longer per step than smaller or distilled ones, but provide better capabilities." In a phone product that trade-off is usually resolved in the direction of speed, because the alternative the caller experiences is dead air. The instruction-following headroom is spent first.

A long policy competes for attention with the conversation itself. The prompt is not read once at setup. It is re-supplied on every turn, alongside a growing transcript. The TACL paper "Lost in the Middle: How Language Models Use Long Contexts" found that performance "is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts". Your deposit rule is clause fourteen of a page-long policy, sitting in the middle, on turn nine of a call where the caller has changed their mind twice.

Instruction following decays across turns. This is measured rather than suspected, in benchmarks rather than on live calls. In "LLMs Get Lost In Multi-Turn Conversation", Laban and colleagues ran over 200,000 simulated conversations and reported that every top open- and closed-weight model tested performed significantly worse in multi-turn conversation than in single-turn, with an average drop of 39% across six generation tasks. The Multi-IF benchmark found the same shape in instruction following specifically: one of the frontier models it evaluated scored 0.877 on the first turn and 0.707 by the third, and every model tested failed instructions more often with each additional turn. And SysBench, which studies whether models obey system messages, names the three failure modes directly: "constraint violation, instruction misjudgement and multi-turn instability". All three measure text, not speech, and none of them placed a phone call. What they establish is narrower and still decisive: instruction following is not stable across turns, and it degrades in the direction of the constraint being dropped rather than the conversation breaking. A phone call is a multi-turn conversation with constraints in the system message, so it inherits that shape and adds a latency budget on top of it.

A natural-language rule has no failure signal. This is the one that decides how expensive the architecture is to own. When the model interpolates something plausible, no exception is thrown, no monitor fires, no dashboard turns red. Benchmarks such as FollowBench exist precisely because judging whether a response satisfied each individual constraint requires deliberate machinery built for the purpose. Nothing supplies that machinery for free at three in the afternoon on a real call. The transcript reads well. The recording sounds professional. The outcome is wrong, and the first person to find out is whoever opens the diary tomorrow.

What does it cost when the call sounds perfect and the outcome is wrong?

Two public records show the shape of the cost, and neither is a vendor's marketing example.

In Moffatt v. Air Canada, 2024 BCCRT 149 (British Columbia Civil Resolution Tribunal, decision issued 14 February 2024), a customer relied on an airline chatbot's description of its bereavement fare policy. The description was wrong. Air Canada argued it could not be held liable for information provided by the chatbot. Tribunal member Christopher C. Rivers was unimpressed: "In effect, Air Canada suggests the chatbot is a separate legal entity that is responsible for its own actions. This is a remarkable submission. While a chatbot has an interactive component, it is still just a part of Air Canada's website." The finding was that "Air Canada did not take reasonable care to ensure its chatbot was accurate", and the airline was ordered to pay 812.02 Canadian dollars, of which 650.88 was damages.

The sum is trivial. The principle is not. A fluent, confident, entirely invented policy statement was treated as the business's own representation, and the business carried it.

The second is scale rather than liability. In March 2024 The Markup reported that New York City's official MyCity business chatbot was telling businesses to break the law. Asked whether an employer could take a cut of workers' tips, it answered yes. Asked about housing vouchers, it said buildings did not have to accept them. Asked about cash, it said there were no regulations requiring businesses to accept it. Each answer was fluent, on-topic and wrong. After the reporting, the city kept the tool publicly accessible and added a disclaimer to the site.

Translate that into a phone line. The failure modes are unglamorous and they are all the same shape: a booking made outside the hours you actually work, an appointment promised for a service you no longer offer, a condition of sale never mentioned because the model judged it irrelevant, a slot given to a caller who should have been routed to a person. None of these produce an angry call. They produce a quiet cost, absorbed by staff, discovered late. We catalogue the operational versions of this in the mistakes businesses make when choosing a voice agent.

Are AI receptionists reliable, or is that the wrong question?

It is close to the wrong question, because reliability in this category is not a property of the vendor, the voice or the model. It is a property of where the rules are held.

Buyers usually evaluate the axis they can hear. Does it sound human. Does it understand accents. Does it interrupt gracefully. Those are real and they are measurable, and we cover them in our review of voice technology accuracy figures. But recognition accuracy and outcome correctness are different axes, and a system can be excellent on the first and unbounded on the second. An agent that transcribes every word perfectly and then books a slot your policy forbids has a perfect accuracy score and a broken outcome.

So the honest answer to "are they reliable" is: as reliable as the weakest thing standing between a plausible sentence and a written record. If that thing is a paragraph of English, the answer is no more reliable than the model's interpretation on that particular turn. If it is an enforced constraint, the answer stops depending on the conversation at all. The vocabulary here, including the terms vendors use to blur the distinction, is in our voice AI glossary.

How can you tell which architecture a vendor is selling you?

You cannot tell from a demo. That is the whole difficulty. A demo is a simple call: a caller who wants a normal thing, at a normal time, with no conditions attached. Both architectures handle it, which is why demos in this category all look the same and why a well-written prompt genuinely does carry a surprising distance. The divergence lives in the calls a demo does not contain.

So ask instead. One question, one follow-up, and the follow-up is the part that works.

What should you look for in an AI phone system?

In rough order of how much money each one moves, and none of them are about the voice:

1. Where the rules live. The subject of this entire article. Everything below is downstream of it.
2. What the system does on an unknown request. A defined default beats a clever improvisation. "I will have someone call you back about that" is a correct outcome and is worth more than a fluent guess.
3. Whether it writes to your system of record or to its own dashboard. What separates taking a message from actually booking is whether the agent reads your real availability before it writes, which is the whole subject of booking into your system of record. An agent that logs a booking somewhere your staff do not look has moved the work rather than removed it. Our integrations page covers where the write actually lands.
4. What the escalation path is, and how quickly it triggers. The valuable behaviour is knowing when to stop, not knowing everything.
5. What runs before a change reaches a live caller. The difference between a system and a demo is what happens on the second Tuesday of the third month.
6. Where the call data goes and who can reach it. A separate decision with its own failure modes; ours is documented on the security page, and the adversarial angle is in our piece on voice agent penetration testing.

If you want this as a procurement instrument rather than an argument, the long form is our vendor evaluation checklist and the scoring method is in the evaluation guide.

What does doing this properly actually involve?

Less glamour than the demo suggests, and most of it happens before anyone writes a line of agent dialogue.

It opens with your policies rather than the technology. The rules your front desk already follows are mostly unwritten and partly contradictory, and the first honest output of the work is those rules written down: what may be booked, by whom, when, under what condition, and what happens when the caller asks for something outside all of it. That artefact is useful whether or not you ever automate anything, which is why we hand it over as a deliverable in its own right.

Then the rules with consequences attached are encoded, so that violating one is not a matter of how the sentence was phrased. The model handles the conversation. The policy handles the commitment.

Then it is tested adversarially against that policy, before a live number is involved. Not a happy-path script. Simulated callers who want the thing you stopped offering, who change the time twice, who attach a condition halfway through, who are ambiguous about which Thursday they mean, who push back when told no, who ask for a price. The point of the exercise is not to see whether the agent sounds good under pressure. It is to see whether the outcome still satisfies the policy when it does not. Failures are reported as policy violations, not as awkward transcripts, because a policy violation is a thing software can detect and a bad vibe is not.

And then the same set runs again before any change reaches a live number, including changes to wording. That is the part most buyers never think to ask about and the part that determines whether the thing still works in month six. How the pieces fit together is on our how it works page.

What a useful next conversation looks like

Not a demo. A working session: about forty-five minutes on your actual call flow, where we take the policies your front desk already follows, push on them until the edge cases show themselves, and write down what the rules turn out to be. You keep that written version whether or not anything else happens between us.

If something does happen next, it is deliberately small. One workflow, missed calls and after hours, roughly two weeks, and nothing else moves until that one is behaving. If you would rather hear it first, there is a live voice demo, and if you would rather send the flow across in writing, tell us what your calls look like.