Bilingual AI Phone Agents: The Accuracy Problem Nobody Warns You About

A bilingual AI phone agent that claims to support French and English but was trained primarily on European French and standard American English will fail your Québec callers — subtly at first, then visibly. Most vendors advertise language count. The accuracy problem nobody warns you about is what happens to that accuracy when your callers actually speak.

Key Takeaways

• Word error rates jump 30–50% on code-switched speech — the sentences where callers mix French and English mid-thought — according to published ASR research on bilingual language pairs.
• Québec French significantly underperforms European French on the major speech models without fine-tuning or regional training data.
• Real phone calls reduce lab-measured accuracy by 5–10 percentage points due to telephony audio compression.
• Language-detection latency adds dead air during code-switched calls that callers read as a broken or confused system.
• "Supports bilingual" on a spec sheet means almost nothing. What matters is the training data, the dialect coverage, and how the system handles mid-sentence language switches.

---

What does "bilingual" actually mean for a phone agent?

Most vendors use "bilingual" to mean the agent has a French mode and an English mode. You pick one when you configure the system. Some allow the caller to choose. A smaller number detect the caller's language automatically at the start of the call.

None of those are the same as handling a bilingual caller.

The share of organizations deploying AI phone agents in customer service rose from approximately 39% in 2025 to 66% in 2026, according to AssemblyAI's contact center AI trends report. That rapid adoption means buyers are making deployment decisions faster than the market has produced honest accuracy data. Vendors are selling capability they measured in the best conditions and deploying into the worst ones.

A bilingual caller in Québec does not stay neatly in one language. They start in French, drop an English product name, switch back to French for the address, and use an anglicism for the company they are calling about. That is not a communication failure — it is how people talk. The question is whether your phone agent can follow it.

---

Why Québec French isn't just "French" to an AI

The speech recognition models powering most commercial AI phone agents are trained on large audio datasets. French is well-represented — but "French" in those datasets means, overwhelmingly, European French. The vocabulary, rhythm, liaison patterns, and pronunciation of Québec French are meaningfully different.

A 2025 study benchmarking large multilingual models on Québec French speech recognition found substantially higher word error rates compared to published benchmarks on European French — even for models widely considered strong multilingual performers. The gap was significant enough that models needed fine-tuning on Québec French data to approach the accuracy they advertised on standard French corpora. That fine-tuning is rarely done by generic voice agent platforms.

A separate peer-reviewed study published in JASA Express Letters found that Whisper's accuracy varied considerably by accent group: native American and Canadian English outperformed British and Australian English, while non-native and regional accents showed the largest performance gaps. The implication for Québec French is direct — accented, regional speech consistently falls below what benchmark tables suggest.

The specific failure surface in Québec includes:

• Common Québec surnames — Tremblay, Bouchard, Bélanger, Lavoie, Gauthier appear constantly on service business calls and are systematically mis-transcribed by models without regional training data.
• Embedded anglicisms — Québécois French mixes English terms naturally ("je vais checker ça," "le parking," "un meeting") in ways that confuse models expecting either pure French or pure English.
• Informal register — Joual and conversational Québec French differ substantially from the formal register that dominates training corpora.
• Rhythm and liaison — Québec French has distinctive speech patterns and liaison rules that affect how phonemes blend across word boundaries.

None of these are edge cases. They are the baseline of how Québec callers speak on a phone call to a contractor, clinic, or property manager.

---

The code-switching problem: when callers mix languages mid-sentence

Code-switching — alternating between two languages within a single conversation, or even within a single sentence — is the hardest technical problem for bilingual ASR systems to handle.

Research published in 2025 found that standard ASR systems show a 30–50% increase in word error rate on code-switched speech compared to monolingual input. Even multilingual models show semantic accuracy drops when forced to handle rapid language transitions, because the underlying architecture still makes language-level assumptions about what words are likely to follow.

The failure is structural. Most ASR systems identify the caller's language at the start of the call and maintain that assumption through the transcript. When the caller switches mid-sentence, the model is still predicting words in the first language. It hears French phonemes but looks for the French vocabulary that should appear there. The English word the caller used gets transcribed as whatever French sound it most resembles — or dropped entirely.

A caller saying "j'ai besoin d'un quote pour le driveway" may have "quote" and "driveway" transcribed as phonetic approximations or simply omitted. That corrupted transcript propagates up: the language model reading it tries to answer a different question than the one asked. The caller hears a response that doesn't match what they said. They repeat themselves or hang up.

---

What accuracy looks like in production versus the demo

Vendors demo clean, single-language, studio-quality calls. That is not what your phones ring.

ASR models that score 95% accuracy or higher in lab conditions typically drop to 85% or lower on real telephony audio, according to AssemblyAI's contact center AI research. Phone networks compress audio using codecs optimized for human intelligibility, not machine transcription. Artifacts that don't impair a human listener — background noise, compression artifacts, packet loss — can meaningfully degrade word error rates.

Add a Québec accent to a compressed phone call, and you stack two performance degraders. Add mid-sentence code-switching, and you have the most difficult conditions a bilingual phone agent will face on a routine Tuesday morning.

Condition	Reliability (without regional tuning)	Primary failure mode
Standard monolingual English	High	Rare vocabulary, heavy non-native accents
European French (standard)	Medium–High	Fast speech, regional variation
Québec French (base model)	Medium	Surnames, anglicisms, informal register
French–English code-switched	Low–Medium	WER spike at language transitions
Québec FR + EN mid-sentence	Low	Combined failure surface

The table above reflects performance patterns drawn from published ASR benchmarks and contact center deployment data — not vendor marketing materials. Your mileage will vary by model, but the rank ordering is consistent across the research.

---

The four signals that expose a weak bilingual agent

Before signing a contract with any bilingual AI phone agent vendor, run these four tests yourself.

1. Local surname test. Call the agent and give a Québec last name — Bélanger, Bouchard, Vaillancourt, Gagnon. Ask it to read the name back. Any mis-transcription or hesitation here indicates the training data is insufficient for your market. This is the first thing your callers will give the agent.

2. Code-switch mid-sentence. Ask a question in French that includes one English noun: "Est-ce que vous faites le snow removal?" or "J'ai besoin d'un quote pour le driveway." If the agent reads the question back incorrectly or ignores the English term, it will fail this same test hundreds of times a month on live calls.

3. Time the pause on language switch. Start a call in English, then switch to French mid-sentence. If there is a pause longer than 800ms while the agent processes the switch, the caller will notice it. Users expect voice interactions to respond in under one second, and a half-second gap during normal conversation signals a confused system, not a thinking one. Callers who interpret that pause as a glitch are more likely to hang up than to repeat themselves.

4. Emergency keyword parity in both languages. Say "j'ai une urgence" and separately "I have an emergency." Both should trigger the same escalation path immediately, without the caller needing to specify their language. If only the English version escalates, the bilingual system is not symmetric — and the FR-speaking caller in a high-stakes situation reaches a dead end.

---

Why this matters more than most business owners realize

Businesses miss roughly 62% of phone calls during business hours — and close to 100% after hours — according to missed call data compiled by Hicira. The entire premise of deploying an AI phone agent is to close that gap. But if the agent then fails to understand 15–20% of what callers say — because of accent, code-switching, or telephony degradation — you have spent money on a system that picks up calls and still loses the caller.

A service business in Québec that switched from voicemail to an AI phone agent for after-hours coverage learned this the hard way with a first deployment. The agent answered every call. It captured roughly half the caller intent. The address field in the CRM was wrong often enough that it required a callback to verify before dispatch. The time saved on reception was partially offset by the time spent cleaning up transcription errors downstream.

AI without the workflow underneath is just an expensive voicemail. AI with the workflow and poor transcription accuracy is something worse: it generates confident wrong answers.

---

What a properly built bilingual phone agent actually requires

The technical bar for reliable bilingual FR/EN phone handling in Québec is higher than most generic platforms invest in:

• Regional fine-tuning on Québec French — not generic French, not Canadian French, but Québec French specifically, with training data that includes regional surnames, anglicisms, and conversational register.
• Code-switching detection at the segment level — the ability to identify that a phrase or sentence is switching languages mid-stream and apply the appropriate transcription path.
• Named entity extraction with a regional lexicon — a post-processing layer that catches proper nouns and local terminology the base ASR model is likely to mis-transcribe.
• Symmetric escalation logic — every intent trigger must exist in both FR and EN, with equivalent sensitivity.
• Evaluation on telephony audio, not clean recordings — the test set must include codec-compressed, real-call audio, not studio microphone recordings.

Accuracy on a European French benchmark does not predict accuracy on a Québec PSTN call with code-switching. The evaluation environment and the deployment environment need to match.

Data residency adds another layer: a Loi 25-compliant deployment requires that audio, transcripts, and caller data stay within Québec. Most generic bilingual platforms route audio through US or European cloud infrastructure by default — which is a compliance problem, not just an accuracy one.

---

If you are evaluating bilingual AI phone agents for a Québec or bilingual Canadian business, speak with the team at Portico Intelligence. We build and operate AI phone agents for FR/EN bilingual markets — with the name recognition, code-switching handling, and Loi 25-compliant data infrastructure that generic platforms skip.