Multilingual voice: Urdu, Arabic and English

0:00 Multilingual voice

Listen to how people actually talk in Karachi: mera order abhi tak nahi aaya, can you check the status? Or in Dubai, where a caller slides between Gulf Arabic and English in one sentence. In Pakistan, the UAE and Saudi Arabia, multilingual and code-switched speech is the default, not the exception. An agent designed for one language at a time will fail a lot of your callers. In this lesson, you'll learn where multilingual voice breaks, how to handle scripts, numbers and dates, how to pick register and politeness, and how to evaluate per language.

0:41 Why multilingual by design

Why is multilingual design a must in these markets? Because your callers don't speak one language at a time. An agent that only works well in English quietly fails a large share of real conversations in Pakistan and the Gulf, and those failures often go unnoticed in overall averages. Designing for code-switching, dialects and scripts from day one is far cheaper than retrofitting it after complaints.

1:10 Where it breaks

Problems appear at every layer. Speech recognition struggles with dialects, like Gulf, Najdi, Egyptian or Levantine Arabic, regional Urdu accents, code-switching, names and numbers. The fix: test on real recordings from your callers, pick models strong in your dialects, and allow keypad input for numbers. Language detection gets confused by short words like okay or haan, so detect on longer turns and avoid flip-flopping. The language model needs clear instructions on which language and register to reply in. And text to speech needs voices tested in each language, with pronunciation help for names and brands.

1:51 Analogy: a multilingual front desk

An analogy: multilingual voice design is like staffing a front desk in a busy mall in Dubai or Karachi. The best receptionists switch languages naturally, use the English word when it's clearer, greet people appropriately, and don't make a customer choose a language from a menu before they can speak. But the official notices on the wall are professionally translated and checked. That's the balance: natural, flexible conversation, and fixed, reviewed wording for anything legal.

2:24 Scripts and formats

Watch scripts and formats. Urdu is written in Perso-Arabic script, right to left, but Roman Urdu, in Latin letters, is common in messages, and recognition may output either. Decide what your systems expect. Arabic speakers may use Western or Eastern Arabic digits, so normalize before calling tools. Callers may mention dates by the Hijri calendar, especially around Ramadan and Eid, so resolve carefully and confirm the Gregorian date. And names are transliterated many ways, like Mohammed, Muhammad or Mohamed, so match flexibly in CRM lookups.

3:01 Register and politeness

Register and politeness matter as much as accuracy. For spoken service in the UAE and Saudi Arabia, many businesses use a friendly, lightly formal Gulf style. Modern Standard Arabic can sound stiff on the phone, but it's safer for legal statements. Decide deliberately and test with local listeners. In Urdu, the respectful aap form is standard, and English technical terms like OTP, account or order are often clearer than pure Urdu. And greetings count: assalam o alaikum, with the right response.

3:36 Code-switching strategies

You have three code-switching strategies. Mirror: reply in the caller's dominant language, allowing common English terms. That's the most natural. Ask once: would you prefer Urdu or English? Simple, but slightly robotic. Or primary with switching: default to Arabic in Saudi Arabia and switch to English when detected. Some platforms offer special handling for mixed-language speech and language detection tools. Test what works on your callers. And one rule holds regardless: AI disclosures, recording notices and legal statements use approved, native-reviewed translations. Never let the model improvise them.

4:14 Worked example: Pakistani telecom

A Pakistani telecom built a bill-inquiry agent. It greets in Urdu with an English option, and introduces itself as an AI assistant. The team tested speech recognition on two hundred real call snippets from Karachi, Lahore and Peshawar, chose the model with the fewest errors on numbers and names, and added keypad entry for account numbers. The prompt allows English words like bill, package and data inside Urdu replies. One voice handles both languages, with a pronunciation dictionary for package names. And success is measured separately for each language, to catch hidden gaps.

4:54 Evaluate per language

The lesson text includes a starter multilingual evaluation set, with Urdu from Karachi, Lahore and Peshawar, Gulf and Najdi Arabic, English, and code-switched lines, each with the expected intent and reply language. Record them, or use real consented samples, run them through your agent, and score intent accuracy, reply language and task success per language and dialect. Never report a single overall success rate. It can hide a language that's failing badly.

5:25 Example 2: Dubai clinic dialects

A second example. A Dubai clinic's agent handles Arabic and English. Testing showed Egyptian and Levantine Arabic speakers were misunderstood more often than Gulf speakers, because the chosen speech model was tuned toward Gulf speech in its test set. The team added test recordings from each dialect, compared two recognition models, and switched to the one with more balanced accuracy. They also added a gentle fallback: if recognition confidence is low twice, the agent offers English or a human. Success rates evened out across dialects.

6:02 Common mistakes

Common multilingual mistakes. Testing only in English and assuming other languages just work. Letting the model translate mandatory legal statements on the fly. Flip-flopping languages on short words like okay or haan. And reporting a single overall success rate that hides a language or dialect that's failing badly. Measure per language, and fix the weakest first.

6:26 Watch me do it: per-language testing

Watch me do it. I take the multilingual evaluation set and adapt it for Nova Dental: fifteen utterances, five each in Gulf Arabic, Egyptian Arabic and English, plus three code-switched lines like hi, abi ahjiz appointment for cleaning. I ask colleagues from different backgrounds to record them on their phones, with consent to use them for testing. I run all of them through the agent and fill a results table: intent correct, reply language correct, task completed. English is fine. Gulf Arabic is mostly fine. Egyptian Arabic fails more often, mostly on dates. So I make three changes in the config. In the prompt, I add: reply in the language the caller uses; for Arabic, use friendly, clear Modern Standard Arabic with common Gulf expressions. I enable the language detection system tool, restricted to switching early in the call to avoid flip-flopping. And I replace the disclosure and recording lines in Arabic with the version our native-speaking colleague approved, fixed in the first message rather than generated. Then I rerun the full set and report success per language, never as one average. Egyptian Arabic improves but remains the weakest, so I note it as the top item for next week, with a plan to compare a second speech recognition model.

7:57 Recap and next step

Recap. In our markets, multilingual and code-switched speech is normal. Test recognition on real dialects, handle scripts, digits and Hijri dates, choose register deliberately, pick a code-switching strategy, and use approved translations for mandatory statements. Your next step: adapt the evaluation set to your callers, record at least five utterances per language or dialect, run them, and compare success rates. Fix the weakest language first.