Abu Dhabi: Researchers at Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) have developed the first benchmark designed to measure how effectively artificial intelligence models understand Arab culture and communicate across regional Arabic dialects, marking a significant step in the advancement of Arabic-language AI.The new benchmark, known as ArabCulture-Dialogue, evaluates AI models' ability to understand and engage with Arab cultural contexts through Modern Standard Arabic (MSA) and 13 national dialects from across the Arab world.The study revealed a significant gap between AI models' ability to understand what Arabic speakers say and their ability to communicate naturally in the dialects used in everyday life.Addressing a major gap in Arabic AIThe research was led by Professor Fajri Koto, Assistant Professor in the Department of Natural Language Processing at MBZUAI, and Muhammad Dehan, a researcher in the same department.Dehan said Arabic is spoken by more than 400 million people worldwide, yet many AI models are trained primarily on Modern Standard Arabic, limiting their ability to operate effectively across the region's diverse dialects and cultural contexts.To address this challenge, the research team recruited 26 native Arabic speakers from 13 countries to create authentic multi-turn conversations covering 12 areas of daily life, including weddings, food, parenting, agriculture, arts and games.Testing AI's cultural understandingResearchers used the benchmark to evaluate AI models through three tasks:Selecting the culturally appropriate response from multiple options.Translating between Modern Standard Arabic and specific dialects.Continuing conversations in a requested dialect.Professor Koto said the strongest models performed well when identifying culturally appropriate responses, achieving scores in the mid-90% range even when conversations shifted between MSA and dialects.However, performance fell significantly when models were asked to generate dialectal Arabic."The strongest models did well at recognising the culturally appropriate answer, scoring in the mid-90s, even when the conversation shifted from MSA to dialect. But when we asked them to produce dialect, for example to translate a line into Emirati, or to continue a conversation in that dialect, performance dropped sharply," Koto said.Emirati and North African dialects among the most challengingThe study found that AI systems performed better when dealing with customs and cultural practices shared across the Arab world.Country-specific customs proved more challenging, while dialogues featuring North African dialects were among the most difficult overall.Emirati Arabic was also identified as one of the more challenging dialects for AI models.Researchers found that the models successfully generated the requested dialect in only about half of all cases.Dehan noted that the cultural knowledge already exists within many models but often requires additional contextual guidance."The paradox is that the cultural knowledge is already present within the models, but they sometimes require only a small amount of guidance."He added that specifying the country or region associated with a conversation improved accuracy.Importance for the UAE's AI ambitionsThe findings carry particular significance for the UAE, which continues to position artificial intelligence as a strategic national priority through initiatives including the UAE National Strategy for Artificial Intelligence 2031 and the development of locally built AI models such as Jais.Professor Koto said the study highlights an important lesson for developers of Arabic-language AI systems.A model that claims to support Arabic does not necessarily understand the language across its many dialects, communities, cultures and identities.Presented at leading AI conferenceThe research was published in a paper titled "Cultural Benchmarking of LLMs in Standard and Dialectal Arabic Dialogues", which was presented at the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2026).Researchers said the benchmark provides a foundation for future work aimed at developing AI systems that better understand the linguistic and cultural diversity of the Arab world while helping preserve local identities and cultural heritage.
