AI in Smaller European Languages: What Breaks, and Why

A smaller European language is one that appears in every vendor's list of supported languages and in almost none of their published scoreboards. Lithuanian is the clean example: named by Google, absent from Anthropic's benchmark table, absent from OpenAI's multilingual evaluation set, and explicitly out of scope in Meta's Llama 4 licence. The practical consequence is that a team working in one of these languages has to run its own checks, because the vendor has not published the number it would need.

Published 5 September 2026. Last updated 5 September 2026. Every figure below links to the paper or vendor page it came from, each one read on that date.

What does it mean that a model “supports” Lithuanian?

Less than it sounds. Listing a language and measuring a language are two different acts, and only one of them is being done. Anthropic publishes a multilingual table covering 14 languages and states plainly that “Claude is capable in many languages beyond those benchmarked in the following table. Test with any languages relevant to your specific use cases.” That is an honest sentence, and it is also an instruction: the vendor is telling you the evaluation is yours to run.

The picture is the same across the market. OpenAI's multilingual MMLU results file records that “we translated MMLU's test set into 14 languages using professional human translators,” and no Baltic language is among them. Google lists Lithuanian among “more than 70 languages” for its consumer assistant with no quality figure attached. Meta's Llama 4 card names twelve supported languages, none of them Lithuanian, and marks use “in languages or capabilities beyond those explicitly referenced as supported in this model card” as out of scope, which turns a quality question into a licence question.

What does the newest EU-wide evidence say?

The most systematic answer available was published in August 2026. Researchers evaluated nine open-weight and commercial models as language agents playing goal-directed dialogue games in self-play across 30 languages, the 24 official EU languages plus six others, scored programmatically rather than by preference. Their headline finding is unusually blunt for an abstract:

“No open-weight model covers the EU-24 well: in every official language both commercial systems outscore every open-weight model, and the two weakest average below 40 points across the EU-24. The commercial systems stay ahead even in languages with four orders of magnitude less public web text, showing that linguistic parity is achievable, but not from public crawls alone.”

Two things follow. First, the deficit is not a law of nature. If a commercial system can hold a lead in a language with four orders of magnitude less public text than English, then the constraint is investment and data curation rather than the language itself. Second, and less comfortably, the same paper measures the price of that coverage: “pooled over models and languages, the median non-English language costs 31% more to run than English, and scores 10% lower.” Coverage, in the authors' phrase, “is also not parity of service.”

Listed, but not measured: the vendor scoreboard

How large is the gap, really?

The best data on this comes from Lithuanian and Latvian institutions rather than from a vendor. A 2025 study by researchers at Tilde and partner universities evaluated open-weight models for the three Baltic languages on translation, multiple-choice comprehension and free-form generation, and was accepted at NoDaLiDa/Baltic-HLT 2025. Its Belebele reading-comprehension table is the most quotable evidence in the field, because it shows that the answer is model-specific to an extreme degree.

Read the last row twice. A 14-billion-parameter model that answers 0.886 in English answers 0.001 in Lithuanian, which is not a degradation but a total failure, and nothing on the model's own page would have warned you. Meanwhile the leading commercial model in the same table sits about two points off its English score. The spread between the best and worst entry in the Lithuanian column is larger than the spread across the entire English column. Model selection matters more for a smaller language than it does for English, and it is the single decision with the largest effect.

The same study reports one more result that matters to anyone considering self-hosting: quantisation is not a neutral cost saving. The authors note that under reduced precision “less spoken languages like Lithuanian, Latvian, and Estonian are affected more” than English and Czech. Squeezing a model to fit a cheaper machine takes more away from the smaller language than from the larger one.

Estonia has now produced the equivalent for its own language. A benchmark published in late 2025 and revised in early 2026 (Lillepalu and Alumäe, arXiv 2510.21193) assembles seven datasets “generated from native Estonian sources without using machine translation” and evaluates six base models and 26 instruction-tuned models, with human raters showing moderate to high correlation with the automated scores. The methodological point travels: benchmarks translated out of English understate the problem they are measuring, because the translation is worst exactly where the language is weakest.

Why fluent output is not correct output

This is the finding that changes how a team reviews work, and it is easy to miss because it is buried in a human-evaluation section. Two native-speaker linguists counted grammatical errors, wrong inflections and invented words in free-form Lithuanian output. Even the strongest model produced 0.94 grammatically incorrect or incorrectly inflected words per 100; a large open model produced 8.01. The paper's own summary is the sentence to put in an induction deck:

“Users can expect at least one linguistic error in every 2-3 sentences from the best open-weight models like Gemma 2, or every sentence for models like Llama 3.1.”

The paper's abstract puts the same effect at roughly one lexical error in every 20 words across the multilingual options tested. In a highly inflected language, that error class is specific and reviewable: declensions, wrongly inflected proper nouns, and words that do not exist but look as though they should. A reader who is checking for meaning will skim straight past all three, because the text reads fluently. Fluency and factual correctness are separate axes, and in smaller languages they come apart in both directions.

What does a smaller language cost in tokens?

More than English, for the same content, before any model has been invoked. The foundational work here is a NeurIPS 2023 study from Oxford which showed that disparity “arises at the tokenization stage, well before a model is even invoked” and that the same text translated into different languages “can have drastically different tokenization lengths, with differences up to 15 times in some cases.” The consequences are named explicitly: unfair treatment “in regard to the cost of accessing commercial language services, the processing time and latency, as well as the amount of content that can be provided as context to the models.”

A 2026 cross-lingual audit puts current numbers on the ordinary case: for most non-English languages, “the same content costs 1.3-1.8x more tokens than in English.” The third consequence in that Oxford list is the one teams forget. A token premium is also a context-window premium: a long document in a morphologically rich language reaches the limit sooner than its English translation would, so the same workflow that fits comfortably in English can quietly stop fitting.

Work in the local language, or switch to English?

This is the question every Lithuanian, Latvian or Estonian team asks, and the honest answer is that it changed. The old advice, well supported on 2023-era open models, was to translate into English first, because comprehension in the local language collapsed. On a current frontier model that advice has reversed. In the 2026 audit (arXiv 2608.26175), Lithuanian ran at 98.3% accuracy natively against 91.7% when machine-translated into English first, while the English path used a fraction of the tokens. The author is careful about the strength of the claim, writing that “with n=60 items per language we read this arm as directional rather than definitive,” and that caution should travel with the number.

So the practical rule for 2026 has two branches. On a strong commercial model, work natively and treat translation as a cost lever rather than a quality lever. On a small, older, self-hosted or heavily quantised model, translating into English first is a genuine rescue rather than a preference, and the Belebele column above shows why. What you must not do is carry an answer from one branch into the other, which is exactly what happens when a team reads a 2023 blog post and applies it to a 2026 deployment.

What is Europe building about it?

Two efforts matter, and they are complementary rather than competing.

TildeOpen is a Latvian-led open-weight foundation model, released under a CC-BY-4.0 licence, with about 30.7 billion parameters, trained on 2 trillion tokens using 768 AMD MI250X GPUs on the LUMI supercomputer, covering 32 languages of which 19 are focus languages “representing over 165 million people.” (Tilde, TildeOpen LLM) Its most-quoted numbers need care. Tilde reports that “compared to LLaMA-3, it is 41% more efficient in Latvian, 37% in Lithuanian, 31% in Finnish, and 28% in Estonian and Polish,” and the very next sentence explains what that means: “this translates to faster text generation performance in local deployments and consequently lower running costs for the same amount of data.” These are tokeniser efficiency figures, not answer-quality figures. They say the model needs fewer tokens to express the same Lithuanian, which is a real and useful advantage in cost, speed and context length. They do not say it answers better. The model card adds the other half of the picture: it “is not yet adapted to follow instructions or aligned with safety features,” (Tilde, TildeOpen-30b model card) so it is a foundation for builders rather than an assistant for staff.

OpenEuroLLM is the public-funding answer. The European Commission describes it as producing “the first family of open-source Large Language Models that cover all official and future EU languages,” with “a total budget of €37.4 million of which €20.6 million comes from the Digital Europe Programme.” (European Commission, OpenEuroLLM) Alongside it, DG Translation published EU MMLU in July 2026 (European Commission, DG Translation, EU MMLU), a benchmark built by nearly 250 students from 21 universities through the European Master's in Translation network, covering over 1,000 questions in a named list that includes Croatian, Czech, Dutch, French, German, Greek, Hungarian, Irish, Italian, Lithuanian, Polish, Portuguese, Romanian, Slovak and Slovenian. Worth knowing before you plan around it: at the time of reading, that announcement published neither scores nor a download link. It is a commitment, not yet a result.

What a team should actually do

Six rules follow from the evidence above, and each one is doable this quarter.

1. Test the model in your language before you standardise on it. Twenty real tasks from your own work, scored by someone who speaks the language natively. The Belebele spread is the argument: two models with similar English scores can be 40 points apart in Lithuanian.

2. Never read an English benchmark as evidence about your language. This is the same error as reading a vendor's supported-language list as a quality claim, and no vendor in the table above publishes the figure that would settle it.

3. Review for inflection, not for readability. Brief reviewers on declensions, proper nouns and invented words. Text that reads well is not evidence of anything, and in a smaller language it is the specific thing that hides the error.

4. Budget the token premium. Roughly 1.3 to 1.8 times English for the same content, which shows up as cost, as latency, and as documents hitting the context limit sooner than the English version would.

5. Be conservative about quantisation and compression. Both were tuned on English, and both take more away from smaller languages, by the two sources cited above.

6. Give the model your terminology. A glossary of the terms your organisation uses, with the exact forms you want, costs an afternoon. It is standard localisation practice rather than a research finding, and it should be described that way rather than dressed up as an evidence-backed technique.

If you are choosing which assistant a team will use in the first place, the working models differ more than the marketing suggests: see what a team has to learn differently for Claude, ChatGPT, Copilot and Gemini. If the decision also touches customer data, the account-type question is settled in the tier-by-tier vendor reference, and the training duty it sits under is covered in Article 4 of the EU AI Act.

The state of the evidence, stated plainly

This field is thin, but it is thin in a describable way. What exists is a small, high-quality cluster: a systematic EU-24 evaluation, a Baltic evaluation produced by Lithuanian and Latvian institutions, a native Estonian benchmark, a tokenisation paper with a Baltic row, and a 2026 cross-lingual audit. Where two independent groups measured the same thing they broadly agreed. Those numbers are trustworthy.

What is missing is almost everything a buyer would want. No major vendor publishes a Lithuanian quality figure for any text model. The strongest “translate to English first” evidence comes from models that are now several generations old; the strongest counter-evidence comes from a 2026 preprint whose author calls that arm directional. The EU's own benchmark names Lithuanian and has published no scores. Any confident blanket statement about whether to prompt in Lithuanian or in English is over-claiming, and the accurate answer is that it depended on the model and the answer flipped between 2023 and 2026.

That is not a reason to avoid working in a smaller language. It is a reason to verify rather than to assume, and to distrust anyone who quotes a single number without saying which model it came from. For the applied side of this on the Baltic market, see multilingual voice agents for Baltic businesses, whether an AI receptionist can handle multiple languages, a Lithuanian-speaking agent in a business setting and AI that speaks Lithuanian. For the language question a step further out, multilingual business communication statistics covers how much of European trade already crosses a language boundary, and the AI glossary defines the terms used above.