In brief: large language models are almost always tested in English. IT4LIA AI Factory, together with the Sapienza NLP research group at Sapienza University of Rome, is measuring what actually happens when an AI has to understand, translate and behave well in Italian. The results (seven scientific papers and one service already up and running) show that the answer isn’t a given, and they matter directly to businesses, startups and researchers alike.

Let’s start with a fact that doesn’t get told often enough: when a company uses ChatGPT, Gemini or an open-source model for an Italian-language service, it’s using a product that was trained and tested mostly in another language. Safety tests, quality benchmarks, academic evaluations: the vast majority are built around English. Italian, like almost every other language in the world, remains a kind of no-man’s-land: everyone assumes the AI will manage, but few bother to dig into how well it actually performs, not least because of training pipelines centred on English.

This is where IT4LIA AI Factory comes in: the Italian initiative that combines supercomputing (the Leonardo, LISA, GAIA, IT4LIA and MEGARIDE systems, plus the new IT4LIA system) with applied research to build an artificial intelligence that works for Italy, not just “also” for Italy. The Sapienza NLP Group, led by Professor Roberto Navigli, is one of the scientific engines behind this work. Over the past year the team has produced seven papers accepted at the field’s leading international conferences (AAAI, EMNLP, LREC, EVALITA, CLiC-it), plus one service that’s already operational. Here they are, one by one, and why they should matter to you even if you’re not a researcher.

An AI can be cautious in English and careless in Italian

First experiment, first warning sign. The study “Mind the Language Gap” (Marafatto & Navigli, LREC 2026) put four language models to the test with “sensitive” questions written specifically in Italian, the same kind of questions a well-behaved model, in English, should refuse to answer or handle with caution. The result: models don’t behave the same way in both languages. A model that’s cautious in English can turn out to be either more permissive, or more clumsy, in Italian.

Why it matters for businesses: if you’re integrating a chatbot or AI assistant into a product aimed at Italian customers, the model’s “default” safety may not be enough. It needs to be verified in the language it will actually be used in, through thorough benchmarking across safety categories.

What’s interesting is that this research hasn’t stayed locked in an academic drawer: it already underpins SECEV4LIA, a “red-teaming” tool (that is, aggressive testing designed to expose the weak points of an AI system), and it’s being used to train an Italian “examiner” that evaluates the responses of models in the Minerva family. In this case, research and infrastructure are talking to each other within a matter of months.

Why AI, when it translates, tends to “go American”

Ask a language model to translate a text full of specific cultural references (a dish, a festival, a figure of speech), and you’ll often see it drift toward whatever it knows best, usually the Anglo-American world. The study “Cultural and Knowledge Biases in LLMs” (Xu, Moroni & Navigli, LREC 2026) measured this phenomenon systematically, classifying content into three levels, from universal to deeply rooted in a specific culture, and looking at how much it helps to feed the model extra external knowledge so it doesn’t “flatten” everything toward a single point of view.

Why it matters for startups: anyone building translation, localisation or content-generation products for the Italian market (or for any other non-English-speaking market) is working on exactly this problem, often without realising it. Cultural specificity shows up with real impact in, among others, the banking, medical, legal and agrifood domains.

“Penna” as a pen or as a wing: can AI still tell the difference?

A word can have more than one meaning, and figuring out the right one from context is one of the oldest problems in language processing: it’s called word sense disambiguation. At the AAAI 2026 conference, one of the most prestigious in the world for artificial intelligence, Roberto Navigli asked a direct question: with today’s LLMs, is this problem solved? The answer: largely, but not entirely. The most advanced models now come close to specialised systems, but they still stumble on less common meanings, a mistake that also affects the quality of machine translation and one that’s hard to spot.

Why it matters for researchers: the paper proposes using this problem not as a challenge to “solve and shelve”, but as a genuine diagnostic tool for measuring how well a model actually understands language, rather than simply producing plausible-sounding text.

Getting an AI to read an entire novel, and checking whether it understood

Summarising a paragraph is easy. Understanding an entire book (characters, subplots, a detail on page 12 that resurfaces on page 340) is a different story altogether. Two papers tackle exactly this:

• INDAQA2 (Gioffré, Moroni and colleagues, presented at EVALITA 2026 as a CALAMITA challenge) is a large Italian benchmark for reading comprehension on long narrative documents, with summary questions, specific detail questions and questions about characters.
• LiteraryQA (Bonomo, Gioffré & Navigli, EMNLP 2025) does complementary work on the international stage: it cleans up a widely used benchmark that’s full of inaccuracies, and finds that the best way to judge an AI’s answers on a long text is… to ask another AI for its opinion, rather than relying on traditional automatic metrics.

Why it matters for businesses: if you work with long documents (contracts, minutes, reports, archives), this is the difference between an AI that “summarises as best it can” and one that genuinely understands what it’s reading.

Easy tasks are not a solved problem

It’s tempting to assume that today’s most powerful models have long since moved past the “easy” challenges and should only be tested on the complex ones. Multi-LMentry (Moroni and colleagues, EMNLP 2025, with Navigli among the authors) proves otherwise: it extended a battery of tasks that are trivial for a human (counting letters, ordering words, simple logical deductions) to nine languages, Italian included, and found that many open-source multilingual models still fall short of human performance, with errors that stay completely invisible if you only look at English.

Why it matters for anyone evaluating AI models (businesses and researchers alike): a model can look highly competent on complex benchmarks and still stumble on trivial tasks in a language other than English. It’s worth checking both.

Keeping track of who’s who across hundreds or thousands of pages

The final piece: xCoRe (Martinelli, Gatti & Navigli, EMNLP 2025) tackles a very concrete problem for anyone working with large volumes of text: figuring out when “he”, “that person” or a different name all refer to the same entity, even hundreds of pages apart or across different documents. Existing systems handle short texts well but struggle with long documents or collections spanning multiple documents. The new model unifies all of this into a single system, achieving the best results ever recorded on this type of task.

Why it matters for businesses: this is the underlying technology for analysing large document archives, think of a public administration, a law firm, a bank or a research centre that needs to cross-reference information scattered across thousands of documents.

From research to product: generating synthetic data on demand

Not everything comes out of academic papers. At SummerSOC 2026, Francesco Leotta (La Sapienza) presented a tutorial showing how the IT4LIA infrastructure can offer synthetic data generation as a ready-to-use service for businesses and researchers. One example from the talk is particularly instructive: a synthetic dataset on bank loans, generated carelessly by a language model, turned out to be biased, it incorrectly linked people’s gender to income and creditworthiness. From there, the talk walks through a series of techniques for fixing the problem, alongside a direct link to SECEV4LIA and the Minerva models mentioned above.

Why it matters for startups: generating good-quality synthetic data, free of hidden bias, is often the real bottleneck for anyone who wants to train a custom model without access to huge amounts of real data (and without the legal risks that come with it).

The full tutorial is available on YouTube, “Dataset generation as a Service”, Francesco Leotta (Sapienza) @ SummerSOC 2026, along with the presentation slides.

Seven scientific papers and one operational service tell the same story from different angles: bringing artificial intelligence to Italian isn’t just a matter of translating an interface, it means verifying, with scientific rigour, that models are safe, fair and genuinely capable of understanding language, from the nuance of a single word all the way to an entire novel. It’s the often invisible work that makes an AI Factory something more than a supercomputer: a laboratory that measures, with hard numbers, just how much is still left to do before an AI “in Italian” is truly an AI built for Italy.

Sources, for further reading