📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva, a sovereign LLM trained from scratch on extensive Italian data, demonstrates impressive technical results but underperforms on real-world academic benchmarks. This raises questions about the scale of native-language investment needed for effective country-specific AI models.
Italy’s Minerva project, a large-scale sovereign language model trained entirely from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored only 4.9% on the INVALSI Italian school-exam benchmark, highlighting significant challenges in achieving country-specific language understanding despite substantial investment.
The Minerva project, led by Sapienza University of Rome and supported by Italy’s national research and supercomputing infrastructure, trained models ranging from 350 million to 7 billion parameters. Despite the extensive training dataset and open access to weights and code, the 3B parameter model scored near chance on the INVALSI test, a standard measure of Italian academic language skills.
Researchers from the project stated that while dataset composition and scale are important, the results suggest that simply increasing data and parameters may not suffice for complex language tasks like academic testing. The findings imply that the current scale of native-language investment may be insufficient for producing high-quality, country-specific AI models.
This outcome contrasts with Italy’s substantial investment in infrastructure and data, raising questions about the effectiveness of scale alone in sovereign-LLM development and whether additional strategies or larger models are necessary.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Fine-Tuning Large Language Models: From Custom Datasets to High-Performance AI Models Using Modern Toolchains
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
Modern Computer Architecture and Organization: A systems-level guide to modern computer architectures, from hardware foundations to AI datacenters
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
NVIDIA DGX Spark™ – Personal AI Desktop Supercomputer – Desktop GB10 Grace Blackwell Chip
Supercomputer performance directly to your desk in a compact, energy-efficient design, enabling enterprise-scale AI and high-performance computing right…
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
From Weights to Wisdom: The Complete Guide to Running and Adapting Opensource AI Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign-LLM Strategies
The results from Minerva challenge assumptions that training large models on native-language data automatically yields high performance on complex language tasks. This has broad implications for European countries pursuing sovereign AI, suggesting that investments in scale and data may need to be significantly larger or more targeted to develop truly effective country-specific models.
It also highlights a potential structural gap in current approaches: without sufficient native-language scale, even well-funded projects may fall short on practical benchmarks, influencing future policy and research directions across Europe.
Background on European Sovereign-LLMs and Italy’s Approach
European efforts to develop sovereign large language models have been divided between approaches like Portugal’s AMÁLIA, which extends multilingual models with regional data, and Italy’s Minerva, which trains from scratch on extensive native-language data. Italy committed significant resources, including 128 GPUs on the Leonardo supercomputer, and made weights, data, and code publicly available from inception.
While Italy’s approach demonstrated technical prowess and infrastructure capacity, the low performance on the INVALSI benchmark exposes a key challenge: scaling data and parameters alone may not produce the desired country-specific knowledge depth, especially for complex tasks like academic assessments.
“Despite extensive training, the low score on the INVALSI test indicates a fundamental challenge in achieving country-specific language expertise.”
— Research team, Minerva project
Unresolved Questions About Native-Language Model Effectiveness
It remains unclear whether increasing model size beyond 7 billion parameters or further refining training strategies could significantly improve performance on complex, country-specific benchmarks. The exact threshold of native-language data and scale needed for effective models is still unknown, and ongoing research aims to clarify this.
Next Steps for European Sovereign-Language Model Development
The Minerva team is continuing to iterate on training methodologies, including ongoing experiments with continual learning and larger models. Future evaluations will aim to determine whether further scaling or different training approaches can overcome current limitations. Policymakers and researchers may need to reassess investment strategies based on these findings, emphasizing not just data volume but also quality and task-specific tuning.
Key Questions
Why did Minerva perform poorly on the INVALSI benchmark despite extensive training?
The results suggest that simply increasing data and parameters may not be enough for complex language understanding tasks; more targeted or larger-scale native-language training may be necessary.
Does this mean sovereign models are not worth the investment?
Not necessarily. The findings highlight the importance of scale and methodology, suggesting that strategic adjustments are needed rather than abandoning native-language models altogether.
Could larger models or different training strategies improve results?
Yes, ongoing research aims to test whether increasing model size beyond current scales or adopting new training approaches can enhance performance on country-specific benchmarks.
What are the broader implications for European AI policy?
The results imply that European nations may need to commit to larger-scale native-language data investments and more sophisticated training strategies to build effective sovereign AI systems.
Source: ThorstenMeyerAI.com
