Local AI or the cloud? Our AI developer on sovereignty and data

The most powerful AI models come from the United States and China. For many organisations, that is not a problem. For government it is, and just as much for companies that work with customer data, contracts or commercially sensitive information. What do you do if you want to use AI, but don't want your data to fall under foreign law?

We asked Thomas, AI developer at Humanoids. He currently works at IVO Rechtspraak, the IT organisation of the Dutch judiciary. There, he helps build an AI solution that processes confidential documents securely. This is part 1 of our conversation. Part 2 is about AI agents.

Sovereignty: who has the final say over your data?

Sovereignty is a country's ability to make its own decisions, without other powers having any influence over them. Digitally, that means: can we deliver our services without another country having a say in them?

"With AI, that becomes concrete straight away," says Thomas. "The most widely used models belong to American companies. Every feature of an AI assistant you build on them is a data exchange with an American system."

Having the servers in Europe changes little. Many providers now offer storage and processing within the EU. But as American companies, they fall under American law, such as the CLOUD Act. That law gives the US government broad powers to request data, even when it is stored in Europe. The rule of thumb within government is therefore simple: confidential data doesn't go in. And it is not just government. A law firm, bookkeeping firm or engineering consultancy also wants to know where its clients' files end up.

Local AI in plain language

AI has become an umbrella term. Usually we mean language models, like the model behind ChatGPT. "A model like that has read enormous amounts of text: books, scientific articles, forum discussions. After that, it can neatly complete whatever text you put in. That's the technology underneath."

The big players, such as OpenAI, Anthropic and Google, offer their models from the cloud. There are also many open models you can run yourself: on your laptop, on a specialised computer or on your own server. "You can unplug your router and still talk to an AI assistant. It simply runs on your own hardware."

One caveat: many of the best open models come from China. Because they run entirely on your own hardware, no data goes to the maker. But if you take sovereignty seriously, you also look at where a model comes from and how it was trained.

The hybrid approach

In practice, it is often not either-or. At IVO Rechtspraak, Thomas is helping build a solution in which a local model first strips documents of personal data. Names, addresses and payment details are removed. Only then does the query go to a powerful model in the cloud.

"For a first filter like that, a simple local model is fine. For a truly substantive answer, for example one based on scientific literature, a small model like that is too limited. Then you pass it on to one of the large models."

Local or cloud: what do you get?

A local model is not a free alternative to the big models. The difference lies in quality and in hardware. A local model with 8 billion parameters will answer a question just fine. Whether that answer is correct is another matter.

"A language model gives the most probable answer. There is no checking built in. As a developer, you have to build that in yourself, test it and improve it. The big players already have all kinds of systems for that, with far more computing power. You won't get anywhere near that locally."

Hardware matters too. "You can already run a model on a MacBook, even with speech input and output. With a specialised device of around €4,000, or a Mac Mini optimised for it, you can run bigger and smarter models. If you want the same level as the big providers, you practically need a data centre. And demand for suitable components is so high that prices have risen sharply."

When is a local model enough?

For well-defined tasks, such as answering questions based on your own documents, a local model is often more than enough. "But if you want an AI that can handle any question, with up-to-date information and complex problems, you'll still end up with the frontier models." By that he means the most powerful models available today, such as those from OpenAI, Anthropic and Google.

That is exactly what makes local AI interesting for small and mid-sized businesses. A local setup costing a few thousand euros is manageable for many companies. For well-defined tasks, such as searching internal knowledge or drafting standard documents, that is often enough. It lets you start with AI without handing over customer data.

A European alternative on the rise

Within the Dutch government, there is no clear strategy yet on moving away from American models. But Thomas is watching Europe with interest. France's Mistral is the key player there. "Their latest models are getting closer and closer to the frontier models, and they run on European soil. That offers real opportunities."

Some Mistral models are also open, so you can run them entirely yourself.

No off-the-shelf solution

Local, cloud or a combination: the right choice depends on your data, your processes and your goals. "Despite all the promises in the AI world, it doesn't work the same for everyone. How you set it up is different for every organisation."

That is exactly what we do at Humanoids. With UX research, we map out which services and processes lend themselves to AI, and which data needs to stay confidential. Our developers then build the solution: local, in the cloud or a combination of both. We do this for government organisations like the Dutch judiciary, and just as much for mid-sized businesses that want to start small.

In part 2, Thomas talks about AI agents: what they are, where they add value and why you should always keep a human in the loop.

Want to know which of your organisation's data should stay local, and where the cloud will do? Book an exploratory call with us.