Local AI or API: Which Should Power Your Assistant?
A model on your own server or an AI called from a provider? Compare speed, quality, cost and data handling before you build your assistant.
Before you build an AI assistant, you need to choose the engine that will write its answers. There are two paths: call an AI from a provider, through an API, or install a model on your own machine. Each path has real advantages and real tradeoffs. Here's how to decide, with the concrete example of Flopia, the Flopix assistant, which tried both: first a small model installed on a server, then Claude, Anthropic's AI, called through an API.
The two paths in one sentence
- API: your site sends the question to an AI provider, which sends back the answer. You don't manage any machine.
- Local: an open model runs on your computer or server, driven by software like Ollama. Everything stays with you.
The rest of the assistant (instructions, documentation, safeguards, tests) is the same either way.
What an API gives you
Quality
Providers offer very large models. They understand nuance, follow long instructions and reason through complex cases. For an assistant that has to analyze a technical problem or write subtle answers, that's a real plus.
Speed
Providers' servers are built for this. Answers usually arrive within a few seconds, and several visitors can type at the same time without waiting in line.
Simplicity
No machine to install, update or monitor. You get a key, you call the service, and that's it.
What an API costs you
- A pay-per-use cost, which depends on how much text is read and written. It follows your traffic: a busy month costs more. Pricing changes often, so check it with each provider.
- Data in transit: your visitors' conversations go through the provider. Read their terms on how that data is stored and used, and mention it in your privacy policy.
- Dependency: the provider can change a model, retire one or change its terms. Your assistant can start behaving differently without you touching anything.
What local gives you
Control over your data
Conversations never leave your server. That was the reason Flopix first chose this path for Flopia: keeping conversations on its own servers, in France.
Predictable costs
You pay for the machine, not the questions. Whether your assistant answers ten questions or a thousand, the server bill stays the same.
Stability
The model version only changes when you decide. An assistant that works today will work the same way tomorrow.
What local costs you
Flopia's case says a lot. Her first versions ran on a qwen3 4B model in its "instruct" version, served by Ollama, on a server with no graphics card. Testing revealed some very concrete constraints:
- Reading took time. On this processor, the model read around 20 tokens (pieces of words) per second. Every sentence of instructions and every documentation excerpt cost seconds of waiting.
- One question at a time. The server handled questions one after another. Flopia's test area announced 10 to 60 seconds per answer at the time.
- A small window. The context was set to 4,096 tokens in total: instructions, documentation, history and the answer all had to fit.
- A less obedient model. A 4-billion-parameter model follows instructions less well than a large one. It took short sentences, targeted scenarios and filters in the code.
These figures apply to that specific machine. With a graphics card or a more powerful server, everything changes, but so does the cost of the machine.
Why Flopia moved to an API
Despite all the tuning, two limits remained: answer quality and waiting time. The small model followed instructions imperfectly, filters were needed to catch its slips, and an answer could take a minute. So Flopix moved Flopia to Claude, Anthropic's model, called through an API.
What changed:
- Answers within a few seconds, even when several visitors type at the same time.
- More accurate answers and instructions that are followed better, including for questions typed in a hurry.
- Two model tiers: a fast model for simple questions, and a more powerful one when the assistant has to take action (creating or editing a website), where a misunderstanding is costly.
What Flopix accepted in return, and says openly:
- Messages go to the provider. The website and the database are still hosted in France, at OVHcloud, but the messages needed for each answer are sent to Anthropic, in the United States. Under Anthropic's commercial terms, data sent through the API isn't used by default to train its models. That's exactly the kind of information that belongs in your privacy policy.
- A pay-per-use cost, kept in check by usage limits and a daily spending cap.
- Dependency on the provider, which is why keeping your settings separate matters (see below).
The most interesting part: almost all the work done on the small model was reused as is. The instructions, the things she must never do, the scenarios, the documentation and the test bench were carried over, then tested again.
Lessons learned on the local model
While tuning Flopia on her small model, the Flopix team learned several things, and most of them still help with an API:
- Keep the general instructions identical from one question to the next. A local engine can then keep them in memory instead of rereading them, and the first answer arrives much faster. With an API, the same habit often lets you benefit from the caching the provider offers.
- Warm up a local model when the visitor opens the chat bubble, not on every page they visit.
- Read less: a few short, relevant documentation excerpts rather than whole pages. Less waiting locally, lower costs through an API.
- Answer without the model when possible: the questions behind the suggestion buttons have answers written in advance, shown instantly.
- Pick the right variant: some models come in a version that "thinks" at length before answering. On a processor, that thinking can take minutes. For a website assistant running locally, the variant that answers directly is the right one.
How to decide
Answer these questions, in order:
- Do the conversations contain sensitive information? If so, go local, or choose a provider with clear, documented guarantees.
- Are the questions simple and repetitive? Opening hours, directions, terms: a small, well-tuned model can be enough. Badly worded questions, complex analysis, actions to carry out: the API.
- What volume do you expect? Low and irregular: the API is often simpler. High and steady: a server can become worthwhile, as long as you accept its limits or invest in a suitable machine.
- Who will look after the machine? If nobody, the API takes that worry off your plate.
- How long will your visitors be willing to wait? In a website chat bubble, a few seconds is comfortable. A minute is long: it's one of the reasons Flopia left the local model behind.
The right habit: keep your settings separate from the engine
Whatever you choose, keep your settings (identity, rules, things it must never do, scenarios, documentation, tests) in a separate place, independent of the engine. The day you change models or paths, you rerun your tests and adjust, without rewriting everything. That's exactly what allowed Flopia to switch engines. To write those settings, read how to write your AI assistant's instructions, and for the bigger picture, adding an AI chatbot to your website.
The course Create your AI assistant (in French) devotes an entire lesson to this choice, then guides you through all the settings, whichever engine you go with.
Frequently asked questions
Can you run an AI on an ordinary computer?
Yes, software like Ollama lets you run open models on a computer or a server. Without a graphics card, you'll have to stick to small models and accept slow answers.
Is a local AI always better for data privacy?
It keeps conversations on your machine, which is a real advantage. But you have to secure the server yourself, limit who has access and decide how long you keep the conversations.
Can I switch to the other option later?
Yes, if your settings are kept separate from the engine. Instructions, documentation, scenarios and tests carry over from one model to another, even though you'll need to test them again. Flopix did exactly that with its assistant, moving from a small local model to an AI called through an API.