Local AI: technical guide, settings and benchmarks

Daniel Platteau

Updated on 23 October 2026. Inference engines change every month: we revise this article regularly (update log at the end of the page).

This article follows on from our comparison Local AI: what hardware for what budget?, written for business owners. Here we address the person who will set up the machine: how to estimate performance before buying, tune the engine, serve several users and avoid exposing the server.

The rule of thumb: memory and bandwidth

To produce each token (a piece of a word), the engine has to read all the model's active weights from memory again. Generation speed therefore has a simple ceiling:

tokens per second ≤ memory bandwidth ÷ size of the active weights

Examples:

• A dense 27-billion-parameter model in 4 bits weighs about 16 to 17 GB. On a Mac mini M6 (170 GB/s advertised), the ceiling is about 10 tokens per second: published measurements give 8 to 9 (StarryHope).
• A 70 GB model in 8 bits on an NVIDIA DGX Spark (273 GB/s) cannot exceed about 3.9 tokens per second: 2.7 was measured (Spawning Point).
• gpt-oss-120b only activates about 5.1 billion parameters per token: despite its 65 GB, it reads only 3 to 4 GB per token, hence 65 to 88 tokens per second on a 128 GB M5 Max (Hardware Corner).

The rule has two limits. It says nothing about prompt processing (prefill), which depends on compute power: this is where NVIDIA cards crush Macs (6,512 versus 1,325 tokens per second on gpt-oss-120b, RTX Pro 6000 versus M5 Max, Hardware Corner). And real measurements stay below the ceiling, often by 10 to 30%.

Quantisation: what you keep and what you lose

A model is published at 16 bits per parameter (BF16). It is compressed to 8, 6, 5 or 4 bits so that it fits in memory and runs faster.

Two native formats deserve a mention. MXFP4 is the format gpt-oss was published in: it is the "official" version of the model, not an after-the-fact compression. NVFP4 is accelerated by NVIDIA Blackwell cards (RTX 50 series): with multi-token prediction, Qwen3.8 27B exceeds 120 tokens per second on an RTX 5090 (DataCamp).

Rule of thumb: Q4_K_M by default, Q5 or Q6 if memory allows, never below Q4 for professional use without testing first. Published rankings are mostly based on English: test in French and Dutch, on your own documents.

Context also costs memory

Every token of the conversation and of the documents provided takes up memory (the "KV cache"). On a long document, this cache can outgrow the model itself.

• Quantise the cache: q8_0 roughly halves its size compared with the default f16, q4_0 divides it by four (Ollama documentation). q8_0 is the right default; q4_0 degrades more.
• Architecture matters: gpt-oss-20b uses sliding-window attention. On an RTX 5060 Ti it takes 12.7 GB with 4,000 tokens of context and 12.8 GB with 64,000 (ComputingForGeeks): long context is almost free.
• The Ollama trap: its default window is 4,096 tokens. Beyond that, the beginning of the document is silently ignored. To summarise a 30-page contract, increase OLLAMA_CONTEXT_LENGTH (or num_ctx) and check that everything still fits in graphics memory.

MoE models: putting the experts in the right place

A mixture-of-experts (MoE) model has an always-active part (attention, shared experts) and "routed" experts, which are large but rarely used. When the model does not fit on a 16 GB card, you keep the active part on the card and send the experts to main memory.

With llama.cpp:

• --cpu-moe: all experts in main memory;
• --n-cpu-moe N: the experts of N layers in main memory, the rest on the card;
• -ot "exps=CPU": the same with a regular expression, for fine control.

The recommended method (Doctor-Shotgun's Hugging Face guide): load everything onto the card (-ngl 999), send all the experts to main memory, then bring them back layer by layer onto the card until graphics memory is almost full, leaving room for the KV cache.

Measured result on an RTX 5070 Ti 16 GB with Qwen3.6 35B-A3B in Q4: 43 tokens per second with standard settings, 112 with an optimised expert cache (GenerelSchwerz). The PC's main memory (preferably DDR5, 32 GB minimum) then becomes a performance component.

Which engine to choose

On that last point, a measurement contradicts a common belief: on a Mac with a 128 GB M5 Max chip, llama.cpp was 10 to 24% faster than MLX for Qwen3.6 (stared). Compare both on your own machine before choosing.

All these engines expose an API compatible with OpenAI's (/v1/chat/completions, /v1/embeddings…). The French YouTuber xavki covers it in detail in his vLLM series: tools written for ChatGPT switch to the local server by changing the address and the key.

Typical configurations, tier by tier

The tiers follow those of our comparison.

Mac mini M6 16 or 24 GB. Ollama is enough:

ollama pull qwen3.5:9b      # 16 GB
ollama pull gpt-oss:20b     # 24 GB

On a Mac, the GPU can only use part of the unified memory by default, around 75% (the proportion varies with the macOS version and the amount of memory). You can raise this ceiling (example for 20 GB on a 24 GB machine):

sudo sysctl iogpu.wired_limit_mb=20480

The setting is not officially supported and is lost on restart. Leave at least 4 to 8 GB to macOS on these small machines, and return to the default value with iogpu.wired_limit_mb=0.

RTX 5060 Ti or 5070 Ti 16 GB PC. gpt-oss-20b fits entirely on the card with 64,000 tokens of context. For Qwen3.6 35B-A3B, llama.cpp with expert placement:

llama-server -m Qwen3.6-35B-A3B-Q4_K_M.gguf -ngl 999 --n-cpu-moe 20 \
  --ctx-size 32768 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 \
  --host 127.0.0.1 --port 8080 --api-key <your-key>

Lower --n-cpu-moe as far as graphics memory allows.

Mac Studio M5 Max 36 GB. Qwen3.8 27B in Q4_K_M (17.1 GB), with vision:

llama-server -m Qwen3.8-27B-Q4_K_M.gguf --ctx-size 32768 -ngl 99 --jinja \
  --cache-type-k q8_0 --mmproj Qwen3.8-27B-mmproj-bf16.gguf

Enable multi-token prediction (MTP) if your version of llama.cpp supports it: about ×1.8 on this model according to a series of 45 settings tested on an RTX 5090 (Context Studios), with no loss, since every proposed token is verified.

Mac Studio M5 Ultra 96 GB. gpt-oss-120b (about 65 GB in MXFP4) leaves some twenty gigabytes for context and a second small model (for example for the embeddings of a RAG setup).

Serving several users

A team server receives simultaneous questions. Total throughput then rises sharply, while speed per user drops slightly.

• Ollama: OLLAMA_NUM_PARALLEL is 1 by default; raising it multiplies the memory reserved for context.
• llama.cpp: -np N creates N request slots that share the total context.
• vLLM: designed for this (continuous batching of requests), mainly used on NVIDIA cards.

Published measurement: on an RTX 5070 Ti, gpt-oss-20b goes from 144 tokens per second for one request to 384 in total for four simultaneous requests (GenerelSchwerz).

Security: do not expose the server

This is the most neglected point. In February 2026, LeakIX counted 12,269 Ollama servers reachable on the Internet without authentication. About a thousand ran a version vulnerable to CVE-2024-37032 ("Probllama"), which can lead to remote code execution.

• Ollama listens on 127.0.0.1 by default and has no authentication. Switching it to OLLAMA_HOST=0.0.0.0 to make it reachable in the office opens it to the whole network, and to the Internet if a port is forwarded. Put it behind a reverse proxy with authentication, or only give access through a VPN.
• vLLM: --api-key only protects the /v1, /v2, /inference and /cohere routes. The documentation cites /invocations, which gives access to the same inference without a key, as well as administration routes (/pause, /update_weights…). It recommends not relying solely on --api-key and filtering routes with a reverse proxy (nginx, Envoy…). xavki gives the same warning in his video.
• llama.cpp: --api-key accepts several keys; it listens locally by default.
• Docker: according to LeakIX, most containerised Ollama deployments run as root; a flaw in the engine can then give control over the host.
• Updates: Ollama, llama.cpp and vLLM release frequent versions, sometimes with security fixes. Schedule them as for any server.

Reference published measurements

Generation, single user, unless stated otherwise.

Common misconceptions

• "A PC with an NPU runs large models." False: the NPU handles small Windows features, not language models (see our article on AI PCs, in French).
• "MLX is always faster on a Mac." Not always: llama.cpp beat it by 10 to 24% on an M5 Max for Qwen3.6.
• "An API key is enough to protect vLLM." False, according to its own documentation.
• "More parameters always means slower." False for MoE models: gpt-oss-120b answers faster than dense models four times smaller.

Test yourself

At ExsIT

The first contact is free: we listen to your needs, then propose an action plan. For a local AI: choice of hardware and model according to your documents and languages, installation, tuning, secure access through a VPN or proxy, backup and updates.

Update log

• 23 October 2026: first version.

Sources

Apple. (n.d.). Mac Studio – Technical specifications. https://www.apple.com/mac-studio/specs/

AppleInsider. (2026, 29 September). M5 Max Mac Studio review: Far better choice than Ultra for most desks. https://appleinsider.com/articles/26/09/29/m5-max-mac-studio-review-far-better-choice-than-ultra-for-most-desks

ComputingForGeeks. (2026). Ollama models cheat sheet: Best models by VRAM. https://computingforgeeks.com/ollama-models-cheat-sheet/

Context Studios. (2026, 19 August). Qwen 3.8 27B : le guide hardware, de la RTX 3090 au DGX Spark. https://www.contextstudios.ai/fr/blog/qwen-3-8-27b-hardware-guide

DataCamp. (2026). Så kör du Qwen3.8-27B lokalt på en NVIDIA RTX 5090. https://www.datacamp.com/sv/tutorial/how-to-run-qwen3-8-27b-locally

Debian. (n.d.). llama-server(1) [manual page, llama.cpp-tools]. https://manpages.debian.org/unstable/llama.cpp-tools/llama-server.1.en.html

Doctor-Shotgun. (2026). Performant local mixture-of-experts CPU inference with GPU acceleration in llama.cpp. Hugging Face. https://huggingface.co/blog/Doctor-Shotgun/llamacpp-moe-offload-guide

GenerelSchwerz. (2026). Benchmark comparison showcase [llama.cpp, RTX 5070 Ti]. GitHub. https://github.com/GenerelSchwerz/llama.cpp/wiki/Benchmark-Comparison-Showcase

Hardware Corner. (2026). Apple M5 Max for local LLMs: First benchmarks vs RTX Pro 6000 and RTX 5090. https://www.hardware-corner.net/m5-max-local-llm-benchmarks-20261233/

Hardware Corner. (2026). RTX 5060 Ti 16GB local LLM benchmarks. https://www.hardware-corner.net/gpu-llm-benchmarks/rtx-5060-ti-16gb/

HP. (n.d.). HP Z2 Mini G1a. https://valuehub.hp.com/en/products/hp-z2-mini-g1a

inferris. (2026). inferris [Qwen3.8-27B, RTX 5090]. PyPI. https://pypi.org/project/inferris/

InsiderLLM. (2026). RTX 5060 Ti review for local AI. https://insiderllm.com/guides/rtx-5060-ti-local-ai-benchmarks/

ivanopcode. (2026). Override the macOS Metal VRAM cap [technical note]. GitHub. https://github.com/ivanopcode/devnote-override-macos-metal-vram-cap

Lobstein, V. (2026, 23 February). 12,000 Ollama instances exposed: When "local-first" meets the real world. LeakIX. https://blog.leakix.net/2026/02/ollama-exposed/

MacRumors. (2026). 2025 vs. 2026 Mac Studio buyer's guide. https://www.macrumors.com/guide/2025-vs-2026-mac-studio/

MindStudio. (2026). M6 Mac mini local AI performance. https://www.mindstudio.ai/blog/m6-mac-mini-local-ai-performance

Ollama. (n.d.). FAQ. https://docs.ollama.com/faq

OpenClaw. (2026). Dual RTX 5060 Ti vs used RTX 3090 for local LLMs. https://openclawdc.com/blog/dual-rtx-5060-ti-vs-rtx-3090-local-llm/

OrcaRouter. (2026). Qwen3.8-27B GGUF: Which quant to download for your GPU. https://www.orcarouter.ai/blog/qwen-3-8-27b-gguf

Spawning Point. (2026). NVIDIA DIGITS DGX Spark review 2026. https://spawningpoint.com/article/nvidia-digits-dgx-spark-review-2026

stared. (2026). Benching local LLMs on Apple Silicon [M5 Max 128 GB]. GitHub. https://github.com/stared/benching-local-llms-on-apple-silicon

StarryHope. (2026). Mac mini M6 vs M5 Pro for local AI. https://www.starryhope.com/minipcs/mac-mini-m6-vs-m5-pro-local-ai/

StarryHope. (2026). The DGX Spark and the reality of local AI hardware in 2026. https://www.starryhope.com/ai/dgx-spark-local-ai-hardware-landscape/

vLLM. (n.d.). Security. https://docs.vllm.ai/en/stable/usage/security/

xavki. (2026, 15 October). vLLM - 06. Le standard API OpenAI pour l'inférence [Video]. YouTube. https://www.youtube.com/watch?v=96FYveXB4qY