Ollama, llama.cpp, vLLM or LM Studio: choosing a local AI runtime

A local AI pilot needs software to load models and serve users.

Ollama, llama.cpp, vLLM and LM Studio are credible candidates, but they are not identical products. Their layers overlap. LM Studio provides a desktop application and local server while using inference engines beneath it. llama.cpp is an inference project with detailed backend controls. Ollama packages local model use behind chat and API entry points. vLLM focuses strongly on serving workloads.

Keep four decisions separate:

• interface: browser, CLI, IDE, API or messaging;
• execution location: desktop, internal server or external infrastructure;
• harness: prompts, tools, retrieval, permissions and workflow controls;
• runtime: the software that executes or serves the model.

First establish who will operate, secure, update and support the proposed system. The separate guide Hosted access versus operating weights: who owns what? owns that delivery-responsibility decision. This guide compares runtime candidates within the proposed operating boundary; it does not select a delivery model for you.

“Local” also needs a precise definition. It might mean execution on one laptop, an API listening on an internal workstation or a shared service on controlled infrastructure. It does not prove whole-workflow privacy. Downloads, model discovery, update checks, connectors and calling applications may cross a network boundary even when inference is local.

Compare the four candidates

Candidate	Operator fit	Control and deployment	Good candidate for	What to verify
Ollama	Developers wanting a managed local route	Local chat and API workflow; cloud features documented separately	Small applications and straightforward API pilots	Model licence and tag, cloud settings, logs, context, concurrency and memory
llama.cpp	Implementers wanting detailed engine control	C/C++ inference, quantisation, CPU/GPU backends and hybrid execution	Device-specific evaluation and GGUF-oriented deployment	Model support, format, quantisation, backend, build and current interfaces
vLLM	Platform teams evaluating shared inference	API serving, batching, caching and distributed execution	Concurrent or shared service evaluation	Model and hardware support, memory behaviour, API details and operational burden
LM Studio	Desktop users wanting graphical model management	Desktop chat, local server, CLI and SDK; uses llama.cpp runtimes and MLX on Apple Silicon	Human-reviewed desktop pilots and document exploration	Platform support, downloaded runtime, offline boundary and added connectors

The product characteristics are drawn from the cited documentation; the operator-fit and candidate-use columns are evaluation hypotheses, not measured superiority.

Ollama: a managed route into local use

Ollama documents model-tag downloads, local chat and API entry points.[1] Consider it for a small local integration.

Its FAQ states that the service listens on loopback address 127.0.0.1:11434 by default. It also documents setting OLLAMA_NO_CLOUD=1, restarting and checking logs to disable cloud features.[2] Those controls do not prove that an entire workflow is private. The model-download process, calling application, logs and connected tools still require assessment.

Memory use depends on the exact model, quantisation, context and concurrency. Measure the intended configuration rather than treating a machine label as sufficient evidence.

llama.cpp: detailed inference control

llama.cpp is a C/C++ inference project supporting quantisation, multiple CPU and GPU backends, and hybrid execution.[3] Its current documentation includes CLI and server routes and centres on GGUF-oriented tooling.

It suits implementers who need detailed control over builds, backends, quantisation and device use. That flexibility creates additional verification work. Do not assume that every model architecture, file format or hardware combination is supported equally.

Check current official instructions for interactive use, serving or engine integration.

vLLM: a candidate for concurrent serving

vLLM documents features including PagedAttention, continuous batching, prefix caching, distributed parallelism and an OpenAI-compatible API server.[4] These features justify evaluating it for a shared service with concurrent requests.

They do not establish that vLLM is suitable for every laptop, or that it is universally faster or cheaper. Results depend on the model, hardware, precision or quantisation, context and request pattern.

API compatibility is not feature parity. Test the behaviours the application actually uses, such as structured schemas, streaming, tool calls, cancellation and error handling.

LM Studio: desktop-led operation

LM Studio combines desktop chat and model management with a local server, CLI and SDK tools. Its documentation says it uses llama.cpp runtimes and MLX on Apple Silicon.[5] It therefore overlaps with the other candidates without being the same kind of component.

LM Studio’s offline documentation says core local chat, document chat and server functions can work offline once the required models and runtimes have been downloaded.[6] Model discovery, model and runtime downloads, and runtime-update checks use the network; on macOS and Windows, the in-app update check does too. MCP tools, connectors or other network clients can also change the boundary.

Verify platform support and runtime before a desktop pilot.

Hypothetical evaluation one: private desktop pilot

Consider an analyst summarising public policy documents on a dedicated desktop. Every summary is reviewed, and no output is published automatically.

Possible candidates include LM Studio for graphical document work, Ollama for a local API prototype and llama.cpp where detailed engine control matters. vLLM may add little unless concurrent or shared serving is required.

Predefine acceptance:

• the selected model’s licence permits the intended use;
• core inference works after required downloads with the network disconnected;
• documents remain within the defined test boundary;
• reviewers accept a predefined proportion of outputs under a written rubric;
• source references can be checked; and
• resource use stays within safe limits.

Use the same exact model artefact where compatible. Record its hash, quantisation, context, prompt and sampling settings. If one runtime requires a different format, conversion, quantisation or backend, record that as a confound. A result cannot fairly be attributed to the runtime alone when the model artefacts or execution conditions differ.

Stop if licensing is unsuitable, offline operation cannot be verified, the workstation becomes unstable or output acceptance misses the predefined threshold. This is a proposed evaluation, not an actual test result.

Hypothetical evaluation two: shared concurrent service

Now consider an internal service used by several approved applications. Requests arrive concurrently and must return structured output.

vLLM is a natural candidate because its documented features address serving and concurrency. Ollama and llama.cpp server routes can also be evaluated if they meet the workload. LM Studio’s local server may support development, but its suitability for the shared operating environment must be tested rather than assumed.

Predefine:

• required request and response schemas;
• target context length and concurrency;
• maximum acceptable memory consumption;
• p50 and p95 latency thresholds;
• accepted-output rate under one review rubric;
• tolerated errors and timeouts;
• restart and recovery expectations; and
• accountable maintenance ownership.

Keep model artefact, prompt set, output limits and workload sequence constant where compatibility permits. Warm-up policy, caching, batching and concurrency must be recorded because they can materially alter results. If a candidate cannot use the same model or feature, report the incompatibility instead of substituting silently.

Test only required API behaviours, including tool calls, schemas or streaming where used. Stop if the service exceeds resource limits, misses error thresholds, cannot provide required behaviours or has no maintenance owner. No production deployment follows automatically from an evaluation.

Evidence to record

For each candidate and proposed run, capture:

• model name, licence, artefact hash and quantisation;
• runtime version or build;
• operating system, CPU/GPU backend and relevant drivers;
• prompt, context length, output limit and concurrency;
• batching, caching and warm-up conditions;
• representative or peak memory use;
• p50 and p95 latency under the defined workload;
• outputs accepted under the common rubric;
• errors, timeouts and invalid structured responses;
• network state and boundary checks; and
• installation, updates, restart and recovery effort.

These checks and records help distinguish a convenient demonstration from evidence needed to assess service readiness; they do not establish that a service is supported or production-ready. They also make clear when a comparison is affected by model format, quantisation, backend or feature differences.

Memory sizing and total cost of ownership need separate treatment. The companion memory-budget and total-cost worksheets cover that analysis rather than compressing it into a runtime table.

Make the decision

Shortlist LM Studio when a human-centred desktop experience and graphical model management matter. Shortlist Ollama when a managed local chat and API route may suit a developer-led pilot. Investigate llama.cpp when backend, device and quantisation control matter. Evaluate vLLM when shared serving and concurrency may justify platform-oriented operation. These are starting hypotheses; the recorded workload, compatibility and operating tests determine the choice.

Then verify the exact model, hardware, configuration and workflow. Runtime selection alone is not proof of privacy, performance, compatibility or production readiness.

Record the candidate to test—or the decision to hold—along with the workload limits, exact configuration, unresolved compatibility questions, acceptance and stop criteria, maintenance owner and next review trigger. After evaluation, record whether to retain the candidate, test another or stop. Return that evidence to Running AI locally; a runtime shortlist is not deployment approval.

If implementation help is needed, define the requested assessment and operating responsibilities separately. A runtime comparison does not establish who will install, secure, maintain or support a deployed system.

Sources and next reading

1. Ollama, “Quickstart”: https://docs.ollama.com/quickstart
2. Ollama, “FAQ”: https://docs.ollama.com/faq
3. llama.cpp, “README”: https://raw.githubusercontent.com/ggml-org/llama.cpp/master/README.md ; “Models”: https://github.com/ggml-org/llama.cpp/blob/master/docs/models.md ; “Server”: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
4. vLLM, “Overview”: https://docs.vllm.ai/en/stable/
5. LM Studio, “App”: https://lmstudio.ai/docs/app
6. LM Studio, “Offline operation”: https://lmstudio.ai/docs/app/offline

Next checks:

• Running AI locally — return to the parent workload decision.
• Local AI memory-budget worksheet — measure the exact candidate's memory fit.
• Local AI total-cost worksheet — compare ownership and hosted costs under the same workload.