# How do you turn a local model into a server your apps can use?

Date: 2026-05-27 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/local-ai-server-vllm-llama-cpp/

**Answer:** Start a server program such as Ollama, llama-server or vLLM, and it will answer on a local address using the same request format as OpenAI's API, so most apps only need a new base URL. Ollama and llama.cpp suit one person or a small group on any computer, and vLLM suits many simultaneous users on a Linux machine with a graphics card.

A chat window is only the start. Once a model runs on your own hardware, you probably want other programs to use it: a coding agent, a script that summarizes files, an internal tool for your team. To do that, you run the model as a **server**, a program that waits for requests on a network address and sends answers back.

Nearly all of these servers speak the same language. They copy the request format of OpenAI's API, so software written for a hosted model can usually point at your machine by changing one setting. The [comparison of Ollama and LM Studio](/en/learn/ollama-lm-studio-easier-running-models-locally/) covers the desktop apps. This article covers the serving side, and how to pick an engine.

## Three engines

| | Ollama | llama-server (llama.cpp) | vLLM |
|---|---|---|---|
| Best for | One person, quick setup | Control on any hardware | Many users at once |
| Runs on | Mac, Windows, Linux | Mac, Windows, Linux, CPU or GPU | Linux; Mac through a community plugin; Windows only through WSL |
| Main model format | Its own library, GGUF | GGUF | Hugging Face models; FP8, AWQ and others |
| Default address | `127.0.0.1:11434` | `127.0.0.1`, port set by you | `localhost:8000` |
| Parallel requests | Default 1, set by `OLLAMA_NUM_PARALLEL` | Slots, set by `--parallel` | Continuous batching |

## A worked example: three ways to start

With Ollama, the server is already running in the background once it is installed, and the API answers on port 11434.

With llama.cpp, you name a model and a context size. Here 32,768 tokens are shared by two slots:

```bash
llama-server -hf <user>/<model>:<tag> -c 32768 --parallel 2 --port 8080
```

With vLLM, on a Linux machine with a suitable graphics card:

```bash
uv pip install vllm --torch-backend=auto
vllm serve Qwen/Qwen2.5-1.5B-Instruct --host 127.0.0.1 --max-model-len 8192 --gpu-memory-utilization 0.8
```

Any of them can then be called the same way, here from Python with the standard OpenAI client:

```python
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="local")
reply = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Summarize this note in two lines: ..."}],
)
print(reply.choices[0].message.content)
```

Change the port to match your engine. On Ollama and vLLM, `model` must be the name the server lists at `/v1/models`.

## Settings that bite

**Memory.** vLLM takes [92% of the graphics memory by default](https://docs.vllm.ai/en/stable/configuration/engine_args.html) so lower `--gpu-memory-utilization` if anything else shares the card. Its [memory guide](https://docs.vllm.ai/en/stable/configuration/conserving_memory/) also recommends limiting `--max-model-len` when you run short.

**Context multiplied by users.** In Ollama, required memory scales with the number of parallel requests times the context length. Four parallel requests at 32,000 tokens cost about the same as one at 128,000. A model that fit fine for one person can fail for four, and the [memory calculator](/en/learn/vram-calculator-which-model-fits/) shows how to plan for this.

**Queues.** When too many requests arrive, Ollama answers with a 503 error, and `OLLAMA_MAX_QUEUE` sets how long the line can get. Your app should retry rather than fail.

## What batching buys you

A language model can work on several requests at once, reading the same weights for all of them. **Continuous batching** adds new requests to the batch as old ones finish, so the machine stays busy. llama-server and vLLM both offer it.

LMSYS measured what this means in practice on a DGX Spark, using another server called SGLang. On Llama 3.1 8B, one request at a time produced 20.5 tokens per second. With 32 requests at once, [total output reached 368 tokens per second](https://lmsys.org/blog/2025-10-13-nvidia-dgx-spark/). That is about 18 times more work from the same chip, though each user then sees about 11.5 tokens per second, which is slower than one user alone. Batching is a benefit for a shared service, not for a single chat.

## Keeping it private

Local means private only as long as the server stays local. Ollama and llama-server listen only to your own machine by default, and vLLM's guide describes `localhost` as its default. Name the address explicitly, as in the command above, then check it from another device.

- Ollama: setting `OLLAMA_HOST` to `0.0.0.0` exposes it to your network.
- llama-server: `--host` changes the address, and `--api-key` turns on a key. There is no key by default.
- vLLM: `--api-key` or the `VLLM_API_KEY` variable does the same.

If others need access, use a VPN or a reverse proxy with its own login rather than opening the port on your router. Chat logs stay on your disk too, as the [cloud or local guide](/en/learn/cloud-local-where-ai-run-2026/) explains.

> **Try it:** Start any of the three engines with a small model, then send it a request with `curl` and with the Python client above. Send five requests at once from separate terminals and note how the speed per request changes.

```prompt
Here is my setup: [engine, model, graphics memory, number of expected users, typical prompt length]. Suggest values for context length and parallel requests that fit my memory, and show the arithmetic. Then list what I should check to confirm the server is reachable only by the people I intend.
```

## Do you need a server at all if you work alone?

You probably already have one. Ollama and LM Studio both run a local API, so a coding agent or a script can use them without any extra work. There is no reason to install vLLM for a single user on a home computer: the advantage of batching appears only when several requests arrive at the same time.

Move to vLLM when you have a real queue: a team sharing a model, or a program sending many requests in parallel. Before then, the simplest setup that keeps your data on your machine is the right one. If you do grow, your apps will not need to change much, since all three accept the same request format.

## Sources

- LLaMA.cpp HTTP Server (README): https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
- Quickstart, vLLM documentation: https://docs.vllm.ai/en/stable/getting_started/quickstart.html
- Ollama FAQ: https://docs.ollama.com/faq

---

From the Guide to Artificial Intelligence, by Robert Scales, published by Holistic Hybrid Inc. Share and adapt it under CC BY-NC-SA 4.0 (https://creativecommons.org/licenses/by-nc-sa/4.0/): credit the guide, link to https://guidetoartificialintelligence.com/en/learn/local-ai-server-vllm-llama-cpp/, no commercial use without permission, and keep the same licence.