How do you serve open weight models on your own hardware?

Serving an open weight model means running a published model file on machines you control, behind an API your applications call. Four parts do the work: the weights, a server that loads them, the memory to hold them and an API in front.

Open weights are not the same as open source. A model’s weights are open when anyone can download them. NTIA’s report on open model weights uses exactly that test: weights that are open to the public to download. The training data and code may stay closed, and the license may still limit use. Serving is also not training. Nothing here changes the model. It only runs it.

Open models are commonly published in two formats. Safetensors files hold the weights for training frameworks. GGUF files are made for inference. The GGUF specification says the format is “designed to be unambiguous by containing all the information needed to load a model.”

One GGUF file carries the weights, the architecture, the tokenizer and the chat template, as metadata a server can read before it loads anything. Its name says what is inside by convention: the model, its size, its version and its encoding. In Llama-3.1-8B-Instruct-Q4_K_M.gguf, the last part is the quantization.

Where the file comes from matters. Download it from the publisher’s own repository, check its checksum, and read its license before you put it into production.

A model’s size is its parameter count times the bits it stores per weight. llama.cpp’s quantize guide puts the trade plainly: “Quantization reduces the precision of model weights (e.g., from 32-bit floats to 4-bit integers), which shrinks the model's size and can speed up inference.”

Its table for Llama 3.1 shows the scale. The 8 billion parameter model is 32.1 GB at full precision and 4.9 GB at Q4_K_M. The 70 billion parameter model goes from 280.9 GB to 43.1 GB.

The cost is some accuracy. The guide measures it as perplexity: how far the smaller model drifts from the original. Neither tells you how it answers your questions. Run the questions your users ask against two or three quantizations and compare the answers before you choose.

A model has to be held in memory while it answers, and the weights are only part of what it needs. Each conversation also holds a cache of what the model has read so far, the KV cache. It grows with the length of the context, and every conversation served at the same time needs its own.

So a sizing has three parts: the weights, the cache for the longest context you allow, and the working buffers on top. On a GPU, all three have to fit in the card’s own memory. A model that does not fit on a GPU can run on the CPU from the host’s memory instead.

Write the sum down before you buy hardware. It decides which model you can run, at which quantization, with how long a context, for how many people at once.

Servers such as Ollama answer a subset of the OpenAI API, and some answer the Anthropic messages API too. An application written for a hosted model then needs one change: the base URL, and a key for your server. Answers stream back token by token, as server-sent events, just as they do from a hosted API.

That is what makes the model a setting rather than a rewrite. The same code can call a hosted model for one job and your own server for records that must stay in the building.

Compatible rarely means complete. Check the parameters your code relies on: tool calls, JSON schema output, log probabilities, images. Whether tool calls work also depends on the model and its chat template, not only on the server.

A server can answer one request at a time per model, or batch several together. The authors of PagedAttention put the case for batching directly: “High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time.”

Batching runs many conversations through each step of the model, so the hardware does more work per second. The price is memory, since every conversation in the batch holds its own KV cache. PagedAttention exists to manage those caches without waste.

Answering one request at a time is simpler and needs less memory. Requests to the same model wait in line, so it suits internal tools with a handful of users at once. A service with many people asking at the same moment needs batching, or more copies of the model.

Five questions settle most of the design:

1. Which model, at which quantization, measured on your own questions?
2. What does it need in memory: weights, the cache for your longest context, and the buffers?
3. How many people will ask at the same moment, and how long may they wait?
4. Which API parameters does your code use, and does the server read all of them?
5. Who may call it: which keys, and what is logged about each request?

Then run it for a week with the people who will use it. Watch the queue and the memory, and how often the answers needed a person to fix them.

Curiosity Spark serves open models in the GGUF format behind OpenAI and Anthropic compatible APIs, on your own machines. Your code changes its address and keeps the rest. It places models on NVIDIA GPUs with CUDA 12 when it finds them, and on the CPU otherwise.

Before it loads a model, Spark reads the GGUF header to estimate what the weights, the KV cache and the buffers need. It then puts the model where it fits. It unloads models nobody is using to make room. A model answers one request at a time, in the order they arrive, and different models answer side by side.

Curiosity Studio takes any OpenAI compatible endpoint as its model, and Spark is one. The tools your teams build in Studio can then run on weights that stay on your hardware. The models page lists the providers built into Studio and which models run on premises. For a network with no connection out at all, read what air gapped AI needs.