SerpApi.Org
I have been using Ollama to run local LLMs for the longest time now, and I don’t have a lot to complain about. It gets the job done, and I don’t have to tweak a lot of things. Recently, Docker announced its own model runner, which offers many of the same features you get with Ollama. It can pull, run, manage, and serve local AI models through Docker Desktop or Docker Engine. It supports GGUF models through llama.cpp, Safetensors models through vLLM, hardware acceleration on supported systems, and models from Docker Hub, Hugging Face, or other OCI registries. I have been using it instead of Ollama for almost a week now, but I can’t say I am sticking to it.
Docker Model Runner fits right into Docker
And it's super easy to set up
Docker Model Runner works differently from running a model inside a regular container. Docker handles the model as its own OCI artifact, while a dedicated inference engine loads and runs it locally. The model remains cached on your storage after the first download, loads into memory when it receives a request, and unloads when it’s no longer being used. This keeps the process familiar if you already use Docker, since the same CLI now handles containers, images, and local models.
For most people, the default llama.cpp backend is the one that matters. It runs quantized GGUF models across Apple Silicon, Windows, and Linux, with automatic Metal acceleration on supported Macs. Docker also offers vLLM for higher-throughput inference with Safetensors models, although that backend requires a supported NVIDIA GPU and currently has much narrower platform support. You can access either backend through OpenAI-compatible and Ollama-compatible APIs, which means many applications already designed for Ollama can connect to Docker Model Runner with little more than a base URL change.
Setting it up through Docker Desktop is ridiculously straightforward. You need Docker Desktop 4.40 or newer on macOS or version 4.41 or newer on Windows. Open Docker Desktop, head into Settings -> AI, and enable Docker Model Runner. You can also enable host-side TCP access here if you want to connect to Open WebUI, a coding tool, or another local application. Once enabled, the new Models section lets you discover, download, run, and chat with models without opening the terminal. I still prefer the command line because the workflow is extremely close to Ollama.
Using Docker Model Runner for everyday tasks
The experience is similar to Ollama
Once the model is downloaded, using Docker Model Runner isn’t very different from using Ollama. Docker Model Runner opens an interactive chat inside the terminal, where you can send prompts and continue the conversation without restarting the model after every response. Docker Desktop also provides a basic chat interface under the Models tab, which is useful when I want to test a model without setting up another application.
The built-in interfaces are enough for quick prompts, but I wouldn’t use them as my primary way to interact with local LLMs. Docker Model Runner becomes far more useful when you connect it to something like Open WebUI. Since it provides an Ollama-compatible API, Open WebUI can treat it almost exactly like an Ollama server. You only need to point OLLAMA_BASE_URL to Docker Model Runner.
This gives you a proper ChatGPT-like interface with conversation history, Markdown rendering, model switching, system prompts, and configurable generation settings. The models still run through Docker Model Runner, while Open WebUI handles the interface.
The same approach works with AI coding tools such as Cline, Continue, Aider, Cursor, and OpenCode.
You can use any placeholder as the API key because local requests don’t require one. After that, you select the model using the same name shown by docker model list, such as ai/qwen2.5-coder. I also like that Docker lets me configure models individually. If a coding model starts with an unnecessarily small context window, I can easily increase it.
Docker Model Runner doesn’t make Ollama obsolete
Ollama is as good
After using both, I don’t think Docker Model Runner produces fundamentally better results than Ollama. When both tools run the same GGUF model through llama.cpp with similar settings, the model, quantization level, context size, and hardware have a much larger effect than the application serving it. Docker does let you use vLLM on supported NVIDIA systems, but that won’t matter to most people running quantized models on a regular laptop.
Docker Model Runner makes the most sense if Docker Desktop is already part of your workflow. It lets you manage models and containers through the same interface, store models as OCI artifacts, use them inside Compose projects, and inspect requests through Docker Desktop. I also prefer its graphical model manager over remembering every CLI command. However, installing and keeping Docker Desktop running only to use local LLMs adds unnecessary weight.
Ollama also has the advantage of a more established ecosystem. Many local AI applications provide a dedicated Ollama option, and most setup guides assume that its API is available on port 11434. Docker Model Runner offers an Ollama-compatible API, so many of these applications still work, but some require manual changes to the base URL or model name. Docker’s support for different inference engines and registries is more flexible, though it also introduces additional settings and platform-specific limitations.
I am better off with Ollama
If you already use Ollama and everything works, there aren’t many practical reasons to switch. You’ll probably run the same models at broadly similar speeds and receive the same outputs. Docker Model Runner is more compelling for someone who already uses Docker Desktop or wants to integrate local models into containerized applications.
