Overview

Relevant source files

• .dockerignore
• .github/workflows/latest.yaml
• .gitignore
• README.md
• api/client.go
• api/client_test.go
• api/types.go
• cmd/cmd.go
• docs/api.md
• docs/api/errors.mdx
• docs/api/streaming.mdx
• docs/api/usage.mdx
• docs/capabilities/structured-outputs.mdx
• docs/capabilities/vision.mdx
• docs/cli.mdx
• docs/context-length.mdx
• docs/development.md
• docs/index.mdx
• docs/quickstart.mdx
• go.mod
• go.sum
• scripts/build_docker.sh
• scripts/push_docker.sh
• scripts/tag_latest.sh
• server/images.go
• server/images_test.go
• server/routes.go

Purpose and Scope

This page provides a high-level overview of Ollama's architecture, core components, and how they interact. It introduces the system from user-facing interfaces down to model execution, storage, and hardware abstraction.

For detailed information about specific subsystems:

• Key Concepts: Definitions of models, manifests, blobs, and runners, see Key Concepts.
• System Requirements: Platform requirements and GPU recommendations, see System Requirements.
• Command Line Interface: Comprehensive guide to CLI commands and the TUI, see Command Line Interface.
• Architecture: Deep dive into the server, scheduler, and storage layers, see Architecture.
• API Reference: Endpoint specifications and request/response schemas, see API Reference.
• Model Management: Modelfiles, registry layers, and conversion, see Model Management.
• Inference Engine: Deep dive into execution backends and KV cache, see Inference Engine.
• GPU and Hardware Support: Hardware discovery and optimization, see GPU and Hardware Support.
• Development Guide: Building from source and testing, see Development Guide.

What is Ollama?

Ollama is a local LLM runtime designed to run large language models on consumer hardware with minimal setup. It serves as both a model manager and an inference server.

Core Capabilities:

• Model Management: Pull models from the Ollama library, create custom models using Modelfile syntax, and manage local storage README.md82-86 cmd/cmd.go222-232
• Inference Execution: Support for text generation (/api/generate), chat (/api/chat), and embeddings (/api/embed) server/routes.go205-235 api/types.go62-194 docs/api.md7-16
• Hardware Acceleration: Automatic detection and utilization of NVIDIA (CUDA), AMD (ROCm), Apple Silicon (Metal), and Vulkan backends docs/development.md19-56
• API Compatibility: Native REST API alongside compatibility layers for OpenAI and Anthropic message formats docs/api.md1-19 server/routes.go55-57 README.md58-68
• Cloud Integration: Support for remote inference via cloud models (using the :cloud suffix) and web search integration server/routes.go61-65 server/routes.go216-222

Architecture Pattern: Ollama follows a client-server model. The ollama serve command starts a background process (the server) that manages model lifecycles and hardware resources. The ollama CLI and other applications act as clients communicating over HTTP api/client.go1-14 server/routes.go100-105

Sources: README.md7-40 server/routes.go1-67 docs/api.md1-20 docs/development.md1-130 api/client.go1-14

Core System Components

System Architecture Diagram

The following diagram bridges the high-level functional areas with specific code entities.

Core Components by Layer:

Layer	Component	File Location	Role
CLI	Command Handlers	cmd/cmd.go	Processes user commands like run, create, pull.
API	Server	server/routes.go97	Gin-based HTTP server handling /api/* and compatibility routes.
Orchestration	Scheduler	server/routes.go99	Manages the lifecycle of runners, including loading/unloading and VRAM allocation.
Model Logic	Model Struct	server/images.go63	Aggregates manifests, configuration, and layer paths (GGUF, adapters) server/images.go63-82
Execution	LlamaServer	server/routes.go202	Interface for interacting with the underlying inference backends.
Discovery	GpuInfo	server/routes.go36	Logic for detecting GPU memory and compute capabilities across platforms.

Sources: server/routes.go1-105 cmd/cmd.go61-136 server/images.go63-82 api/client.go36-41

High-Level Architecture

Request Flow

When a user sends a prompt, the request traverses multiple layers to reach the hardware-accelerated runner.

Key Interactions:

• Model Resolution: The server resolves a name to a manifest and associated blobs via GetModel server/routes.go207
• Runner Allocation: scheduleRunner ensures a model is loaded and ready for inference, handling potential evictions of idle models server/routes.go202-235
• Streaming: Responses are typically streamed as newline-delimited JSON (ndjson) to provide real-time feedback docs/api.md31-33 api/client.go170-210

Sources: server/routes.go147-235 api/client.go170-210 docs/api.md31-41

Model Storage and Naming

Ollama uses a content-addressable storage (CAS) system. Models are defined by a Manifest which lists the digests of individual Blobs (layers) server/images.go63-82

• Naming Convention: model:tag (e.g., gemma4:latest). Tags default to latest docs/api.md23-25
• Layer Types: A model consists of multiple layers including the base weights (GGUF or Safetensors), prompt templates, system messages, and configuration server/images.go63-82
• Cloud Models: Names suffixed with :cloud bypass local storage and are routed through a cloud proxy server/routes.go59-65

Sources: server/images.go63-82 docs/api.md23-25 server/routes.go59-65

API Surface

The Ollama server exposes a REST API on port 11434 by default.

• Native API: /api/generate, /api/chat, /api/embed, /api/pull, /api/push, /api/create, /api/tags docs/api.md7-19
• Compatibility Layers:
  • OpenAI: Supports /v1/chat/completions and /v1/embeddings server/routes.go56
  • Anthropic: Supports the Messages API for seamless integration with tools like Claude Code README.md58-68
• Streaming: Most generation endpoints support streaming by default, which can be disabled via the stream: false parameter api/types.go84-85 docs/api.md31-33

Sources: docs/api.md7-19 api/types.go62-180 server/routes.go55-57

Scheduler and Memory Management

The Scheduler is the brain of the server's resource management server/routes.go99

• Runner Lifecycle: It tracks active runnerRef instances, managing their loading and keep_alive durations server/routes.go202-235
• Concurrency: It handles multiple concurrent requests and can manage multiple loaded models if VRAM allows api/types.go111-118
• Eviction: When memory is full, the scheduler unloads least recently used models that have no active requests. Orphaned blobs are eligible for pruning after a grace period server/images.go39-40

For details on scheduling logic, see Request Scheduling and Runner Management.

Sources: server/routes.go97-103 server/routes.go202-235 api/types.go111-118 server/images.go39-40

Summary of Key Features

Feature	Description
Multi-Platform	Native support for macOS, Windows, and Linux README.md11-35
Tool Calling	Models can interact with external tools via the /api/chat endpoint api/types.go150-152 server/images.go194-213
Thinking Support	Integration for reasoning models (e.g., DeepSeek R1) with specialized thinking output api/types.go104-110 server/images.go215-223
Multimodal	Support for vision models that can process images alongside text api/types.go96-98 server/images.go169-171
Image Generation	Experimental support for diffusion-based image generation models docs/api.md19 server/images.go50

Sources: README.md11-35 api/types.go59-202 server/images.go50-223