Run a local model API and connect it to a simple app workflow.
A good Ollama build starts with one runnable app path and an acceptance test.
Learners place Ollama in the local-serving landscape.
Why this matters: This prevents treating a laptop demo as a production serving plan.
Problem anchor: a teacher wants a private demo that drafts lesson explanations without sending prompts to a hosted API. Ollama is useful because a learner can pull a model, run it locally, and call a simple HTTP API on localhost.
The KB verdict is clear: Ollama is best for local development, demos, and privacy-sensitive prototypes; it is not the default for high-concurrency production serving with autoscaling, tenant policy, and detailed routing.
The worked artifact is a campus helpdesk assistant that must answer from a local Ollama endpoint during a privacy review. It loads the selected model, streams a short answer, records the model tag and latency, and returns a clear setup message if the endpoint is missing. The negative test disconnects Ollama and confirms the service fails closed instead of routing the student question to an external provider.
Ollama manages local model availability and exposes chat/generate endpoints.
Learners understand the operational pieces they will touch.
Why this matters: This catches failures where the app assumes a model is running but it is not.
Ollama commands such as pull, run, show, ps, and create manage local model artifacts and active sessions. The local server exposes HTTP endpoints for chat, generate, embeddings, and model management. Streaming JSON is convenient, but clients must parse it deliberately.
| Option | Part | Role | Failure to watch | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Model cache | Model cache | keeps pulled artifacts local | missing model or wrong tag | Before first run. | Medium | Medium |
| Local server | Local server | serves chat/generate over HTTP | port unavailable or service stopped | App integration. | Medium | Medium |
| Streaming API | Streaming API | returns incremental JSON chunks | client treats chunks as one JSON object | Interactive UI. | Medium | Medium |
Custom local models should record base model, prompt template, and context settings.
Learners make local behavior reproducible.
Why this matters: This prevents demos that depend on hidden notebook state.
A Modelfile can define the base model, system prompt, prompt template, parameters such as context length, adapters, and license metadata. It gives the local demo a stable contract instead of relying on one-off CLI flags.
FROM llama3.1
PARAMETER num_ctx 8192
SYSTEM You are a concise tutor for agentic AI concepts. Explain with one example and one check question.
Review: num_ctx increases memory use, and the base model license still controls allowed use.
A smoke test proves the endpoint works before UI polish.
Learners implement the integration path.
Why this matters: This makes local serving testable.
import requests, time payload = { "model": "llama3.1", "stream": False, "messages": [{"role": "user", "content": "Explain KV cache in one paragraph."}], } start = time.time() resp = requests.post("http://localhost:11434/api/chat", json=payload, timeout=60) resp.raise_for_status() data = resp.json() latency = time.time() - start assert "message" in data and "content" in data["message"] print({"latency_s": round(latency, 2), "preview": data["message"]["content"][:80]})
The endpoint can return newline-delimited streaming JSON chunks, so treating the whole response as one JSON object can fail or lose partial-output handling.
Checklist: model tag exists locally; service URL is configurable; streaming and non-streaming modes are parsed correctly; latency is logged; prompt data stays local; and the app fails clearly when the model is missing.
Ollama should hand off to gateways or serving runtimes when concurrency, policy, and reliability matter.
Learners leave with realistic deployment boundaries.
Why this matters: This prevents local demos from overpromising scale and governance.
Local performance depends on model size, quantization, VRAM, and offload behavior. Ollama does not automatically provide multi-user auth, budgets, audit logs, autoscaling, or production routing. For many users, move behind a gateway, hosted API, vLLM, or another serving stack.
Retrieval prompt: rebuild the Ollama project by naming target behavior, pulled model, Modelfile, API call, streaming choice, latency log, missing-model error, and production graduation criteria.
Build a local app that calls Ollama for one lesson explanation. Include pull/run instructions, Modelfile, non-streaming smoke test, streaming UI note, latency logging, and a clear migration note for production. Next rung: add provider abstraction.
Ollama acts as a local model server. Which of the following best describes what Ollama is responsible for — and what it is NOT responsible for?
Ollama owns the local lifecycle — pulling, caching, and serving the model — but production concerns like auth, scaling, and rate-limiting are outside its scope.
You run ollama pull llama3 and then ollama run llama3. Later you restart your machine. What happens to the model, and why?
Pulled models are written to a local disk cache (e.g., ~/.ollama/models), so they persist across reboots without re-downloading.
In a Modelfile, what are the two most important things you should set deliberately, and what risk does ignoring memory limits in a Modelfile introduce?
A Modelfile's SYSTEM prompt locks in stable behavior, while an oversized context window or model can exhaust RAM and crash the server.
Your Python app calls Ollama's /api/generate endpoint with "stream": true but you read only the first line of the response and close the connection. What will happen?
Streaming responses arrive as a sequence of JSON chunks; reading only the first line and closing means you miss most or all of the generated text.
A teammate wants to move your Ollama demo to production to serve real users. Which comparison best explains why Ollama alone is NOT the right choice for that step?
Ollama is optimized for local development; production gaps — multi-user auth, load balancing, uptime SLAs — are addressed by vLLM (self-hosted at scale) or managed hosted APIs.