Understand accelerator tradeoffs for training, inference, and cost planning.
Beginner hardware literacy means asking what work must finish, how fast, with how much memory, and at what cost.
Beginner hardware literacy means asking what work must finish, how fast, with how much memory, and at what cost.
Why this matters: Separate training, batch inference, and interactive serving.
Problem anchor: a small team wants to run three AI jobs: nightly embedding of new documents, live chat over a 7B model, and occasional fine-tuning. The beginner mistake is to ask, "Which GPU should we buy?" The better question is, "What job, latency, batch size, memory, and utilization are we buying for?"
A CPU can be enough for small classical models, light batch jobs, and orchestration. A GPU helps when large tensor operations dominate and the model fits memory. A TPU can be compelling in ecosystems designed for it, especially large training or batch workloads, but it changes the software and platform assumptions. Managed APIs can be best when the team does not want to operate hardware at all.
Create one card per workload. Live chat: 7B instruct model, first token under two seconds, context 8k, 20 concurrent users, streaming required, GPU memory and KV cache are the limit. Nightly embeddings: 50,000 chunks, can finish in two hours, batching matters more than first-token latency, retryable on cheaper capacity. Fine-tuning: occasional, checkpointed, can use rented accelerators and stop when metrics plateau.
The card stops magical thinking. It shows that the same team may use CPU workers for orchestration, one GPU endpoint for chat, spot or scheduled GPU jobs for embeddings, and hosted or rented capacity for rare training.
Accelerators are fast when the model fits, data arrives on time, and precision choices preserve acceptable quality.
Accelerators are fast when the model fits, data arrives on time, and precision choices preserve acceptable quality.
Why this matters: Name the main memory consumers in inference and training.
For LLM inference, memory is consumed by model weights, runtime overhead, and KV cache for active tokens. Longer context and more concurrent requests can run out of memory even when the model weights fit. For training, activations, gradients, optimizer state, and batch size add pressure beyond the raw model file.
Precision changes the tradeoff. Lower precision or quantization can reduce memory and increase feasible throughput, but it must be tested for quality and supported kernels. Data movement matters too: a powerful accelerator can sit idle if preprocessing, disk, or network cannot feed it fast enough.
A chat service shows 35 percent GPU utilization and p95 latency above five seconds. The beginner guess is "buy a bigger GPU." The trace shows the real issue: requests queue behind long document-summary jobs, and the model endpoint accepts prompts with huge context windows. Separate batch traffic, cap context for interactive routes, and measure again before changing hardware.
This reinforces module 1: hardware choice follows workload isolation. A powerful chip cannot fix a traffic policy that mixes patient batch work with impatient chat work.
| Option | Workload | Likely limiter | First check | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Interactive LLM serving | Streaming answers from a loaded model | GPU memory, KV cache, queue time, decode speed | time to first token, p95 latency, memory headroom | Use for chat and agent endpoints. | Medium | Medium |
| Offline embedding | Batch vectors for many chunks | batch size, encoder throughput, upsert speed, queue retries | chunks per minute and failed batches | Use for refresh jobs and backfills. | Medium | Medium |
| Training or fine-tuning | Repeated forward/backward passes | accelerator memory, data loader, checkpointing, interconnect | steps per second and validation metric per dollar | Use for adaptation experiments. | Medium | Medium |
Each hardware path bundles a software stack and operating model, not just silicon.
Each hardware path bundles a software stack and operating model, not just silicon.
Why this matters: Compare hardware options without one-size-fits-all claims.
Decision this forces: Choose the right level of rigor for GPUs, TPUs, and AI Hardware: baseline, production design, or advanced optimization.
A GPU path usually fits broad PyTorch, CUDA, and serving-engine ecosystems. TPUs fit workloads and frameworks designed for them, often inside a cloud platform. CPUs are easiest to operate but can be too slow for large neural workloads. Managed model APIs shift capacity planning to a provider while reducing low-level control.
NVIDIA MIG is useful when supported hardware can be partitioned into isolated GPU instances for smaller workloads. It improves sharing and isolation, but it does not remove memory limits or make one device equivalent to many full GPUs. TPUs similarly require platform-aware design rather than simple drop-in replacement thinking.
| Option | Path | Good fit | Watch out | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| CPU | General-purpose compute | Orchestration, small ML, preprocessing, light batch jobs | Large neural inference may miss latency or throughput goals | Start here when the model is small or latency is loose. | Low | Low |
| GPU | Widely supported accelerator | LLM serving, embeddings, vision, fine-tuning, many PyTorch workloads | Memory, driver/runtime compatibility, cold starts, and idle spend | Use when tensor compute and ecosystem support matter. | High | Medium |
| TPU or managed accelerator | Cloud-specialized accelerator path | Large training or batch workloads designed for the stack | Framework support, data pipeline design, quota, and portability | Use when the platform and workload align. | High | High |
A practical estimate records model size, memory headroom, concurrency, batch behavior, and cost per useful task.
A practical estimate records model size, memory headroom, concurrency, batch behavior, and cost per useful task.
Why this matters: Write a simple hardware planning function.
model_gb = 14 # rough quantized weight size gpu_memory_gb = 24 runtime_overhead_gb = 3 kv_cache_per_user_gb = 0.6 available_for_cache = gpu_memory_gb - model_gb - runtime_overhead_gb max_active_users = int(available_for_cache // kv_cache_per_user_gb) print({ "available_for_cache_gb": round(available_for_cache, 1), "rough_max_active_users": max_active_users, }) assert max_active_users > 0, "model fits weights but leaves no room for serving"
No. Serving also needs runtime overhead and KV cache for active tokens, so exact weight fit can still fail under real requests.
When an AI assistant recommends "use an A100" or "switch to TPU," demand the missing assumptions. Good advice names workload, framework, model size, precision, context length, batch size, concurrency, latency target, data pipeline, autoscaling behavior, and budget.
The lesson sticks when the learner can explain hardware fit as workload, memory, latency, utilization, and operations.
The lesson sticks when the learner can explain hardware fit as workload, memory, latency, utilization, and operations.
Why this matters: Recall the hardware fit loop from memory.
Decision this forces: Choose the right level of rigor for GPUs, TPUs, and AI Hardware: baseline, production design, or advanced optimization.
Answer this without notes: "How do I decide whether an AI workload needs CPU, GPU, TPU, or a managed API?" A good answer starts with workload type, latency target, memory, batch behavior, data movement, software ecosystem, utilization, and cost per useful result.
Spaced recall: return to the first module. The question was never "Which chip is best?" It was "Which workload promise must be met, and what hardware path can the team operate honestly?"
Choose one feature, such as live lesson chat or nightly embedding refresh. Fill out: model, input size, output size, precision, latency target, concurrency, batch window, memory estimate, data source, retry plan, observability fields, and cost per useful result. Recommend CPU, GPU, TPU, or managed API and list the first load test you would run.
Next rung: connect this to cost and latency optimization, where the same hardware decision becomes part of a route, cache, and budget policy.
From memory, reconstruct the hardware fit decision: classify the workload, estimate memory and latency constraints, compare CPU/GPU/TPU/managed paths, run a capacity check, then choose the path the team can operate.
Create a one-page release-ready plan for hardware selection worksheet for GPU, TPU, memory, and utilization needs. Include: the user problem, realistic input, mechanism, design choice, runnable or reviewable check, metric (utilization and throughput per dollar), failure case (the selected accelerator is fast but memory-bound for the real workload), owner, and the next rung after this lesson.
You're launching a chatbot that must respond to each user within 2 seconds. Which workload type best describes this, and why does it matter for hardware selection?
Interactive serving is defined by strict per-request latency requirements; memory bandwidth (how fast the chip feeds data to compute) often limits token generation speed more than raw FLOPS.
A 70-billion-parameter model stored in 16-bit precision needs roughly 140 GB of GPU memory just for weights. A team wants to serve it on a single GPU with 80 GB of VRAM. Which technique is the most direct lever to make the weights fit?
Quantization reduces the number of bits used to store each weight, directly shrinking total memory footprint — halving precision roughly halves the memory required for weights.
A startup runs occasional, unpredictable spikes of AI inference — sometimes zero traffic, sometimes a burst of thousands of requests. Which hardware approach is most likely to give the best cost per useful result?
Managed APIs charge per token or per request, so idle time costs nothing — this beats owning accelerators when traffic is unpredictable and utilization would otherwise be low.
Explain in plain language why a system can run out of capacity during a sudden traffic spike even when its average load looks fine on a dashboard.
Averages smooth over spikes; hardware fails at peak concurrency, not average load, so planning only to average demand leaves a system vulnerable to burst failures.
An AI assistant recommends a specific GPU configuration for your workload but doesn't mention the model's sequence length, batch size, or whether you need training or inference. What is the most important reason to flag this advice as incomplete?
Hardware sizing is always conditional on workload specifics (model size, precision, sequence length, concurrency targets); a recommendation without those assumptions cannot be verified or trusted.