Glossary
Definitions for the technical terms used on this site. The goal is consistency: when a word appears in multiple places, it should mean the same thing.
Common terms
Large Language Model (LLM)
A neural network trained to predict and generate text. Local LLMs run entirely on your own hardware instead of through a cloud API.
Quantization
A technique that reduces the numerical precision of model weights, typically from 16-bit to 4-bit or 8-bit, to lower memory usage and increase inference speed.
KV cache
Key-value cache. Memory used during text generation to store intermediate attention states, allowing the model to generate subsequent tokens without recomputing the entire context.
Unified memory
A memory architecture where the CPU, GPU, and other accelerators share a single pool of RAM, eliminating copies across PCIe.
Memory bandwidth
The rate at which data can be read from or written to memory. For large-model inference, bandwidth is often the limiting factor.
Mixture of Experts (MoE)
A model architecture that uses multiple specialized sub-networks (experts) and a routing mechanism, activating only a subset per token.
Retrieval-Augmented Generation (RAG)
A pattern where a model retrieves relevant documents from a knowledge base before generating a response, keeping answers grounded in private data.
AI agent
A system that uses a language model to plan and execute multi-step tasks, often invoking tools such as search, code execution, or APIs.
NPU
Neural Processing Unit. A dedicated accelerator for AI inference, typically optimized for low-power, sustained neural-network execution.
iGPU
Integrated GPU. A graphics processor built into the same chip as the CPU, sharing system or unified memory.
VRAM
Video RAM. Dedicated memory on a discrete graphics card. Systems without discrete GPUs, such as Strix Halo, use unified memory instead.
TOPS
Tera Operations Per Second. A marketing metric for AI compute throughput, commonly used for NPU performance claims.
Context length
The maximum amount of text a model can consider at once. Longer contexts require more memory and bandwidth.
Token
A unit of text processed by a language model. Tokens can be words, parts of words, or characters, depending on the tokenizer.
Inference
The process of running a trained model to generate outputs, as opposed to training the model.
ROCm
AMD's open-source GPU compute stack, roughly equivalent to NVIDIA's CUDA for running AI and HPC workloads on AMD GPUs.