Gpu on François Duthilleul

Confidential GPU

Mon, 01 Jan 0001 00:00:00 +0000

A Confidential GPU is a GPU whose memory, computation state, and data transfers are hardware-encrypted and isolated from the host system — extending the Trusted Execution Environment (TEE) boundary that technologies like TDX and SEV-SNP provide at the CPU level to encompass the GPU accelerator as well. Without that extension, a GPU sits outside the CPU TEE: it cannot read TEE memory, and any data offloaded to it would leave the confidential boundary. The primary implementation today is NVIDIA Confidential Computing, introduced on Hopper (H100), continued on Blackwell, and planned for Rubin as the third generation. In Confidential Computing mode, the GPU encrypts data resident in High Bandwidth Memory (HBM) under keys managed by its on-die security processor, and three hardware properties close the trust gap to the CPU TEE: an on-die Root of Trust that verifies GPU firmware authenticity before the OS can talk to the device; device attestation via the NVIDIA Remote Attestation Service (NRAS), which produces signed evidence that the GPU is genuine NVIDIA hardware in Confidential Computing mode with unmodified firmware (analogous to Intel DCAP or AMD KDS for CPU TEEs); and encrypted PCIe transfers between the CPU TEE and GPU TEE at line rate using a hardware AES-256-GCM implementation, so a host administrator, hypervisor, or co-tenant with DMA access to the bus sees only ciphertext. GPU attestation is verified alongside CPU attestation before secrets (model decryption keys, dataset credentials) are released to the combined CPU+GPU TEE. The technology requires no application code changes — existing TensorFlow, PyTorch, and CUDA workloads run unmodified inside the confidential boundary. The primary threat model is the same as CPU-level confidential computing (protecting data-in-use from the infrastructure operator) but applied to AI workloads: model intellectual property theft, training data exfiltration, and inference input/output interception during GPU computation.

CUDA (Compute Unified Device Architecture)

Mon, 01 Jan 0001 00:00:00 +0000

CUDA (Compute Unified Device Architecture) is NVIDIA’s software platform for general-purpose computing on GPUs. Its objective is to give developers a familiar C/C++ (and Fortran, Python bindings) programming model with explicit control over kernels (functions that run on the device), streams (ordered queues of work), and memory spaces (host, device, unified). CUDA sits above the GPU driver and below frameworks such as cuDNN, NCCL, and higher-level ML stacks; it is the layer that makes it practical to implement custom operators, HPC solvers, and inference engines that are not covered by a closed library. For AI, virtually every major training and inference stack ultimately depends on CUDA (or a CUDA-compatible runtime) on NVIDIA hardware.

cuDNN (CUDA Deep Neural Network library)

Mon, 01 Jan 0001 00:00:00 +0000

cuDNN (CUDA Deep Neural Network library) is NVIDIA’s library of highly optimized GPU kernels for operations that dominate deep learning: convolutions, matrix multiplies used in attention, pooling, normalization (batch/layer), activations, and recurrent cells. Its objective is to deliver near-peak performance on CUDA-capable GPUs without every framework author hand-writing assembly-tuned kernels. PyTorch, TensorFlow, and many inference engines call cuDNN (directly or via cuBLAS) under the hood for training and serving. cuDNN sits between raw CUDA and application code; version alignment with the CUDA toolkit and driver is mandatory for supported deployments.

GPU (Graphics Processing Unit)

Mon, 01 Jan 0001 00:00:00 +0000

A GPU (Graphics Processing Unit) is a specialized processor designed to execute a very large number of arithmetic operations in parallel. Its original objective was real-time rendering; in modern AI and HPC infrastructure the same silicon is used to accelerate matrix multiplications, convolutions, and other kernels that dominate neural network training and inference. Unlike a general-purpose host, a GPU optimizes for throughput: many warps or wavefronts hide memory latency while the device keeps SIMD units busy. In a data-center stack, GPUs typically sit in PCIe or NVLink-attached servers (or on integrated AI appliances) and are scheduled by frameworks such as PyTorch, TensorFlow, or vLLM through a runtime such as CUDA or ROCm.

MIG (Multi-Instance GPU)

Mon, 01 Jan 0001 00:00:00 +0000

MIG (Multi-Instance GPU) is an NVIDIA GPU partitioning mode on datacenter accelerators (e.g. A100, H100) that splits one physical card into up to seven GPU instances (GIs), each with isolated streaming multiprocessors, memory bandwidth, and HBM capacity. The objective is higher utilization in multi-tenant environments: several smaller models or dev/test workloads share one expensive GPU without time-slicing contention as severe as full-card sharing. Each MIG instance appears to the OS and CUDA as a separate GPU with fixed resources; workloads cannot oversubscribe another instance’s memory. MIG suits inference and modest training more often than massive single-job training that needs the entire GPU and NVLink domain.

NCCL (NVIDIA Collective Communications Library)

Mon, 01 Jan 0001 00:00:00 +0000

NCCL (NVIDIA Collective Communications Library) implements collective operations—all-reduce, broadcast, reduce-scatter, all-gather, and others—optimized for NVIDIA GPUs across NVLink within a node and RDMA (InfiniBand or RoCE) across nodes. Its objective in AI is to make distributed training and multi-GPU inference (tensor parallelism) scale: gradient shards must merge every step; attention and MLP partitions must exchange activations with minimal latency. Frameworks (PyTorch DDP/FSDP, vLLM tensor parallel) call NCCL (or delegate to it) rather than hand-rolling socket code.

NIM (NVIDIA Inference Microservices)

Mon, 01 Jan 0001 00:00:00 +0000

NIM (NVIDIA Inference Microservices) are container images and Helm charts that deliver ready-to-run inference endpoints for specific models (LLMs, vision, embedding, reranking, and more). The objective is to shrink time-to-production: instead of assembling CUDA drivers, frameworks, model weights, and an OpenAI-compatible server yourself, operators pull a NIM that bundles a performance-tuned engine (often TensorRT-LLM or Triton-backed paths), default model artifacts or download hooks, health checks, and a stable HTTP/gRPC API. NIMs are sized for GPU deployment and target enterprise MLOps teams that want versioned, scannable containers with predictable resource requests rather than bespoke notebooks turned into scripts.

NVLink

Mon, 01 Jan 0001 00:00:00 +0000

NVLink is NVIDIA’s proprietary high-speed interconnect between GPUs (and, on some platforms, between GPUs and CPUs) inside a server or across an NVLink switch system (e.g. NVL72-class racks). Its objective is to move tensors—activations, gradients, KV cache shards, or partial attention results—at much higher bandwidth and lower latency than PCIe or general Ethernet, so multi-GPU training and large-model inference (tensor parallelism) are not bottlenecked on the bus. NVLink domains define which GPUs can treat each other’s memory as peer-accessible for CUDA and NCCL without leaving the box.

Quantization

Mon, 01 Jan 0001 00:00:00 +0000

Quantization is the process of representing a model’s weights and/or activations with fewer bits than full FP32 training precision—commonly FP16, BF16, FP8, INT8, or INT4 (GPTQ, AWQ, GGUF-style formats). The objective is lower GPU memory (larger models or more concurrent sessions per card), higher throughput, and sometimes faster kernels on hardware with native low-precision units, at the cost of possible quality degradation if pushed too aggressively. Quantization can be applied post-training (calibration on a sample dataset) or during training (quantization-aware training). For inference, serving engines vLLM and NIM load quantized checkpoints and dispatch to vendor libraries (TensorRT-LLM, CUTLASS, etc.) that implement fused low-precision matmuls.

RDMA (Remote Direct Memory Access)

Mon, 01 Jan 0001 00:00:00 +0000

RDMA (Remote Direct Memory Access) allows a network adapter to transfer data between the memory of two machines with little CPU overhead, low latency, and often kernel bypass (userspace stacks such as verbs on InfiniBand or RoCE). Its objective in AI infrastructure is to keep GPUs fed and synchronized: distributed training exchanges gradients quickly, disaggregated inference (llm-d) moves KV cache blocks between prefill and decode nodes, and NVMe-oF storage delivers checkpoints without the host spending cycles copying every byte. DPUs and SmartNICs also use RDMA paths for storage and east-west traffic while the host CPU runs models.

ROCm (Radeon Open Compute)

Mon, 01 Jan 0001 00:00:00 +0000

ROCm (Radeon Open Compute) is AMD’s software stack for GPU compute on datacenter Instinct accelerators (and select consumer GPUs in community setups). Its objective mirrors CUDA for NVIDIA: provide kernel compilers (HIP), math libraries (rocBLAS, rocFFT), collective communication (RCCL, analogous to NCCL), and framework integrations so PyTorch and inference runtimes can execute training and inference on AMD hardware. ROCm is positioned as an open platform (Linux-first) for customers who want accelerator choice or who standardize on AMD in HPC and AI clusters.

TensorRT

Mon, 01 Jan 0001 00:00:00 +0000

TensorRT is NVIDIA’s SDK for optimizing and deploying trained neural networks for inference on NVIDIA GPUs. Its objective is minimum latency and maximum throughput: it ingests a model (ONNX, TensorFlow, PyTorch export, or framework-specific parsers), applies graph optimizations (layer fusion, constant folding, kernel autotuning), selects precisions (FP32, FP16, INT8, FP8), and produces a serialized engine executed by a lightweight runtime. For LLMs, TensorRT-LLM extends this with attention-specific fusions, inflight batching, and multi-GPU serving patterns; many NIM microservices bundle TensorRT-LLM–optimized engines rather than raw PyTorch loops.

Training

Mon, 01 Jan 0001 00:00:00 +0000

Training is the phase of machine learning where model parameters are adjusted to minimize a loss on a dataset. For deep learning, that means repeated forward passes (compute predictions), backward passes (propagate gradients via autodiff), and optimizer steps (update weights)—from scratch pretraining, continued pretraining, or fine-tuning (full, LoRA, or other parameter-efficient methods). The objective is model quality (accuracy, perplexity, task metrics) within a compute and time budget, not millisecond response to end users. Training jobs are batch-oriented: large minibatches, epochs over terabytes of tokens or images, checkpointing to durable storage, and experiment tracking. LLM training at scale uses distributed strategies—data parallel, tensor parallel, pipeline parallel, and expert parallel for MoE—coordinated by frameworks such as PyTorch with FSDP or DeepSpeed.

vLLM

Mon, 01 Jan 0001 00:00:00 +0000

vLLM is an open-source library and serving stack for large language model (LLM) inference. Its objective is to turn a trained model into a production service that sustains many concurrent users with low latency and high tokens per second per GPU. vLLM targets the inference phase (prefill + decode), not training: it loads weights onto accelerators, batches incoming prompts, schedules decode steps, and streams completions back to clients over HTTP/gRPC (often via an OpenAI-compatible API). It has become a de facto engine behind many private and cloud AI gateways because it ships integrations for Hugging Face models, LoRA adapters, tensor parallelism, pipeline parallelism, speculative decoding, and quantization (GPTQ, AWQ, FP8).