Nvidia on François Duthilleul

Confidential GPU

Mon, 01 Jan 0001 00:00:00 +0000

A Confidential GPU is a GPU whose memory, computation state, and data transfers are hardware-encrypted and isolated from the host system — extending the Trusted Execution Environment (TEE) boundary that technologies like TDX and SEV-SNP provide at the CPU level to encompass the GPU accelerator as well. Without that extension, a GPU sits outside the CPU TEE: it cannot read TEE memory, and any data offloaded to it would leave the confidential boundary. The primary implementation today is NVIDIA Confidential Computing, introduced on Hopper (H100), continued on Blackwell, and planned for Rubin as the third generation. In Confidential Computing mode, the GPU encrypts data resident in High Bandwidth Memory (HBM) under keys managed by its on-die security processor, and three hardware properties close the trust gap to the CPU TEE: an on-die Root of Trust that verifies GPU firmware authenticity before the OS can talk to the device; device attestation via the NVIDIA Remote Attestation Service (NRAS), which produces signed evidence that the GPU is genuine NVIDIA hardware in Confidential Computing mode with unmodified firmware (analogous to Intel DCAP or AMD KDS for CPU TEEs); and encrypted PCIe transfers between the CPU TEE and GPU TEE at line rate using a hardware AES-256-GCM implementation, so a host administrator, hypervisor, or co-tenant with DMA access to the bus sees only ciphertext. GPU attestation is verified alongside CPU attestation before secrets (model decryption keys, dataset credentials) are released to the combined CPU+GPU TEE. The technology requires no application code changes — existing TensorFlow, PyTorch, and CUDA workloads run unmodified inside the confidential boundary. The primary threat model is the same as CPU-level confidential computing (protecting data-in-use from the infrastructure operator) but applied to AI workloads: model intellectual property theft, training data exfiltration, and inference input/output interception during GPU computation.

CUDA (Compute Unified Device Architecture)

Mon, 01 Jan 0001 00:00:00 +0000

CUDA (Compute Unified Device Architecture) is NVIDIA’s software platform for general-purpose computing on GPUs. Its objective is to give developers a familiar C/C++ (and Fortran, Python bindings) programming model with explicit control over kernels (functions that run on the device), streams (ordered queues of work), and memory spaces (host, device, unified). CUDA sits above the GPU driver and below frameworks such as cuDNN, NCCL, and higher-level ML stacks; it is the layer that makes it practical to implement custom operators, HPC solvers, and inference engines that are not covered by a closed library. For AI, virtually every major training and inference stack ultimately depends on CUDA (or a CUDA-compatible runtime) on NVIDIA hardware.

cuDNN (CUDA Deep Neural Network library)

Mon, 01 Jan 0001 00:00:00 +0000

cuDNN (CUDA Deep Neural Network library) is NVIDIA’s library of highly optimized GPU kernels for operations that dominate deep learning: convolutions, matrix multiplies used in attention, pooling, normalization (batch/layer), activations, and recurrent cells. Its objective is to deliver near-peak performance on CUDA-capable GPUs without every framework author hand-writing assembly-tuned kernels. PyTorch, TensorFlow, and many inference engines call cuDNN (directly or via cuBLAS) under the hood for training and serving. cuDNN sits between raw CUDA and application code; version alignment with the CUDA toolkit and driver is mandatory for supported deployments.

DOCA (Data Center Infrastructure on a Chip Architecture)

Mon, 01 Jan 0001 00:00:00 +0000

DOCA (Data Center Infrastructure on a Chip Architecture) is NVIDIA’s software framework for building and operating services on BlueField DPUs. Its objective is to standardize how operators and ISVs develop infrastructure applications—OVS offload, firewall/VNF, storage targets, RDMA/RoCE control, TLS inspection, telemetry agents—on Arm cores and hardware accelerators embedded in the NIC, using a consistent set of libraries instead of ad hoc kernel modules on the host. DOCA spans drivers, userspace APIs, reference pipelines, and marketplace-packaged applications; it is the DPU counterpart to CUDA on GPUs, oriented toward I/O and packet processing rather than tensor math.

MIG (Multi-Instance GPU)

Mon, 01 Jan 0001 00:00:00 +0000

MIG (Multi-Instance GPU) is an NVIDIA GPU partitioning mode on datacenter accelerators (e.g. A100, H100) that splits one physical card into up to seven GPU instances (GIs), each with isolated streaming multiprocessors, memory bandwidth, and HBM capacity. The objective is higher utilization in multi-tenant environments: several smaller models or dev/test workloads share one expensive GPU without time-slicing contention as severe as full-card sharing. Each MIG instance appears to the OS and CUDA as a separate GPU with fixed resources; workloads cannot oversubscribe another instance’s memory. MIG suits inference and modest training more often than massive single-job training that needs the entire GPU and NVLink domain.

NCCL (NVIDIA Collective Communications Library)

Mon, 01 Jan 0001 00:00:00 +0000

NCCL (NVIDIA Collective Communications Library) implements collective operations—all-reduce, broadcast, reduce-scatter, all-gather, and others—optimized for NVIDIA GPUs across NVLink within a node and RDMA (InfiniBand or RoCE) across nodes. Its objective in AI is to make distributed training and multi-GPU inference (tensor parallelism) scale: gradient shards must merge every step; attention and MLP partitions must exchange activations with minimal latency. Frameworks (PyTorch DDP/FSDP, vLLM tensor parallel) call NCCL (or delegate to it) rather than hand-rolling socket code.

NIM (NVIDIA Inference Microservices)

Mon, 01 Jan 0001 00:00:00 +0000

NIM (NVIDIA Inference Microservices) are container images and Helm charts that deliver ready-to-run inference endpoints for specific models (LLMs, vision, embedding, reranking, and more). The objective is to shrink time-to-production: instead of assembling CUDA drivers, frameworks, model weights, and an OpenAI-compatible server yourself, operators pull a NIM that bundles a performance-tuned engine (often TensorRT-LLM or Triton-backed paths), default model artifacts or download hooks, health checks, and a stable HTTP/gRPC API. NIMs are sized for GPU deployment and target enterprise MLOps teams that want versioned, scannable containers with predictable resource requests rather than bespoke notebooks turned into scripts.

NVLink

Mon, 01 Jan 0001 00:00:00 +0000

NVLink is NVIDIA’s proprietary high-speed interconnect between GPUs (and, on some platforms, between GPUs and CPUs) inside a server or across an NVLink switch system (e.g. NVL72-class racks). Its objective is to move tensors—activations, gradients, KV cache shards, or partial attention results—at much higher bandwidth and lower latency than PCIe or general Ethernet, so multi-GPU training and large-model inference (tensor parallelism) are not bottlenecked on the bus. NVLink domains define which GPUs can treat each other’s memory as peer-accessible for CUDA and NCCL without leaving the box.

TensorRT

Mon, 01 Jan 0001 00:00:00 +0000

TensorRT is NVIDIA’s SDK for optimizing and deploying trained neural networks for inference on NVIDIA GPUs. Its objective is minimum latency and maximum throughput: it ingests a model (ONNX, TensorFlow, PyTorch export, or framework-specific parsers), applies graph optimizations (layer fusion, constant folding, kernel autotuning), selects precisions (FP32, FP16, INT8, FP8), and produces a serialized engine executed by a lightweight runtime. For LLMs, TensorRT-LLM extends this with attention-specific fusions, inflight batching, and multi-GPU serving patterns; many NIM microservices bundle TensorRT-LLM–optimized engines rather than raw PyTorch loops.