GPU Workloads on Kubernetes: Time-Slicing, MIG and MPS

How does GPU scheduling work in Kubernetes?

Each GPU worker node needs the vendor driver and a compatible device plugin. The plugin advertises GPU capacity on the node as an extended resource. An application requests that GPU resource as a limit in its Pod definition, and the scheduler considers only nodes that can satisfy it.

Clusters with different GPU models or sharing profiles can use node labels, node selectors, affinity and taint or toleration rules. The scheduler does not measure whether a model fits in memory, its inference latency or GPU utilization. Match the resource request, image, driver, CUDA libraries and node profile before deployment.

When should you allocate a full GPU?

For model training, large-batch inference, rendering and HPC jobs that use much of GPU memory, a full GPU per Pod is the most predictable starting point. Because the resource is not shared with other Pods, performance variation, memory contention and noisy-neighbor effects are reduced.

Small models, low-traffic inference endpoints and development notebooks may not continuously fill GPU compute and memory. Sharing can improve utilization and cost in that case, but isolation and performance objectives must be explicit.

What is GPU time-slicing?

Time-slicing advertises multiple logical resources on one physical GPU so workloads take turns using GPU time. NVIDIA device plugin configuration can define a replica count and allocate those resources independently to Pods. This supports capacity sharing for intermittent workloads that do not saturate a GPU alone.

A replica is not a guaranteed slice of physical GPU memory or compute. Time-slicing does not provide memory or fault isolation between replicas, so one workload can affect others by consuming memory or failing. Requesting more replicas also does not guarantee proportionally more GPU compute.

What does MIG provide?

Multi-Instance GPU partitions a supported NVIDIA GPU into predefined hardware profiles. Each MIG instance can be advertised to Kubernetes with separate compute and memory resources. Hardware-level memory and fault isolation helps multiple teams or inference workloads share one physical GPU with clearer boundaries.

MIG is not available on every GPU model, and profile sizes depend on the hardware. Changing MIG mode or geometry can require stopping node workloads and may require a node restart in some environments. The model must fit the profile memory, while multi-GPU communication or full-GPU training should be tested separately.

When is NVIDIA MPS useful?

Multi-Process Service is a client-server runtime that helps compatible CUDA processes execute concurrently on the same GPU. When one process does not saturate the GPU, it can overlap kernel and data-copy work from different processes and reduce context-switching overhead.

MPS is not hardware partitioning like MIG and does not provide multi-tenant memory or fault isolation. Test the user model, failure behavior, process accounting, memory limits and how observability tools report the MPS server.

How should you choose time-slicing, MIG or MPS?

Start with the isolation requirement. Full-GPU allocation or MIG on supported hardware gives stronger boundaries for untrusted workloads or independent teams. Time-slicing can offer simple capacity sharing for small, intermittent jobs within one trust boundary. MPS can be evaluated for compatible CUDA processes that do not saturate the GPU individually.

• Full GPU: the most predictable performance and strong workload separation
• Time-slicing: broad compatibility, but no memory or fault isolation
• MIG: hardware profiles and isolation on supported GPUs
• MPS: concurrency and utilization improvements for compatible CUDA processes

What should you measure before production?

Measure GPU utilization, peak GPU memory, request latency, throughput, queue time, error rate and job completion time with a representative model and dataset. Repeat the test with increasing concurrent Pod counts when sharing is enabled, and do not replace service objectives such as p95 or p99 latency with averages alone.

VKE is the open-source Kubernetes engine developed by vMind. It automatically provides node-level autoscaling, load balancers and storage classes. Choose the sharing model from latency, memory and fault measurements on a representative workload.