Amazon SageMaker can now split one GPU into isolated partitions

7
Most partitions on one GPU: 24
Partitions on one p4d endpoint: 17 sec
Added to provisioning in testing

Who it's for: Teams hosting models
Where it works: SageMaker AI endpoints
When: Available now

Key points

Faults stay inside their own partition
Dedicated memory bandwidth per partition
Slice instead of a whole GPU
Deploy more models than GPUs

Where to use it

Hosting several modelsPut each model in its own GPU partition
Small models, high trafficRun more copies of a small model
Cost trackingMap GPU cost to each model or team

What it does

Amazon SageMaker AI can now divide one GPU into separate partitions, each with its own share of compute and memory, so several models can run on one accelerator.

Who it's for

Teams that host AI models on AWS endpoints and want to use their GPUs more fully.

Price and where to try it

The post does not say what it costs. It works on SageMaker AI endpoints that use inference components, not single-model or serverless endpoints.

Source

AWS: SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints

Written with AI from the source above. If something looks off, the source is what counts.