Resource Management

• 1: Resource managers
• 2: Dynamic Resource Allocation
• 2.1: DRA API Objects
• 2.2: How DRA Works
• 2.3: Observability of Dynamic Resources
• 2.4: DRA Features
• 2.5: Device Taints and Tolerations
• 3: Pod-level resource managers

1 - Resource managers

In order to support latency-critical and high-throughput workloads, Kubernetes offers a suite of Resource Managers. The managers aim to co-ordinate and optimize the alignment of node's resources for pods configured with a specific requirement for CPUs, devices, and memory (hugepages) resources.

Topology manager

Topology Manager is a kubelet component that aims to coordinate the set of components that are responsible for these optimizations. To learn more, read Control Topology Management Policies on a Node.

CPU manager

CPU Manager is a kubelet component that provides exclusive resource allocation for CPU resources. It consults with the Topology Manager to make resource assignment decisions. To learn more, read Control CPU Management Policies on the Node.

Policies for assigning CPUs to Pods

Once a Pod is bound to a Node, the kubelet on that node may need to either multiplex the existing hardware (for example, sharing CPUs across multiple Pods) or allocate hardware by dedicating some resource (for example, assigning one of more CPUs for a Pod's exclusive use).

By default, the kubelet uses CFS quota to enforce pod CPU limits.  When the node runs many CPU-bound pods, the workload can move to different CPU cores depending on whether the pod is throttled and which CPU cores are available at scheduling time. Many workloads are not sensitive to this migration and thus work fine without any intervention.

However, in workloads where CPU cache affinity and scheduling latency significantly affect workload performance, the kubelet allows alternative CPU management policies to determine some placement preferences on the node. This is implemented using the CPU Manager and its policy. There are two available policies:

• none: the none policy explicitly enables the existing default CPU affinity scheme, providing no affinity beyond what the OS scheduler does automatically.  Limits on CPU usage for Guaranteed pods and Burstable pods are enforced using CFS quota.
• static: the static policy allows containers in Guaranteed pods with integer CPU requests access to exclusive CPUs on the node. This exclusivity is enforced using the cpuset cgroup controller.

CPU Manager doesn't support offlining and onlining of CPUs at runtime.

Static policy

The static policy enables finer-grained CPU management and exclusive CPU assignment. This policy manages a shared pool of CPUs that initially contains all CPUs in the node. The amount of exclusively allocatable CPUs is equal to the total number of CPUs in the node minus any CPU reservations set by the kubelet configuration. CPUs reserved by these options are taken, in integer quantity, from the initial shared pool in ascending order by physical core ID.  This shared pool is the set of CPUs on which any containers in BestEffort and Burstable pods run. Containers in Guaranteed pods with fractional CPU requests also run on CPUs in the shared pool. Only containers that are part of a Guaranteed pod and have integer CPU requests are assigned exclusive CPUs.

As Guaranteed pods whose containers fit the requirements for being statically assigned are scheduled to the node, CPUs are removed from the shared pool and placed in the cpuset for the container. CFS quota is not used to bound the CPU usage of these containers as their usage is bound by the scheduling domain itself. In others words, the number of CPUs in the container cpuset is equal to the integer CPU limit specified in the pod spec. This static assignment increases CPU affinity and decreases context switches due to throttling for the CPU-bound workload.

Consider the containers in the following pod specs:

spec:
  containers:
  - name: nginx
    image: nginx

The pod above runs in the BestEffort QoS class because no resource requests or limits are specified. It runs in the shared pool.

spec:
  containers:
  - name: nginx
    image: nginx
    resources:
      limits:
        memory: "200Mi"
      requests:
        memory: "100Mi"

The pod above runs in the Burstable QoS class because resource requests do not equal limits and the cpu quantity is not specified. It runs in the shared pool.

spec:
  containers:
  - name: nginx
    image: nginx
    resources:
      limits:
        memory: "200Mi"
        cpu: "2"
      requests:
        memory: "100Mi"
        cpu: "1"

The pod above runs in the Burstable QoS class because resource requests do not equal limits. It runs in the shared pool.

spec:
  containers:
  - name: nginx
    image: nginx
    resources:
      limits:
        memory: "200Mi"
        cpu: "2"
      requests:
        memory: "200Mi"
        cpu: "2"

The pod above runs in the Guaranteed QoS class because requests are equal to limits. And the container's resource limit for the CPU resource is an integer greater than or equal to one. The nginx container is granted 2 exclusive CPUs.

spec:
  containers:
  - name: nginx
    image: nginx
    resources:
      limits:
        memory: "200Mi"
        cpu: "1.5"
      requests:
        memory: "200Mi"
        cpu: "1.5"

The pod above runs in the Guaranteed QoS class because requests are equal to limits. But the container's resource limit for the CPU resource is a fraction. It runs in the shared pool.

spec:
  containers:
  - name: nginx
    image: nginx
    resources:
      limits:
        memory: "200Mi"
        cpu: "2"

The pod above runs in the Guaranteed QoS class because only limits are specified and requests are set equal to limits when not explicitly specified. And the container's resource limit for the CPU resource is an integer greater than or equal to one. The nginx container is granted 2 exclusive CPUs.

Static policy options

Here are the available policy options for the static CPU management policy, listed in alphabetical order:

align-by-socket (alpha, hidden by default)

Align CPUs by physical package / socket boundary, rather than logical NUMA boundaries (available since Kubernetes v1.25)

distribute-cpus-across-cores (alpha, hidden by default)

Allocate virtual cores, sometimes called hardware threads, across different physical cores (available since Kubernetes v1.31)

distribute-cpus-across-numa (beta, visible by default)

Spread CPUs across different NUMA domains, aiming for an even balance between the selected domains (available since Kubernetes v1.23)

full-pcpus-only (GA, visible by default)

Always allocate full physical cores (available since Kubernetes v1.22, GA since Kubernetes v1.33)

strict-cpu-reservation (GA, visible by default)

Prevent all the pods regardless of their Quality of Service class to run on reserved CPUs (available since Kubernetes v1.32, GA since Kubernetes v1.35)

prefer-align-cpus-by-uncorecache (GA, visible by default)

Align CPUs by uncore (Last-Level) cache boundary on a best-effort way (available since Kubernetes v1.32)

You can toggle groups of options on and off based upon their maturity level using the following feature gates:

• CPUManagerPolicyBetaOptions (default enabled). Disable to hide beta-level options.
• CPUManagerPolicyAlphaOptions (default disabled). Enable to show alpha-level options.

You will still have to enable each option using the cpuManagerPolicyOptions field in the kubelet configuration file.

For more detail about the individual options you can configure, read on.

full-pcpus-only

If the full-pcpus-only policy option is specified, the static policy will always allocate full physical cores. By default, without this option, the static policy allocates CPUs using a topology-aware best-fit allocation. On SMT enabled systems, the policy can allocate individual virtual cores, which correspond to hardware threads. This can lead to different containers sharing the same physical cores; this behaviour in turn contributes to the noisy neighbours problem. With the option enabled, the pod will be admitted by the kubelet only if the CPU request of all its containers can be fulfilled by allocating full physical cores. If the pod does not pass the admission, it will be put in Failed state with the message SMTAlignmentError.

distribute-cpus-across-numa

If the distribute-cpus-across-numapolicy option is specified, the static policy will evenly distribute CPUs across NUMA nodes in cases where more than one NUMA node is required to satisfy the allocation. By default, the CPUManager will pack CPUs onto one NUMA node until it is filled, with any remaining CPUs simply spilling over to the next NUMA node. This can cause undesired bottlenecks in parallel code relying on barriers (and similar synchronization primitives), as this type of code tends to run only as fast as its slowest worker (which is slowed down by the fact that fewer CPUs are available on at least one NUMA node). By distributing CPUs evenly across NUMA nodes, application developers can more easily ensure that no single worker suffers from NUMA effects more than any other, improving the overall performance of these types of applications.

align-by-socket

If the align-by-socket policy option is specified, CPUs will be considered aligned at the socket boundary when deciding how to allocate CPUs to a container. By default, the CPUManager aligns CPU allocations at the NUMA boundary, which could result in performance degradation if CPUs need to be pulled from more than one NUMA node to satisfy the allocation. Although it tries to ensure that all CPUs are allocated from the minimum number of NUMA nodes, there is no guarantee that those NUMA nodes will be on the same socket. By directing the CPUManager to explicitly align CPUs at the socket boundary rather than the NUMA boundary, we are able to avoid such issues. Note, this policy option is not compatible with TopologyManager single-numa-node policy and does not apply to hardware where the number of sockets is greater than number of NUMA nodes.

distribute-cpus-across-cores

If the distribute-cpus-across-cores policy option is specified, the static policy will attempt to allocate virtual cores (hardware threads) across different physical cores. By default, the CPUManager tends to pack CPUs onto as few physical cores as possible, which can lead to contention among CPUs on the same physical core and result in performance bottlenecks. By enabling the distribute-cpus-across-cores policy, the static policy ensures that CPUs are distributed across as many physical cores as possible, reducing the contention on the same physical core and thereby improving overall performance. However, it is important to note that this strategy might be less effective when the system is heavily loaded. Under such conditions, the benefit of reducing contention diminishes. Conversely, default behavior can help in reducing inter-core communication overhead, potentially providing better performance under high load conditions.

strict-cpu-reservation

The reservedSystemCPUs parameter in KubeletConfiguration, or the deprecated kubelet command line option --reserved-cpus, defines an explicit CPU set for OS system daemons and kubernetes system daemons. More details of this parameter can be found on the Explicitly Reserved CPU List page. By default, this isolation is implemented only for guaranteed pods with integer CPU requests not for burstable and best-effort pods (and guaranteed pods with fractional CPU requests). Admission is only comparing the CPU requests against the allocatable CPUs. Since the CPU limit is higher than the request, the default behaviour allows burstable and best-effort pods to use up the capacity of reservedSystemCPUs and cause host OS services to starve in real life deployments. If the strict-cpu-reservation policy option is enabled, the static policy will not allow any workload to use the CPU cores specified in reservedSystemCPUs.

prefer-align-cpus-by-uncorecache

If the prefer-align-cpus-by-uncorecache policy is specified, the static policy will allocate CPU resources for individual containers such that all CPUs assigned to a container share the same uncore cache block (also known as the Last-Level Cache or LLC). By default, the CPUManager will tightly pack CPU assignments which can result in containers being assigned CPUs from multiple uncore caches. This option enables the CPUManager to allocate CPUs in a way that maximizes the efficient use of the uncore cache. Allocation is performed on a best-effort basis, aiming to affine as many CPUs as possible within the same uncore cache. If the container's CPU requirement exceeds the CPU capacity of a single uncore cache, the CPUManager minimizes the number of uncore caches used in order to maintain optimal uncore cache alignment. Specific workloads can benefit in performance from the reduction of inter-cache latency and noisy neighbors at the cache level. If the CPUManager cannot align optimally while the node has sufficient resources, the container will still be admitted using the default packed behavior.

Memory manager

Memory Manager is a kubelet component that provides exclusive resource allocation for memory resources. It consults with the Topology Manager to make resource assignment decisions. To learn more, read Control Memory Management Policies on a Node.

Policies for assigning memory to Pods

The Kubernetes Memory Manager allocates RAM (memory, and optionally Linux huge pages) resources for pods in the Guaranteed QoS class.

The Memory Manager employs hint generation protocol to yield the most suitable NUMA affinity for a pod. The Memory Manager feeds the central manager (Topology Manager) with these affinity hints. Based on both the hints and Topology Manager policy, the pod is rejected or admitted to the node.

Moreover, the Memory Manager ensures that the memory which a pod requests is allocated from a minimum number of NUMA nodes.

To learn more, read Control Memory Management Policies on a Node.

Device manager

Device Manager is a kubelet component that allocates hardware devices to pods using the device plugin API. It consults with the Topology Manager, using topology information provided by device plugins, to make resource assignment decisions. To learn more, read Device Plugin Integration with the Topology Manager.

Pod-level resource managers

Pod-level resource support for kubelet resource managers (Topology, CPU, and Memory) allows resource managers to use .spec.resources directly for NUMA alignment and exclusive allocation decisions.

To learn more, see the dedicated Pod-level resource managers concept page, or read how to Assign Pod-level CPU and memory resources.

What's next

• Node Resource Managers

2 - Dynamic Resource Allocation

This section introduces dynamic resource allocation (DRA) in Kubernetes.

DRA is a Kubernetes feature that lets you request and share resources among Pods. These resources are often attached devices like hardware accelerators.

With DRA, device drivers and cluster admins define device classes that are available to claim in workloads. Kubernetes allocates matching devices to specific claims and places the corresponding Pods on nodes that can access the allocated devices.

Allocating resources with DRA offers a similar experience to dynamic volume provisioning, in which you use PersistentVolumeClaims to claim storage capacity from storage classes, and request the claimed capacity for use in your Pods.

Benefits of DRA

DRA provides a flexible way to categorize, request, and use devices in your cluster. Using DRA provides benefits like the following:

• Flexible device filtering: use common expression language (CEL) to perform fine-grained filtering for specific device attributes.
• Device sharing: share the same resource with multiple containers or Pods by referencing the corresponding resource claim.
• Device configuration: attach vendor-specific device configurations to your resource claim, enabling per-workload device configuration, rather than today's per-node device configuration
• Centralized device categorization: device drivers and cluster admins can use device classes to provide app operators with hardware categories that are optimized for various use cases. For example, you can create a cost-optimized device class for general-purpose workloads, and a high-performance device class for critical jobs.
• Simplified Pod requests: with DRA, app operators don't need to specify device quantities in Pod resource requests. Instead, the Pod references a resource claim, and the device configuration in that claim applies to the Pod.

These benefits provide significant improvements in the device allocation workflow when compared to device plugins, which require per-container device requests, don't support device sharing, and don't support expression-based device filtering.

Types of DRA users

The workflow of using DRA to allocate devices involves the following types of users:

• Device owner: responsible for devices. Device owners might be commercial vendors, the cluster operator, or another entity. To use DRA, devices must have DRA-compatible drivers that do the following:

  • Create ResourceSlices that provide Kubernetes with information about nodes and resources.
  • Update ResourceSlices when resource capacity in the cluster changes.
  • Configure devices according to the claim, and attach them to containers via Container Device Interface (CDI).
  • Optionally, create DeviceClasses that workload operators can use to claim devices.

• Cluster admin: responsible for configuring clusters and nodes, attaching devices, installing drivers, and similar tasks. To use DRA, cluster admins do the following:

  • Attach devices to nodes.
  • Install device drivers that support DRA.
  • Optionally, create DeviceClasses that workload operators can use to claim devices.

• Workload operator: responsible for deploying and managing workloads in the cluster. To use DRA to allocate devices to Pods, workload operators do the following:

  • Create ResourceClaims or ResourceClaimTemplates to request specific configurations within DeviceClasses.
  • Deploy workloads that use specific ResourceClaims or ResourceClaimTemplates.

Limitations

• The Kubernetes scheduler doesn't support preemption for DRA resources. This means that an existing Pod that's running on a node and is using DRA resources can't be preempted by a higher-priority Pod that also needs DRA resources. The high-priority Pod will remain in a pending state until the device becomes available, which happens when the conflicting Pod terminates or is manually deleted.

What's next

• Set Up DRA in a Cluster
• Allocate devices to workloads using DRA
• Access DRA device metadata
• For more information on the design, see the Dynamic Resource Allocation with Structured Parameters KEP.

2.1 - DRA API Objects

This page describes the Kubernetes API kinds that dynamic resource allocation (DRA) uses to categorize, request, and allocate devices.

DRA terminology

DRA uses the following Kubernetes API kinds to provide the core allocation functionality. All of these API kinds are included in the resource.k8s.io/v1 API group.

DeviceClass

Defines a category of devices that can be claimed and how to select specific device attributes in claims. The DeviceClass parameters can match zero or more devices in ResourceSlices. To claim devices from a DeviceClass, ResourceClaims select specific device attributes.

ResourceClaim

Describes a request for access to attached resources, such as devices, in the cluster. ResourceClaims provide Pods with access to a specific resource. ResourceClaims can be created by workload operators or generated by Kubernetes based on a ResourceClaimTemplate.

ResourceClaimTemplate

Defines a template that Kubernetes uses to create per-Pod ResourceClaims for a workload. ResourceClaimTemplates provide Pods with access to separate, similar resources. Each ResourceClaim that Kubernetes generates from the template is bound to a specific Pod. When the Pod terminates, Kubernetes deletes the corresponding ResourceClaim.

ResourceSlice

Represents one or more resources that are attached to nodes, such as devices. Drivers create and manage ResourceSlices in the cluster. When a ResourceClaim is created and used in a Pod, Kubernetes uses ResourceSlices to find nodes that have access to the claimed resources. Kubernetes allocates resources to the ResourceClaim and schedules the Pod onto a node that can access the resources.

DeviceClass

A DeviceClass lets cluster admins or device drivers define categories of devices in the cluster. DeviceClasses tell operators what devices they can request and how they can request those devices. You can use common expression language (CEL) to select devices based on specific attributes. A ResourceClaim that references the DeviceClass can then request specific configurations within the DeviceClass.

To create a DeviceClass, see Set Up DRA in a Cluster.

ResourceClaims and ResourceClaimTemplates

A ResourceClaim defines the resources that a workload needs. Every ResourceClaim has requests that reference a DeviceClass and select devices from that DeviceClass. ResourceClaims can also use selectors to filter for devices that meet specific requirements, and can use constraints to limit the devices that can satisfy a request. ResourceClaims can be created by workload operators or can be generated by Kubernetes based on a ResourceClaimTemplate. A ResourceClaimTemplate defines a template that Kubernetes can use to auto-generate ResourceClaims for Pods.

Use cases for ResourceClaims and ResourceClaimTemplates

The method that you use depends on your requirements, as follows:

• ResourceClaim: you want multiple Pods to share access to specific devices. You manually manage the lifecycle of ResourceClaims that you create.
• ResourceClaimTemplate: you want Pods to have independent access to separate, similarly-configured devices. Kubernetes generates ResourceClaims from the specification in the ResourceClaimTemplate. The lifetime of each generated ResourceClaim is bound to the lifetime of the corresponding Pod.
• PodGroup ResourceClaimTemplate: you want PodGroups to have independent access to separate, similarly-configured devices that can be shared by their Pods. Kubernetes generates one ResourceClaim for the PodGroup from the specification in the ResourceClaimTemplate. The lifetime of each generated ResourceClaim is bound to the lifetime of the corresponding PodGroup. This requires the DRAWorkloadResourceClaims feature to be enabled.

When you define a workload, you can use Common Expression Language (CEL) to filter for specific device attributes or capacity. The available parameters for filtering depend on the device and the drivers.

If you directly reference a specific ResourceClaim in a Pod, that ResourceClaim must already exist in the same namespace as the Pod. If the ResourceClaim doesn't exist in the namespace, the Pod won't schedule. This behavior is similar to how a PersistentVolumeClaim must exist in the same namespace as a Pod that references it.

You can reference an auto-generated ResourceClaim in a Pod, but this isn't recommended because auto-generated ResourceClaims are bound to the lifetime of the Pod or PodGroup that triggered the generation.

To learn how to claim resources using one of these methods, see Allocate Devices to Workloads with DRA.

Prioritized list

You can provide a prioritized list of subrequests for requests in a ResourceClaim or ResourceClaimTemplate. The scheduler will then select the first subrequest that can be allocated. This allows users to specify alternative devices that can be used by the workload if the primary choice is not available.

In the example below, the ResourceClaimTemplate requested a device with the color black and the size large. If a device with those attributes is not available, the pod cannot be scheduled. With the prioritized list feature, a second alternative can be specified, which requests two devices with the color white and size small. The large black device will be allocated if it is available. If it is not, but two small white devices are available, the pod will still be able to run.

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: prioritized-list-claim-template
spec:
  spec:
    devices:
      requests:
      - name: req-0
        firstAvailable:
        - name: large-black
          deviceClassName: resource.example.com
          selectors:
          - cel:
              expression: |-
                device.attributes["resource-driver.example.com"].color == "black" &&
                device.attributes["resource-driver.example.com"].size == "large"
        - name: small-white
          deviceClassName: resource.example.com
          selectors:
          - cel:
              expression: |-
                device.attributes["resource-driver.example.com"].color == "white" &&
                device.attributes["resource-driver.example.com"].size == "small"
          count: 2

If the pod is eligible for multiple nodes in the cluster, the scheduler will use the index of chosen subrequests from any prioritized lists as one of the inputs when it scores each node. So nodes that can allocate devices requested in a higher ranked subrequest are more likely to be chosen than nodes that can only allocate devices for lower ranked subrequests.

The decision is made on a per-Pod basis, so if the Pod is a member of a ReplicaSet or similar grouping, you cannot rely on all the members of the group having the same subrequest chosen. Your workload must be able to accommodate this.

Workload ResourceClaims

When you organize Pods with the Workload API, you can reserve ResourceClaims for entire PodGroups instead of individual Pods and generate ResourceClaimTemplates for a PodGroup instead of a single Pod, allowing the Pods within a PodGroup to share access to devices allocated to the generated ResourceClaim.

This feature targets two problems:

• The ResourceClaim API's status.reservedFor list can only contain 256 items. Since kube-scheduler only records individual Pods in that list, only 256 Pods can share a ResourceClaim. By allowing PodGroups to be recorded in status.reservedFor, many more than 256 Pods can share a ResourceClaim.
• Pods can only share a ResourceClaim when its exact name is known. For complex workloads that replicate groups of Pods, ResourceClaims shared by the Pods in each group need to be created and deleted explicitly when the set of groups scales up and down. By generating ResourceClaims for each PodGroup, a single ResourceClaimTemplate can form the basis for ResourceClaims that are both replicated automatically and shareable among the Pods in a PodGroup.

The PodGroup API defines a spec.resourceClaims field with the same structure and similar meaning as the spec.resourceClaims field in the Pod API:

apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
  name: training-group
  namespace: some-ns
spec:
  ...
  resourceClaims:
  - name: pg-claim
    resourceClaimName: my-pg-claim
  - name: pg-claim-template
    resourceClaimTemplateName: my-pg-template

Like claims made by Pods, claims for PodGroups defining a resourceClaimName refer to a ResourceClaim by name. Claims defining a resourceClaimTemplateName refer to a ResourceClaimTemplate which replicates into one ResourceClaim for the entire PodGroup that can be shared amongst its Pods.

When a Pod defines a claim with a name, resourceClaimName, and resourceClaimTemplateName that all match one of its PodGroup's spec.resourceClaims, then kube-scheduler reserves the ResourceClaim for the PodGroup instead of the Pod. If the Pod's claim does not match one made by its PodGroup, then kube-scheduler reserves the ResourceClaim for the Pod. In either case, reservation is recorded in the ResourceClaim's status.reservedFor. PodGroup reservations and the corresponding resource allocation persist in the ResourceClaim until the PodGroup is deleted, even if the group no longer has any Pods.

When a Pod claim matching a PodGroup claim defines a resourceClaimTemplateName, then one ResourceClaim is generated for the PodGroup. Other Pods in the group defining the same claim will share that generated ResourceClaim instead of prompting a new ResourceClaim to be generated for each Pod. Whether or not a resourceClaimTemplateName claim matches a PodGroup claim, the name of the generated ResourceClaim is recorded in the Pod's status.resourceClaimStatuses.

A matching PodGroup claim for a ResourceClaimTemplate only prompts the creation of a ResourceClaim when the DRAWorkloadResourceClaims feature is enabled. Instead of creating per-Pod ResourceClaims when the feature is disabled, no ResourceClaim is created to prevent creating spurious per-Pod ResourceClaims during cluster upgrades or rollouts/rollbacks of the feature between kube-apiserver and kube-controller-manager.

ResourceClaims generated from a ResourceClaimTemplate for a PodGroup follow the lifecycle of the PodGroup. The ResourceClaim is first created when both the PodGroup and its ResourceClaimTemplate exist. The ResourceClaim is deleted after the PodGroup has been deleted and the ResourceClaim is no longer reserved.

Consider the following example:

apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
  name: training-group
  namespace: some-ns
spec:
  ...
  resourceClaims:
  - name: pg-claim
    resourceClaimName: my-pg-claim
  - name: pg-claim-template
    resourceClaimTemplateName: my-pg-template
---
apiVersion: v1
kind: Pod
metadata:
  name: training-group-pod-1
  namespace: some-ns
spec:
  ...
  schedulingGroup:
    podGroupName: training-group
  resourceClaims:
  - name: pod-claim
    resourceClaimName: my-pod-claim
  - name: pod-claim-template
    resourceClaimTemplateName: my-pod-template
  - name: pg-claim
    resourceClaimName: my-pg-claim
  - name: pg-claim-template
    resourceClaimTemplateName: my-pg-template

In this example, the training-group PodGroup has one Pod named training-group-pod-1. The Pod's pod-claim and pod-claim-template claims do not match any claim made by the PodGroup, so those claims are not affected by the PodGroup: ResourceClaim my-pod-claim becomes reserved for the Pod and a ResourceClaim is generated from ResourceClaimTemplate my-pod-template and also becomes reserved for the Pod. The pg-claim and pg-claim-template do match claims made by the PodGroup. ResourceClaim my-pg-claim becomes reserved for the PodGroup and a ResourceClaim is generated from ResourceClaimTemplate my-pg-template and also becomes reserved for the PodGroup.

Associating ResourceClaims with Workload API resources is controlled by the DRAWorkloadResourceClaims feature gate in the kube-apiserver, kube-controller-manager, kube-scheduler, and kubelet.

ResourceSlice

Each ResourceSlice represents one or more devices in a pool. The pool is managed by a device driver, which creates and manages ResourceSlices. The resources in a pool might be represented by a single ResourceSlice or span multiple ResourceSlices.

ResourceSlices provide useful information to device users and to the scheduler, and are crucial for dynamic resource allocation. Every ResourceSlice must include the following information:

• Resource pool: a group of one or more resources that the driver manages. The pool can span more than one ResourceSlice. Changes to the resources in a pool must be propagated across all of the ResourceSlices in that pool. The device driver that manages the pool is responsible for ensuring that this propagation happens.
• Devices: devices in the managed pool. A ResourceSlice can list every device in a pool or a subset of the devices in a pool. The ResourceSlice defines device information like attributes, versions, and capacity. Device users can select devices for allocation by filtering for device information in ResourceClaims or in DeviceClasses.
• Nodes: the nodes that can access the resources. Drivers can choose which nodes can access the resources, whether that's all of the nodes in the cluster, a single named node, or nodes that have specific node labels.

Drivers use a controller to reconcile ResourceSlices in the cluster with the information that the driver has to publish. This controller overwrites any manual changes, such as cluster users creating or modifying ResourceSlices.

Consider the following example ResourceSlice:

apiVersion: resource.k8s.io/v1
kind: ResourceSlice
metadata:
  name: cat-slice
spec:
  driver: "resource-driver.example.com"
  pool:
    generation: 1
    name: "black-cat-pool"
    resourceSliceCount: 1
  # The allNodes field defines whether any node in the cluster can access the device.
  allNodes: true
  devices:
  - name: "large-black-cat"
    attributes:
      color:
        string: "black"
      size:
        string: "large"
      cat:
        bool: true

This ResourceSlice is managed by the resource-driver.example.com driver in the black-cat-pool pool. The allNodes: true field indicates that any node in the cluster can access the devices. There's one device in the ResourceSlice, named large-black-cat, with the following attributes:

• color: black
• size: large
• cat: true

A DeviceClass could select this ResourceSlice by using these attributes, and a ResourceClaim could filter for specific devices in that DeviceClass.

Naming and prioritization

The order in which the Kubernetes scheduler evaluates devices for allocation is determined by the lexicographical sorting of ResourceSlice and resource pool names. The scheduler uses a first-fit strategy, meaning it selects the first available device that satisfies the claim's requirements.

This allows the priority of resource allocation to be influenced by the names assigned to pools and ResourceSlices. Note that pools without binding conditions are always evaluated before those with binding conditions, regardless of their names.

For drivers built using the k8s.io/dynamic-resources/kubeletplugin Go package or the ResourceSlice controller from that module, these components automatically handle ResourceSlice naming to ensure they are evaluated in the order specified by the driver.

Admin access

You can mark a request in a ResourceClaim or ResourceClaimTemplate as having privileged features for maintenance and troubleshooting tasks. A request with admin access grants access to in-use devices and may enable additional permissions when making the device available in a container:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: large-black-cat-claim-template
spec:
  spec:
    devices:
      requests:
      - name: req-0
        exactly:
          deviceClassName: resource.example.com
          allocationMode: All
          adminAccess: true

Admin access is a privileged mode and should not be granted to regular users in multi-tenant clusters. Only users authorized to create ResourceClaim or ResourceClaimTemplate objects in namespaces labeled with resource.kubernetes.io/admin-access: "true" (case-sensitive) can use the adminAccess field. This ensures that non-admin users cannot misuse the feature.

Admin access is controlled by the DRAAdminAccess feature gate in the kube-apiserver, kube-scheduler, and kubelet.

List type attributes

This feature improves the ResourceSlice API, allowing DRA drivers to specify list values for device attributes instead of only scalars. This is useful for modeling more complex internal node topologies, for example when a CPU has adjacency to multiple PCIe roots.

For ResourceClaim authors (end users), this means that the matchAttribute and distinctAttribute work better for these cases.

• matchAttribute — the two attributes must have a non-empty list intersection, rather than be identical (scalar values are treated as single-item lists). This just means that if one driver publishes a single value for, say, the PCIe root, and another driver publishes a list, the constraint is met as long as the single value appears somewhere in the list.
• distinctAttribute — the attribute values must be pairwise-disjoint (no value shared between any two devices)

To help ResourceClaim authors use attributes that may be lists inside CEL expressions, this feature also introduces an includes() CEL function.

# Scalar attribute (backward compatible)
# assume: device.attributes["dra.example.com"].model = "model-a"
device.attributes["dra.example.com"].model.includes("model-a")  # true
device.attributes["dra.example.com"].model.includes("model-b")  # false

# List-type attribute (requires DRAListTypeAttributes)
# assume: device.attributes["dra.example.com"].supported-models= ["model-a", "model-b"]
device.attributes["dra.example.com"].supported-models.includes("model-a")  # true
device.attributes["dra.example.com"].supported-models.includes("model-c")  # false

Details for DRA Driver Authors

By default, each DeviceAttribute holds exactly one scalar value: a boolean, an integer, a string, or a semantic version string. The DRAListTypeAttributes feature gate extends DeviceAttribute with four list-type fields, allowing a device to advertise multiple values for a single attribute:

• bools — a list of boolean values
• ints — a list of 64-bit integer values
• strings — a list of strings (each at most 64 characters)
• versions — a list of semantic version strings per semver.org spec 2.0.0 (each at most 64 characters)

The total number of individual attribute values per device (scalar fields plus all list elements combined) is limited to 48. When any device in a ResourceSlice uses this feature or other advanced features such as taints, the ResourceSlice will be limited to at most 64 devices. use list-type attributes or other advanced features such as taints.

Here is an example of a device advertising multiple supported models using a list-type string attribute:

kind: ResourceSlice
apiVersion: resource.k8s.io/v1
metadata:
  name: example-resourceslice
spec:
  nodeName: worker-1
  pool:
    name: pool
    generation: 1
    resourceSliceCount: 1
  driver: dra.example.com
  devices:
  - name: gpu-0
    attributes:
      dra.example.com/supported-models:
        strings:
        - model-a
        - model-b

List type attributes is controlled by the DRAListTypeAttributes feature gate in the kube-apiserver and kube-scheduler.

Derived attributes

The matchAttribute and distinctAttribute constraints normally require devices to publish attributes under the exact same name. If a GPU driver publishes pcie_locality and a NIC driver publishes pcie_root (or embeds the same information in a string like numa0-pcie1), the scheduler has no way to recognize that these represent the same thing, so devices from the two drivers can't be co-located without first agreeing on a shared attribute name.

derivedAttributes lets you bridge this gap inline, without waiting for drivers to standardize on shared attribute names. Add one or more derivedAttributes entries to a request, under .spec.devices.requests[].exactly or .spec.devices.requests[].firstAvailable[]. Each entry defines a CEL expression that the scheduler evaluates against every candidate device for that request. The result becomes a virtual attribute that can be referenced from a matchAttribute or distinctAttribute constraint exactly like a driver-provided device attribute.

apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: gpu-nic-numa-alignment
spec:
  devices:
    requests:
    - name: gpu
      exactly:
        deviceClassName: gpu.example.com
        count: 1
        derivedAttributes:
        - name: derived/numa
          expression: device.attributes["gpu.example.com"].numa
    - name: nic
      exactly:
        deviceClassName: nic.example.com
        count: 1
        derivedAttributes:
        - name: derived/numa
          expression: device.attributes["nic.example.com"].numaNode
    constraints:
    - requests: ["gpu", "nic"]
      matchAttribute: derived/numa

In this example, the gpu and nic drivers publish topology information under different attribute names (numa and numaNode). Each request computes a common derived/numa value from its own device's attributes, and the matchAttribute constraint aligns the two requests on that virtual attribute, even though the underlying drivers never agreed on a shared attribute name.

A few things to know about derivedAttributes:

• Naming: name must be a DNS subdomain followed by a / and a C identifier, the same format used for driver-provided attribute names (for example, example.com/numaNode or derived/numaNode). If the name matches a driver-provided attribute already published by the driver, the derived attribute's value shadows the driver-provided one for constraint matching. Use a domain prefix that no driver would use, such as derived/, if you want to avoid shadowing unintentionally. You can define up to 32 derived attributes per request.
• Must be used by a constraint: every derived attribute must be referenced by at least one matchAttribute or distinctAttribute constraint that applies to the request (or subrequest) that defines it. Otherwise the ResourceClaim fails validation.
• Evaluation scope and order: expression is evaluated once per candidate device, after the request's own CEL selectors (.selectors[].cel) have already filtered that device. As a result, derived attributes can't be referenced from selector expressions, and aren't exposed through device.attributes in the CEL environment.
• Return type: expression must evaluate to a scalar (string, int, bool, or a semantic version) or, when the DRAListTypeAttributes feature gate is also enabled, a list of one of those scalar types.
• Cost limits: each expression has a maximum length and a limit on its estimated CEL evaluation cost. On top of that, the combined estimated cost of all derivedAttributes expressions in a ResourceClaim is also capped, to bound the total overhead added to a single scheduling attempt. The ResourceClaim is rejected if any of these limits are exceeded.
• Runtime errors abort scheduling: if evaluating an expression fails for a candidate device, for example because it references an attribute that device doesn't have, the scheduler aborts allocation and the Pod fails to schedule, rather than silently skipping that device. Write expressions defensively, for example by checking that an attribute exists before reading it.

Derived attributes is controlled by the DRADerivedAttributes feature gate in the kube-apiserver and kube-scheduler.

For a list of standard device attributes that DRA drivers can publish, see the Standard Device Attributes reference.

Extended resource allocation by DRA

You can provide an extended resource name for a DeviceClass. The scheduler will then select the devices matching the class for the extended resource requests. This allows users to continue using extended resource requests in a pod to request either extended resources provided by device plugin, or DRA devices. The same extended resource can be provided either by device plugin, or DRA on one single cluster node. The same extended resource can be provided by device plugin on some nodes, and DRA on other nodes in the same cluster.

In the example below, the DeviceClass is given an extendedResourceName example.com/gpu. If a pod requested for the extended resource example.com/gpu: 2, it can be scheduled to a node with two or more devices matching the DeviceClass.

apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: gpu.example.com
spec:
  selectors:
  - cel:
      expression: device.driver == 'gpu.example.com' && device.attributes['gpu.example.com'].type
        == 'gpu'
  extendedResourceName: example.com/gpu

In addition, users can use a special extended resource to allocate devices without having to explicitly create a ResourceClaim. Using the extended resource name prefix deviceclass.resource.kubernetes.io/ and the DeviceClass name. This works for any DeviceClass, even if it does not specify an extended resource name. The resulting ResourceClaim will contain a request for an ExactCount of the specified number of devices of that DeviceClass.

Extended resource allocation by DRA is controlled by the DRAExtendedResource feature gate in the kube-apiserver, kube-scheduler, kube-controller-manager, and kubelet.

For a hands-on walkthrough of requesting extended resources, see Assign Extended Resources to a Container.

2.2 - How DRA Works

This page describes how Kubernetes allocates devices to workloads with dynamic resource allocation (DRA), and how pre-scheduled Pods interact with the process.

How resource allocation with DRA works

The following sections describe the workflow for the various types of DRA users and for the Kubernetes system during dynamic resource allocation.

Workflow for users

1. Driver creation: device owners or third-party entities create drivers that can create and manage ResourceSlices in the cluster. These drivers optionally also create DeviceClasses that define a category of devices and how to request them.
2. Cluster configuration: cluster admins create clusters, attach devices to nodes, and install the DRA device drivers. Cluster admins optionally create DeviceClasses that define categories of devices and how to request them.
3. Resource claims: workload operators create ResourceClaimTemplates or ResourceClaims that request specific device configurations within a DeviceClass. In the same step, workload operators modify their Kubernetes manifests to request those ResourceClaimTemplates or ResourceClaims.

Workflow for Kubernetes

1. ResourceSlice creation: drivers in the cluster create ResourceSlices that represent one or more devices in a managed pool of similar devices.

2. Workload creation: the cluster control plane checks new workloads for references to ResourceClaimTemplates or to specific ResourceClaims.

   • If the workload uses a ResourceClaimTemplate, a controller named the resourceclaim-controller generates ResourceClaims for the workload.
   • If the workload uses a specific ResourceClaim, Kubernetes checks whether that ResourceClaim exists in the cluster. If the ResourceClaim doesn't exist, the Pods won't deploy.

3. ResourceSlice filtering: for every Pod, Kubernetes checks the ResourceSlices in the cluster to find a device that satisfies all of the following criteria:

   • The nodes that can access the resources are eligible to run the Pod.
   • The ResourceSlice has unallocated resources that match the requirements of the Pod's ResourceClaim.

4. Resource allocation: after finding an eligible ResourceSlice for a Pod's ResourceClaim, the Kubernetes scheduler updates the ResourceClaim with the allocation details. The scheduler uses a first-fit strategy and evaluates pools and ResourceSlices in lexicographical order by their names. Drivers can prioritize specific slices or pools by naming them appropriately. For details, see Naming and prioritization.

5. Pod scheduling: when resource allocation is complete, the scheduler places the Pod on a node that can access the allocated resource. The device driver and the kubelet on that node coordinate via gRPC to configure the device and the Pod's access to the device, unless the driver declared optional node operations for devices that do not require node-local preparation or cleanup.

Pre-scheduled Pods

When you - or another API client - create a Pod with spec.nodeName already set, the scheduler gets bypassed. If some ResourceClaim needed by that Pod does not exist yet, is not allocated or not reserved for the Pod, then the kubelet will fail to run the Pod and re-check periodically because those requirements might still get fulfilled later.

Such a situation can also arise when support for dynamic resource allocation was not enabled in the scheduler at the time when the Pod got scheduled (version skew, configuration, feature gate, etc.). kube-controller-manager detects this and tries to make the Pod runnable by reserving the required ResourceClaims. However, this only works if those were allocated by the scheduler for some other pod.

It is better to avoid bypassing the scheduler because a Pod that is assigned to a node blocks normal resources (RAM, CPU) that then cannot be used for other Pods while the Pod is stuck. To make a Pod run on a specific node while still going through the normal scheduling flow, create the Pod with a node selector that exactly matches the desired node:

apiVersion: v1
kind: Pod
metadata:
  name: pod-with-cats
spec:
  nodeSelector:
    kubernetes.io/hostname: name-of-the-intended-node
  ...

You may also be able to mutate the incoming Pod, at admission time, to unset the .spec.nodeName field and to use a node selector instead.

Device binding conditions

Device Binding Conditions allow the Kubernetes scheduler to delay Pod binding until external resources, such as fabric-attached GPUs or reprogrammable FPGAs, are confirmed to be ready.

This waiting behavior is implemented in the PreBind phase of the scheduling framework. During this phase, the scheduler checks whether all required device conditions are satisfied before proceeding with binding.

This improves scheduling reliability by avoiding premature binding and enables coordination with external device controllers.

To use this feature, device drivers (typically managed by driver owners) must publish the following fields in the Device section of a ResourceSlice. Cluster administrators must enable the DRADeviceBindingConditions and DRAResourceClaimDeviceStatus feature gates for the scheduler to honor these fields.

bindingConditions

A list of condition types that must be set to True (in the .status.conditions field of the associated ResourceClaim) before the Pod can be bound. These conditions typically represent readiness signals, such as DeviceAttached or DeviceInitialized.

bindingFailureConditions

A list of condition types that, if set to True in status.conditions field of the associated ResourceClaim, indicate a failure state. If any of these conditions are True, the scheduler will abort binding and reschedule the Pod.

bindsToNode

if set to true, the scheduler records the selected node name in the status.allocation.nodeSelector field of the ResourceClaim. This does not affect the Pod's spec.nodeSelector. Instead, it sets a node selector inside the ResourceClaim, which external controllers can use to perform node-specific operations such as device attachment or preparation.

All condition types listed in bindingConditions and bindingFailureConditions are evaluated from the status.conditions field of the ResourceClaim. External controllers are responsible for updating these conditions using standard Kubernetes condition semantics (type, status, reason, message, lastTransitionTime).

The scheduler waits up to 600 seconds (default) for all bindingConditions to become True. If the timeout is reached or any bindingFailureConditions are True, the scheduler clears the allocation and reschedules the Pod. A cluster administration can configure this timeout duration by editing the kube-scheduler configuration file.

An example of configuring this timeout in KubeSchedulerConfiguration is given below:

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: default-scheduler
  pluginConfig:
  - name: DynamicResources
    args:
      apiVersion: kubescheduler.config.k8s.io/v1
      kind: DynamicResourcesArgs
      bindingTimeout: 60s

Example

Here is an example of a ResourceSlice that you might see in a cluster where there's a DRA driver in use, and that driver supports binding conditions:

apiVersion: resource.k8s.io/v1
kind: ResourceSlice
metadata:
  name: gpu-slice-1
spec:
  driver: dra.example.com
  nodeSelector:
    nodeSelectorTerms:
    - matchExpressions:
      - key: accelerator-type
        operator: In
        values:
        - "high-performance"
  pool:
    name: gpu-pool
    generation: 1
    resourceSliceCount: 1
  devices:
    - name: gpu-1
      attributes:
        vendor:
          string: "example"
        model:
          string: "example-gpu"
      bindsToNode: true
      bindingConditions:
        - dra.example.com/is-prepared
      bindingFailureConditions:
        - dra.example.com/preparing-failed

This example ResourceSlice has the following properties:

• The ResourceSlice targets nodes labeled with accelerator-type=high-performance, so that the scheduler uses only a specific set of eligible nodes.
• The scheduler selects one node from the selected group (for example, node-3) and sets the status.allocation.nodeSelector field in the ResourceClaim to that node name.
• The dra.example.com/is-prepared binding condition indicates that the device gpu-1 must be prepared (the is-prepared condition has a status of True) before binding.
• If the gpu-1 device preparation fails (the preparing-failed condition has a status of True), the scheduler aborts binding.
• The scheduler waits up to 600 seconds (default) for the device to become ready.
• External controllers can use the node selector in the ResourceClaim to perform node-specific setup on the selected node.

Device binding conditions is controlled by the DRADeviceBindingConditions feature gate in the kube-apiserver and kube-scheduler.

Node allocatable resources

Devices managed by DRA can have an underlying footprint composed of node allocatable resources, such as cpu, memory, or hugepages. This feature integrates these DRA-based requests into the scheduler's standard accounting alongside regular Pod spec requests for these resources.

DRA drivers define how devices consume node allocatable resources using two distinct models:

• Direct Resource Mapping (mapping): The DRA device directly provides a standard node resource (such as a custom CPU core pool or memory block). The claim allocation directly maps to standard CPU or memory capacity on the node.
• Auxiliary Device Overhead (overhead): The DRA device (such as a GPU or accelerator) requires host resources (such as host RAM) as secondary overhead to operate when allocated to a Pod or container.

Considerations for Pod Authors

When authoring a PodSpec using claims for these types of devices, there are a few things to be aware of:

• When Pod-level resources are used, the scheduler strictly validates them against both container requests and limits:
  • The sum of all container requests and DRA claim resources must not exceed the Pod-level requests; otherwise, the Pod will fail to schedule.
  • Each individual container's limit plus its DRA allocations must not exceed the Pod-level limits; otherwise, the Pod will fail to schedule.
• A container's total resource requirement is the sum of its container-level resources and any node allocatable resources from its associated resource claims.
• Claim Sharing Restriction: Claims that use direct resource mappings (mapping) cannot be shared across multiple Pods. Claims for devices with overhead can support device sharing and overhead is tracked per Pod or per container.
• Pods with DRA claims support in-place resizing for standard requests in spec. The scheduler ensures that resized standard requests combined with static DRA allocations still fit on the node.

Details for DRA Driver Authors

DRA drivers declare this node allocatable resource footprint using the nodeAllocatableResources field on devices within a ResourceSlice. This defines the translation of the requested DRA device or capacity into standard resources that are tracked in the node's status.allocatable (note that extended resources are not supported for this field). This is useful both for drivers that directly expose native resources (like a CPU or Memory DRA driver) and for devices that require auxiliary node dependencies (like an accelerator that needs host memory).

The nodeAllocatableResources field supports two different use cases:

• Mapping: Used when the DRA device directly represents the standard resource (e.g., a CPU or Memory DRA driver). The scheduler calculates the exact quantity by scaling the capacity using capacityMultiplier, or scaling the device count using deviceMultiplier.
• Overhead: Used when the device requires auxiliary node dependencies (e.g., host memory consumed by a GPU). This can be defined as a flat perPod cost or a variable perContainer cost that scales linearly with the number of referencing containers.

Example: CPU DRA Driver (Mapping)

Here is an example where a CPU DRA driver exposes a CPU socket as a pool of 128 CPUs using DRA consumable capacity. The capacityKey links the consumed cpu.example.com/cpu capacity directly to the node's standard cpu allocatable resource:

apiVersion: resource.k8s.io/v1
kind: ResourceSlice
metadata:
  name: my-node-cpus
spec:
  driver: cpu.example.com
  nodeName: my-node
  pool:
    name: socket-cpus
    generation: 1
    resourceSliceCount: 1
  devices:
  - name: socket0cpus
    allowMultipleAllocations: true
    capacity:
      "cpu.example.com/cpu": "128"
    nodeAllocatableResources:
      mapping:
        cpu:
          capacityKey: "cpu.example.com/cpu"
  - name: socket1cpus
    allowMultipleAllocations: true
    capacity:
      "cpu.example.com/cpu": "128"
    nodeAllocatableResources:
      mapping:
        cpu:
          capacityKey: "cpu.example.com/cpu"
          capacityMultiplier: 1

Example: Accelerator with Auxiliary Resources (Overhead)

Here is an example of a resource slice where an accelerator requires an additional 8Gi of memory per Pod to function:

apiVersion: resource.k8s.io/v1
kind: ResourceSlice
metadata:
  name: my-node-xpus
spec:
  driver: xpu.example.com
  nodeName: my-node
  pool:
    name: xpu-pool
    generation: 1
    resourceSliceCount: 1
  devices:
  - name: xpu-model-x-001
    attributes:
      example.com/model:
        string: "model-x"
    nodeAllocatableResources:
      overhead:
        memory:
          perPod: "8Gi"

After a Pod is successfully bound to the node, the exact quantities of node allocatable resources allocated via DRA are aggregated by the kube-scheduler and embedded directly into the Pod's status.nodeAllocatableResourceClaimStatuses field. This provides a clear, persistent handoff from the scheduler to the kubelet.

Crucially, the kubelet natively consumes this API to perfectly align system-level boundaries:

• cgroups: Pod and container cgroups would now include DRA based allocations, preventing workloads from being artificially throttled by the kernel.
• OOM Scores: The kubelet factors the container's DRA memory requests into its effective memory request.

Node allocatable resources is an alpha feature and is enabled when the DRANodeAllocatableResources feature gate is enabled in the kube-apiserver, kube-scheduler, and kubelet.

2.3 - Observability of Dynamic Resources

This page describes how to observe the status and health of resources that are dynamically allocated with DRA.

Observability of dynamic resources

You can check the status of dynamically allocated resources by using any of the following methods:

• kubelet device metrics
• ResourceClaim status
• Device health monitoring

kubelet device metrics

The PodResourcesLister kubelet gRPC service lets you monitor in-use devices. The DynamicResource message provides information that's specific to dynamic resource allocation, such as the device name and the claim name. For details, see Monitoring device plugin resources.

ResourceClaim device status

DRA drivers can report driver-specific device status data for each allocated device in the status.devices field of a ResourceClaim. For example, the driver might list the IP addresses that are assigned to a network interface device. Updating this field requires specific synthetic RBAC permissions, see Hardening Guide - Dynamic Resource Allocation and Harden Dynamic Resource Allocation in Your Cluster.

The accuracy of the information that a driver adds to a ResourceClaim status.devices field depends on the driver. Evaluate drivers to decide whether you can rely on this field as the only source of device information.

If you disable the DRAResourceClaimDeviceStatus feature gate, the status.devices field automatically gets cleared when storing the ResourceClaim. A ResourceClaim device status is supported when it is possible, from a DRA driver, to update an existing ResourceClaim where the status.devices field is set.

In the following example, the status.devices field of a ResourceClaim has been populated by the driver (resource-driver.example.com) responsible for managing the allocated device:

apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: macvlan-eth0
spec:
...
status:
  allocation:
    devices:
      results:
      - device: eth0
        driver: resource-driver.example.com
        pool: nic-worker-a
        request: macvlan-eth0
        shareID: 8e7acdf9-0290-4ecd-a801-a654b021d2b7
        consumedCapacity:
          resource-driver.example.com/bandwidth: 1G
  devices:
  - conditions:
    - lastTransitionTime: "2025-10-21T08:38:17Z"
      message: Device successfully allocated and assigned to the pod
      reason: NetworkReady
      status: "True"
      type: NetworkReady
    device: eth0
    driver: resource-driver.example.com
    networkData:
      hardwareAddress: 00:01:ec:84:fb:51
      interfaceName: net1
      ips:
      - 10.10.1.2/24
      - 2001:db8::1/64
    pool: nic-worker-a
    shareID: 8e7acdf9-0290-4ecd-a801-a654b021d2b7

If a device has not been allocated, a driver's request to update the status.devices field of the ResourceClaim with that device is rejected. When a device is deallocated (removed from status.allocation.devices), the corresponding entry in status.devices is automatically removed.

For details about the status.devices field, see the ResourceClaim API reference.

Device Health Monitoring

Kubernetes provides a mechanism for monitoring and reporting the health of dynamically allocated infrastructure resources. For stateful applications running on specialized hardware, it is critical to know when a device has failed or become unhealthy. It is also helpful to find out if the device recovers.

To use this functionality, the ResourceHealthStatus feature gate must be enabled (beta and enabled by default since v1.36), and the DRA driver must implement the DRAResourceHealth gRPC service.

When a DRA driver detects that an allocated device has become unhealthy, it reports this status back to the kubelet. This health information is then exposed directly in the Pod's status. The kubelet populates the allocatedResourcesStatus field in the status of each container, detailing the health of each device assigned to that container. Each resource health entry can include an optional message field with additional human-readable context about the health status, such as error details or failure reasons.

If the kubelet does not receive a health update from a DRA driver within a timeout period, the device's health status is marked as "Unknown". DRA drivers can configure this timeout on a per-device basis by setting the health_check_timeout_seconds field in the DeviceHealth gRPC message. If not specified, the kubelet uses a default timeout of 30 seconds. This allows different hardware types (for example, GPUs, FPGAs, or storage devices) to use appropriate timeout values based on their health-reporting characteristics.

This provides crucial visibility for users and controllers to react to hardware failures. For a Pod that is failing, you can inspect this status to determine if the failure was related to an unhealthy device.

Resource pool status

You can query the availability of devices in resource pools using the ResourcePoolStatusRequest API. This provides visibility into how many devices are available, allocated, or unavailable across your cluster's DRA resource pools.

To check resource pool status:

1. Create a ResourcePoolStatusRequest specifying the driver name (required) and optionally a limit on the number of pools returned. You can also limit it to a single pool by specifying a pool name:

   apiVersion: resource.k8s.io/v1alpha3
   kind: ResourcePoolStatusRequest
   metadata:
     name: check-gpus
   spec:
     driver: example.com/gpu
     # Optional: filter to a specific pool
     # poolName: my-pool
     # Optional: limit number of pools returned (default: 100, max: 1000)
     # limit: 10
   

2. Wait for the controller to process the request:

   kubectl wait --for=condition=Complete resourcepoolstatusrequest/check-gpus --timeout=30s
   

3. Read the status to see pool availability:

   kubectl get resourcepoolstatusrequest/check-gpus -o yaml
   

   The status includes:

   • poolCount: total number of pools matching the filter (may exceed the number of pools listed if truncated by the limit).
   • pools: a list of pool details, each containing:
     • driver and poolName: identify the pool.
     • generation: the latest pool generation observed across ResourceSlices.
     • resourceSliceCount: the number of ResourceSlices making up the pool.
     • totalDevices: total devices in the pool.
     • allocatedDevices: devices currently allocated to claims.
     • availableDevices: devices available for allocation (totalDevices - allocatedDevices - unavailableDevices).
     • unavailableDevices: devices not available due to taints or other conditions.
     • nodeName: the node associated with the pool, if any.
     • validationError: set when the pool's data could not be fully validated (for example, during a generation rollout). When set, device count fields may be unset.
     • partitionSummary: for partitionable pools, per-partition-type allocatability (see Partition summary).
     • shareableSummary: for pools with shareable devices, aggregate capacity usage (see Shareable summary).
   • conditions: includes Complete (success) or Failed (error) condition types.

4. Delete the request when done:

   kubectl delete resourcepoolstatusrequest/check-gpus
   

ResourcePoolStatusRequest objects are processed once by a controller in kube-controller-manager. The spec is immutable once created, and the entire object becomes immutable once the status is populated. To get updated availability data, delete and recreate the request. Completed requests are automatically cleaned up after 1 hour.

This feature requires explicit RBAC permissions on the ResourcePoolStatusRequest resource. No default ClusterRoles include this permission.

Resource pool status is controlled by the DRAResourcePoolStatus feature gate in the kube-apiserver and kube-controller-manager.

Partition summary

A single physical device such as a GPU may be advertised as several partition types (for example, a full GPU versus a half-sized MIG slice) that draw from the same shared counters. Because these partitions compete for the same underlying capacity, a plain device count does not tell you how many of each type can still be allocated. For partitionable pools, the partitionSummary view answers that question. For each partition type it reports:

• attribute: the fully qualified name of the device attribute whose value groups this entry. It is the ResourceSlice's spec.partitionTypeAttribute, or the request's spec.defaultPartitionTypeAttribute when the slice declares none.
• type: the value of that attribute on the device (for example, Full or Half).
• total: the number of devices of this partition type in the pool.
• allocatable: how many additional devices of this partition type could still be allocated given current shared-counter consumption.

The named attribute must be a string attribute. If a partitionable device's partition-type attribute is missing or is not a string (for example, an integer, boolean, or version value), the pool reports a validation error instead of a partition summary. There is no special handling for list-type attributes; a non-string attribute is simply not a valid partition-type attribute.

To produce this view, the driver labels each partitionable device with a string attribute whose value names the partition type, and names that attribute in the ResourceSlice's partitionTypeAttribute field:

apiVersion: resource.k8s.io/v1
kind: ResourceSlice
# ...
spec:
  # Every partitionable device in this slice carries this attribute; devices
  # that share a value share the same shared-counter cost.
  partitionTypeAttribute: gpu.example.com/profile

If a driver has not yet been updated to declare partitionTypeAttribute, a request can still obtain a partition summary by naming a fallback attribute in its spec. A slice's own partitionTypeAttribute always takes precedence; the request-level default applies only to devices whose slice does not declare one:

apiVersion: resource.k8s.io/v1alpha3
kind: ResourcePoolStatusRequest
metadata:
  name: check-gpu-partitions
spec:
  driver: gpu.example.com
  # Fallback grouping attribute for slices that don't declare one themselves.
  defaultPartitionTypeAttribute: gpu.example.com/profile

When neither the slice nor the request names an attribute, a partitionable pool reports no partitionSummary.

The partitionSummary view is controlled by the DRAPartitionableDevicesType feature gate in the kube-apiserver and kube-controller-manager, which in turn requires the DRAResourcePoolStatus and DRAPartitionableDevices feature gates to be enabled.

Shareable summary

For pools that contain shareable devices (devices that set allowMultipleAllocations and can be consumed by multiple claims), shareableSummary reports aggregate capacity usage across the pool:

• fullyAvailableDevices: shareable devices with no capacity consumed.
• partiallyAvailableDevices: shareable devices with some, but not all, capacity consumed.
• capacity: per capacity name, the aggregate total, consumed, and available (total minus consumed, never negative) amounts across the pool.

The shareableSummary is populated only when at least one device in the pool is shareable. It is part of the resource pool status feature (the DRAResourcePoolStatus feature gate) and does not require DRAPartitionableDevicesType; the shareable devices it summarizes come from the consumable capacity feature.

DRA device metadata in containers

DRA drivers can expose device metadata such as device attributes (PCI bus addresses or mediated device UUIDs) and network configuration directly to containers as JSON files. This lets applications discover information about allocated devices without querying the Kubernetes API or using custom controllers.

KEP-5304 defines a device metadata protocol that drivers must follow so that applications see a consistent layout across drivers and clusters. The DRA kubelet plugin library implements this protocol.

Device metadata follows the same rules as device access: it is available inside a container only when that container requests the device. For details, see Request devices in workloads using DRA.

Device metadata protocol

The protocol consists of four rules:

1. File paths. Metadata files live inside containers under /var/run/kubernetes.io/dra-device-attributes. For a directly referenced ResourceClaim, the path is resourceclaims/<claimName>/<requestName>/<driverName>-metadata.json. For a claim created from a ResourceClaimTemplate, the path is resourceclaimtemplates/<podClaimName>/<requestName>/<driverName>-metadata.json, where podClaimName is pod.spec.resourceClaims[].name.

   When a request uses a prioritized list, only the top-level request name is used for the <requestName> path segment. The requests[].name field in the file contains the full <request>/<subrequest> reference, such as gpu/high-memory.

   The path constants are defined in k8s.io/dynamic-resource-allocation/api/metadata.

2. JSON API. Each file is a stream of one or more DeviceMetadata objects. Each object has apiVersion and kind, following Kubernetes API conventions. The same metadata is encoded once per configured API version in the order selected by the driver. Consumers use the first version that they can decode and skip unknown versions. A malformed object in a known version is an error.

3. Generation. The initial file has metadata.generation set to 1. Each update increments the generation so that consumers can detect changes.

4. Container exposure. The DRA kubelet plugin library uses CDI to bind-mount each file read-only. Other implementations can use a different mechanism as long as the file appears at the required path and is read-only.

Enable device metadata in a driver

Device metadata is a driver-side feature. It has no Kubernetes feature gate and is disabled by default in the DRA kubelet plugin library. A driver must enable the feature and explicitly select the versions that it writes:

kubeletplugin.EnableDeviceMetadata(true, []schema.GroupVersion{
	metadatav1beta1.SchemeGroupVersion,
	metadatav1alpha1.SchemeGroupVersion,
})

The v1beta1 version is required. A driver can also write v1alpha1 for compatibility with older consumers. The order in the slice is the order in the metadata stream; the framework does not sort the versions. Drivers should put the newest version first. Enabling device metadata with no versions, without v1beta1, or with an unknown version causes the plugin to fail during startup.

For each prepared device, the driver can populate Device.Metadata with kubeletplugin.DeviceMetadata. Drivers should include the attributes that they publish for that device in its ResourceSlice, so workloads see the same information at runtime. Drivers can also include attributes that are only relevant at runtime. For network devices, drivers can add interface names, IP addresses, and hardware addresses after CNI configuration by calling UpdateRequestMetadata.

The kubelet plugin API links above describe integration for driver authors. The DRA framework does not define a universal command-line flag, so cluster operators enable the feature through the deployment configuration provided by their driver.

When enabled, the DRA kubelet plugin library writes metadata files while preparing allocated devices. It also writes CDI specifications to /var/run/cdi by default. The container runtime must be configured to discover CDI specifications from that directory. The library determines the minimum CDI specification version required for each generated specification.

When one request allocates devices from multiple DRA drivers, each driver writes its own metadata file. Consumers that know the driver name should construct the exact path from the claim, request, and driver names. Go consumers can use ReadResourceClaimMetadata or ReadResourceClaimTemplateMetadata to read and merge all per-driver files for a request.

Metadata schema

Each object in a metadata file conforms to the DeviceMetadata API (metadata.resource.k8s.io/v1beta1).

The schema contains:

• Standard object metadata for the ResourceClaim, including its name, namespace, UID, and metadata generation.
• The optional podClaimName for a claim generated from a ResourceClaimTemplate.
• A list of requests. Each request has a required name and a list of allocated devices.
• The driver, pool, and name for each device.
• Optional device attributes and network data.

Attribute values use the same representation as ResourceSlice device attributes. Each attribute has exactly one scalar value (int, bool, string, or version) or list value (ints, bools, strings, or versions). Device capacity values are not included in device metadata.

Network data can contain interfaceName, ips, and hardwareAddress. For field constraints, see the DeviceMetadata API documentation.

The following example shows one object in a metadata stream for a GPU device allocated through a ResourceClaimTemplate:

{
  "kind": "DeviceMetadata",
  "apiVersion": "metadata.resource.k8s.io/v1beta1",
  "metadata": {
    "name": "pod0-gpu-2kqrd",
    "namespace": "gpu-test1",
    "uid": "c7e7b22e-239b-4498-b27c-7f1344481e14",
    "generation": 1
  },
  "podClaimName": "gpu",
  "requests": [
    {
      "name": "gpu",
      "devices": [
        {
          "driver": "gpu.example.com",
          "pool": "worker-0",
          "name": "gpu-0",
          "attributes": {
            "driverVersion": {
              "version": "1.0.0"
            },
            "index": {
              "int": 0
            },
            "model": {
              "string": "LATEST-GPU-MODEL"
            },
            "uuid": {
              "string": "gpu-18db0e85-99e9-c746-8531-ffeb86328b39"
            }
          }
        }
      ]
    }
  ]
}

The DRA kubelet plugin does not validate metadata before writing it. Go consumers can opt in to generated validation when decoding a stream. Decoding and validation have separate results: a validation error does not prevent a successfully decoded object from being returned. For usage, see Access DRA device metadata.

Immediate and deferred metadata

For immediate metadata, the driver supplies attributes or network data while it prepares the claim. The DRA kubelet plugin writes the file with generation 1 before the consuming container starts.

For deferred metadata, the driver can prepare a device without attributes or network data. The initial generation 1 file contains the device identity. The driver later calls UpdateRequestMetadata to replace the complete stream atomically and increment the generation. An update requires the initial file to exist. If device preparation returns no devices for a request, the framework creates neither a metadata file nor a metadata CDI device for that request.

Metadata remains available to each consuming container for the lifetime of that container. The framework removes the metadata files and CDI specifications after the claim is unprepared.

To learn how to use device metadata in your workloads, see Access DRA device metadata.

Custom drivers

Custom drivers that do not use the DRA kubelet plugin library must implement the device metadata protocol themselves. This includes writing the versioned DeviceMetadata stream at the correct paths, incrementing metadata.generation on every update, and exposing files read-only through CDI or an equivalent mechanism.

2.4 - DRA Features

This page describes optional DRA features for advanced use cases. Some of these features require support from the DRA driver. Each feature notes its maturity and the feature gate that enables it.

Partitionable devices

Devices represented in DRA don't necessarily have to be a single unit connected to a single machine, but can also be a logical device comprised of multiple devices connected to multiple machines. These devices might consume overlapping resources of the underlying phyical devices, meaning that when one logical device is allocated other devices will no longer be available.

In the ResourceSlice API, this is represented as a list of named CounterSets, each of which contains a set of named counters. The counters represent the resources available on the physical device that are used by the logical devices advertised through DRA.

Logical devices can specify the ConsumesCounters list. Each entry contains a reference to a CounterSet and a set of named counters with the amounts they will consume. So for a device to be allocatable, the referenced counter sets must have sufficient quantity for the counters referenced by the device.

CounterSets must be specified in separate ResourceSlices from devices. Devices can consume counters from any CounterSet defined in the same resource pool as the device.

Here is an example of two devices, each consuming 6Gi of memory from a shared counter with 8Gi of memory. Thus, only one of the devices can be allocated at any point in time. The scheduler handles this and it is transparent to the consumer as the ResourceClaim API is not affected.

apiVersion: resource.k8s.io/v1
kind: ResourceSlice
metadata:
  name: resourceslice-with-countersets
spec:
  nodeName: worker-1
  pool:
    name: pool
    generation: 1
    resourceSliceCount: 2
  driver: dra.example.com
  sharedCounters:
  - name: gpu-1-counters
    counters:
      memory:
        value: 8Gi
---
apiVersion: resource.k8s.io/v1
kind: ResourceSlice
metadata:
  name: resourceslice-with-devices
spec:
  nodeName: worker-1
  pool:
    name: pool
    generation: 1
    resourceSliceCount: 2
  driver: dra.example.com
  devices:
  - name: device-1
    consumesCounters:
    - counterSet: gpu-1-counters
      counters:
        memory:
          value: 6Gi
  - name: device-2
    consumesCounters:
    - counterSet: gpu-1-counters
      counters:
        memory:
          value: 6Gi

Partitionable devices is controlled by the DRAPartitionableDevices feature gate in the kube-apiserver and kube-scheduler.

Device compatibility groups

Device compatibility groups let a DRA driver declare which partitioned devices can be co-allocated on the same physical hardware. Without this feature, incompatible device combinations are only detected when the kubelet prepares the Pod on a node — resulting in a failed preparation. With compatibility groups, the scheduler rejects incompatible combinations at scheduling time, before any node-side work begins.

This is most useful for hardware that supports mutually exclusive operating modes. For example, a GPU that can run in either MIG mode or vGPU mode: a device in MIG mode and a device in vGPU mode cannot be co-allocated because they consume overlapping physical resources in incompatible ways. By declaring compatibilityGroups, the driver makes this constraint visible to the scheduler.

This feature builds on partitionable devices: the compatibilityGroups field lives on device.consumesCounters[] entries, which only exist for partitionable devices. Both the DRADeviceCompatibilityGroups and DRAPartitionableDevices feature gates must be enabled in the kube-apiserver and kube-scheduler.

How it works

A driver defines a compatibilityGroups list for each device.consumesCounters[] entry in a ResourceSlice. The list contains at most 2 opaque string names that represent the operating mode or partition type of that device on that particular counter set.

When the scheduler allocates multiple devices that draw from the same counter set, it computes the intersection of their compatibilityGroups. Allocation succeeds only if that intersection is non-empty — meaning every co-allocated device shares at least one common group name. Devices drawing from different counter sets are never compared against each other.

A device that declares no groups (an unset, nil, or empty list) is treated as a special case: it is only co-allocatable with other no-group devices on the same counter set. It is never co-allocatable with a device that declares one or more groups.

The constraint applies across all claims being allocated in a single scheduling cycle: if two claims each allocate a device from the same counter set, the cross-claim group intersection is also enforced.

Example

Consider a GPU that can operate in either MIG mode or vGPU mode. The driver publishes two devices, each consuming 4 GiB from the same shared memory counter of 8 GiB. Based on counter capacity alone, both devices could be allocated together. Each device declares its operating mode as a compatibility group, making the two modes mutually exclusive:

apiVersion: resource.k8s.io/v1
kind: ResourceSlice
metadata:
  name: gpu-counters
spec:
  nodeName: worker-1
  pool:
    name: gpu-pool
    generation: 1
    resourceSliceCount: 2
  driver: gpu.example.com
  sharedCounters:
  - name: gpu-0-memory
    counters:
      memory:
        value: 8Gi
---
apiVersion: resource.k8s.io/v1
kind: ResourceSlice
metadata:
  name: gpu-devices
spec:
  nodeName: worker-1
  pool:
    name: gpu-pool
    generation: 1
    resourceSliceCount: 2
  driver: gpu.example.com
  devices:
  - name: gpu-0-mig
    consumesCounters:
    - counterSet: gpu-0-memory
      counters:
        memory:
          value: 4Gi
      compatibilityGroups:
      - mig
  - name: gpu-0-vgpu
    consumesCounters:
    - counterSet: gpu-0-memory
      counters:
        memory:
          value: 4Gi
      compatibilityGroups:
      - vgpu

In this example:

• gpu-0-mig belongs to the mig group.
• gpu-0-vgpu belongs to the vgpu group.

If a Pod or PodGroup requests two devices from this pool, the scheduler checks whether the two chosen devices share a common compatibility group on the gpu-0-memory counter set. Since {"mig"} ∩ {"vgpu"} = ∅, the pair is rejected — even though the counter set has enough memory for both. Both requests can only be satisfied by two MIG devices (or two vGPU devices) from a pool where such pairs exist.

Constraints

• Each consumesCounters[] entry may declare at most 2 group names.
• Group names must be unique within a single entry.
• Group names are opaque to Kubernetes; they are meaningful only within the publishing driver's pool.
• Groups are compared per counter set: groups on one counter set have no effect on co-allocation decisions for a different counter set.

Version-skew safety

When the DRADeviceCompatibilityGroups feature gate is disabled (the default for alpha), the kube-apiserver strips the compatibilityGroups field from any new or updated ResourceSlice — unless the old object already had the field set. The scheduler then treats devices in any pool that previously had grouped devices as belonging to an incomplete pool and skips them entirely.

Only a non-empty list counts as the field being set: compatibilityGroups: null and compatibilityGroups: [] are treated identically to omitting the field. Devices with them behave exactly like devices with no groups — they do not cause the scheduler to treat the pool as incomplete.

Device compatibility groups is controlled by the DRADeviceCompatibilityGroups feature gate in the kube-apiserver and kube-scheduler. The DRAPartitionableDevices feature gate must also be enabled.

Consumable capacity

The consumable capacity feature allows the same devices to be consumed by multiple independent ResourceClaims, with the Kubernetes scheduler managing how much of the device's capacity is used up by each claim. This is analogous to how Pods can share the resources on a Node; ResourceClaims can share the resources on a Device.

The device driver can set allowMultipleAllocations field added in .spec.devices of ResourceSlice to allow allocating that device to multiple independent ResourceClaims or to multiple requests within a ResourceClaim.

Users can set capacity field added in spec.devices.requests of ResourceClaim to specify the device resource requirements for each allocation.

For the device that allows multiple allocations, the requested capacity is drawn from — or consumed from — its total capacity, a concept known as consumable capacity. Then, the scheduler ensures that the aggregate consumed capacity across all claims does not exceed the device’s overall capacity. Furthermore, driver authors can use the requestPolicy constraints on individual device capacities to control how those capacities are consumed. For example, the driver author can specify that a given capacity is only consumed in increments of 1Gi.

Here is an example of a network device which allows multiple allocations and contains a consumable bandwidth capacity.

kind: ResourceSlice
apiVersion: resource.k8s.io/v1
metadata:
  name: resourceslice
spec:
  nodeName: worker-1
  pool:
    name: pool
    generation: 1
    resourceSliceCount: 1
  driver: dra.example.com
  devices:
  - name: eth1
    allowMultipleAllocations: true
    attributes:
      name:
        string: "eth1"
    capacity:
      bandwidth:
        requestPolicy:
          default: "1M"
          validRange:
            min: "1M"
            step: "8"
        value: "10G"

The consumable capacity can be requested as shown in the below example.

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: bandwidth-claim-template
spec:
  spec:
    devices:
      requests:
      - name: req-0
        exactly:
          deviceClassName: resource.example.com
          capacity:
            requests:
              bandwidth: 1G

The allocation result will include the consumed capacity and the identifier of the share.

apiVersion: resource.k8s.io/v1
kind: ResourceClaim
...
status:
  allocation:
    devices:
      results:
      - consumedCapacity:
          bandwidth: 1G
        device: eth1
        shareID: "a671734a-e8e5-11e4-8fde-42010af09327"

In this example, a multiply-allocatable device was chosen. However, any resource.example.com device with at least the requested 1G bandwidth could have met the requirement. If a non-multiply-allocatable device were chosen, the allocation would have resulted in the entire device. To force the use of a only multiply-allocatable devices, you can use the CEL criteria device.allowMultipleAllocations == true.

DistinctAttribute constraint

When requesting multiple devices in a ResourceClaim, you can use the DistinctAttribute constraint to ensure that each allocated device has a different value for a specified attribute. This constraint was introduced with the consumable capacity feature.

The DistinctAttribute constraint is particularly useful when working with multiply-allocatable devices. It prevents the scheduler from allocating the same device multiple times within a single ResourceClaim, even when that device allows multiple allocations.

Beyond preventing duplicate allocations, this constraint helps optimize performance by ensuring devices are distributed based on their attributes. For example, you can use it to distribute devices across different NUMA nodes to optimize memory bandwidth and reduce contention.

Granular status authorization

Starting in Kubernetes v1.36, DRA enforces fine-grained authorization checks for updates to ResourceClaim status by using synthetic subresources and node-aware verbs.

For security hardening guidance, including RBAC examples for scheduler and DRA drivers, see Hardening Guide - Dynamic Resource Allocation.

For a step-by-step cluster administrator procedure, see Harden Dynamic Resource Allocation in Your Cluster.

Optional node operations

In Dynamic Resource Allocation (DRA), the kubelet coordinates with a node-local driver via gRPC to prepare allocated devices before container start (NodePrepareResources) and to unprepare them upon Pod termination (NodeUnprepareResources). While this setup is critical for node-local hardware such as GPUs or FPGAs, some resources are managed entirely in the control plane and require no node-local setup.

The optional node operations feature allows resource drivers to declare that specific node-local gRPC operations can be skipped. When configured, the kubelet bypasses driver lookup and gRPC calls for those devices, eliminating the need to deploy and maintain empty node-local drivers on every worker node.

Driver configuration

Driver authors can specify the skipNodeOperations field in .spec.skipNodeOperations of a ResourceSlice. This field is a list of unique strings specifying the node-local operations to bypass for all devices in that slice.

Valid values are:

• "NodePrepareResources": Skips NodePrepareResources gRPC calls. This value cannot be specified unless "NodeUnprepareResources" is also listed (or "*" is specified). This limitation avoids Pods getting stuck in Terminating if a node-local plugin is missing, since the plugin is not checked during Pod startup when preparation is skipped.
• "NodeUnprepareResources": Skips NodeUnprepareResources gRPC calls.
• "*": Skips all node-local resource operations.

Here is an example of a ResourceSlice for a control-plane resource that skips all node-local operations:

apiVersion: resource.k8s.io/v1
kind: ResourceSlice
metadata:
  name: control-plane-resources
spec:
  nodeName: worker-1
  pool:
    name: central-pool
    generation: 1
    resourceSliceCount: 1
  driver: control-plane.example.com
  skipNodeOperations:
  - "*"
  devices:
  - name: virtual-device-1

Allocation result and execution

When the Kubernetes scheduler allocates a device to a ResourceClaim, it copies the skipNodeOperations list from the ResourceSlice into the allocation result:

apiVersion: resource.k8s.io/v1
kind: ResourceClaim
...
status:
  allocation:
    devices:
      results:
      - device: virtual-device-1
        driver: control-plane.example.com
        pool: central-pool
        skipNodeOperations:
        - "*"

When a Pod runs on a node, the kubelet reads the allocation results. If all allocated devices for a given driver within a ResourceClaim skip a specific operation, the kubelet completely bypasses calling that gRPC hook for that driver.

Operational considerations

In-place driver updates

Because the skipNodeOperations setting is copied from the ResourceSlice into the ResourceClaim at allocation time, running Pods and active allocations retain whatever setting was in place when they were scheduled.

If a driver's node operation requirements are updated in place (for example, changing from requiring node operations to skipping them), existing claims will still use the previous configuration. To avoid issues—such as terminating Pods hanging while waiting for a decommissioned node plugin—cluster administrators should ensure no active claims exist for a driver before altering its node operation requirements or removing node-local driver DaemonSets.

Node declared features integration

To prevent Pods from being scheduled onto nodes where the kubelet does not support skipping DRA operations (which would cause the kubelet to fail while waiting for a missing node plugin), this feature integrates with Node Declared Features. When a Pod uses a ResourceClaim with skipNodeOperations configured, the Kubernetes scheduler verifies that the target node declares support for the DRAOptionalNodeOperations feature in its .status.declaredFeatures before scheduling the Pod.

Optional node operations is controlled by the DRAOptionalNodeOperations feature gate in the kube-apiserver, kube-scheduler, and kubelet.

DRA device metadata in containers

DRA drivers can expose device metadata such as device attributes (PCI bus addresses or mdevUUID for mediated devices) or network configuration directly to containers as JSON files. This lets applications inside the container discover information about allocated devices without querying the Kubernetes API or building custom controllers.

KEP-5304 defines a device metadata protocol that drivers must follow so applications inside the container see a consistent layout across drivers and clusters. The DRA kubelet plugin library implements this protocol for you; the rest of this section describes how to use it.

Device metadata follows the same rules as device access: it is available inside a container only when that container requests the device in its container specification, and not otherwise. For how to request DRA devices in Pods and containers, see Request devices in workloads using DRA.

Device metadata protocol

The protocol consists of four rules:

1. File paths. Metadata files live inside containers under /var/run/kubernetes.io/dra-device-attributes. For a directly referenced ResourceClaim the path is resourceclaims/<claimName>/<requestName>/<driverName>-metadata.json; for a claim created from a ResourceClaimTemplate the path is resourceclaimtemplates/<podClaimName>/<requestName>/<driverName>-metadata.json (where podClaimName is pod.spec.resourceClaims[].name).

   In cases where the ResourceClaim request uses the prioritized list feature, only the top-level request name is used for the <requestName> segment in the file path (that is, the /<subrequest> portion is dropped). Inside the JSON file, the requests[].name field carries the full <request>/<subrequest> reference (for example, gpu/high-memory) so that consumers can identify which alternative was allocated.

   The path constants are defined in k8s.io/dynamic-resource-allocation/api/metadata.

2. JSON API. Each file is a stream of one or more DeviceMetadata objects serialized as versioned JSON with apiVersion and kind, following Kubernetes API conventions. The same metadata is encoded once per supported API version (newest first). All objects in the stream are semantically equivalent; consumers should use the first object they can decode.

3. Generation. When a driver updates a metadata file the embedded metadata.generation field must increase so consumers can detect changes.

4. Container exposure. Files are typically exposed via CDI bind-mounts, but other mechanisms are permitted as long as the file appears at the correct path and is read-only inside the container.

How device metadata works

Device metadata is a driver-side feature that does not require any Kubernetes API changes or feature gates. Using the DRA kubelet plugin library is a common way to implement a driver, but drivers can be built in other ways as well. Drivers that use the kubelet plugin enable this feature by passing the EnableDeviceMetadata and MetadataVersions options when starting the plugin. MetadataVersions specifies which API versions are serialized into the metadata file and must be set explicitly by the driver. Check the documentation of your DRA driver to learn whether device metadata is supported and how to enable it.

When device metadata is enabled, the driver generates metadata files and CDI bind-mount specifications while preparing the allocated devices for the pod, before the consuming containers start. The metadata appears inside containers at the well-known paths as defined above.

When a single request allocates devices from multiple DRA drivers, each driver writes its own metadata file. Containers enumerate *-metadata.json files in the request directory to discover all devices.

The Go package k8s.io/dynamic-resource-allocation/devicemetadata provides utilities for reading and decoding these metadata files by applications inside the container.

Metadata schema

Each metadata file conforms to the DeviceMetadata API (metadata.resource.k8s.io/v1alpha1). The following example shows a metadata file for a GPU device allocated through a ResourceClaimTemplate:

{
  "kind": "DeviceMetadata",
  "apiVersion": "metadata.resource.k8s.io/v1alpha1",
  "metadata": {
    "name": "pod0-gpu-2kqrd",
    "namespace": "gpu-test1",
    "uid": "c7e7b22e-239b-4498-b27c-7f1344481e14",
    "generation": 1
  },
  "podClaimName": "gpu",
  "requests": [
    {
      "name": "gpu",
      "devices": [
        {
          "driver": "gpu.example.com",
          "pool": "worker-0",
          "name": "gpu-0",
          "attributes": {
            "driverVersion": {
              "version": "1.0.0"
            },
            "index": {
              "int": 0
            },
            "model": {
              "string": "LATEST-GPU-MODEL"
            },
            "uuid": {
              "string": "gpu-18db0e85-99e9-c746-8531-ffeb86328b39"
            }
          }
        }
      ]
    }
  ]
}

Immediate and deferred metadata

Drivers provide metadata in one of two ways:

Immediate

The driver populates metadata while preparing the claim on the node and writes the metadata file before the container starts. This is typical for GPU drivers where device information is known at preparation time.

Deferred

In some cases, for example a network driver, the device information is not available during device allocation time but becomes available after the pod sandbox is created. In those cases the driver creates the CDI mount with an empty metadata file and writes the actual metadata later via an NRI hook that runs before the container starts. This ensures applications never see a missing or partially written file. Each update must increment metadata.generation so consumers can detect changes. The MetadataUpdater API in the DRA kubelet plugin library handles generation bookkeeping automatically for driver authors.

In both cases, metadata remains available to each consuming container for the lifetime of that container. Metadata files are cleaned up after all containers in the Pod have terminated.

To learn how to use device metadata in your workloads, see Access DRA device metadata.

Custom drivers

Custom, hand-crafted drivers that do not use the DRA kubelet plugin library must implement the device metadata protocol themselves. That means writing DeviceMetadata JSON at the correct file paths, incrementing metadata.generation on every update, and exposing the files read-only inside the container through CDI or an equivalent mechanism.

2.5 - Device Taints and Tolerations

This page describes device taints and tolerations in DRA, which let drivers and admins keep Pods off specific devices, or evict Pods already using them.

Device taints and tolerations

Device taints are similar to node taints: a taint has a string key, a string value, and an effect. The effect is applied to the ResourceClaim which is using a tainted device and to all Pods referencing that ResourceClaim. The "NoSchedule" effect prevents scheduling those Pods. Tainted devices are ignored when trying to allocate a ResourceClaim because using them would prevent scheduling of Pods.

The "NoExecute" effect implies "NoSchedule" and in addition causes eviction of all Pods which have been scheduled already. This eviction is implemented in the device taint eviction controller in kube-controller-manager by deleting affected Pods.

The "None" effect is ignored by the scheduler and eviction controller. DRA drivers can use it to communicate exceptions to admins or other controllers, for example degraded health of a device. Admins can also use it to do dry-runs of pod eviction in DeviceTaintRules (more on that below).

ResourceClaims can tolerate taints. If a taint is tolerated, its effect does not apply. An empty toleration matches all taints. A toleration can be limited to certain effects and/or match certain key/value pairs. A toleration can check that a certain key exists, regardless which value it has, or it can check for specific values of a key. For more information on this matching see the node taint concepts.

Eviction can be delayed by tolerating a taint for a certain duration. That delay starts at the time when a taint gets added to a device, which is recorded in a field of the taint.

Taints apply as described above also to ResourceClaims allocating "all" devices on a node. All devices must be untainted or all of their taints must be tolerated. Allocating a device with admin access (described above) is not exempt either. An admin using that mode must explicitly tolerate all taints to access tainted devices.

You can add taints to devices in the following ways, by using the DeviceTaintRule API kind.

Taints set by the driver

A DRA driver can add taints to the device information that it publishes in ResourceSlices. Consult the documentation of a DRA driver to learn whether the driver uses taints and what their keys and values are.

Taints set by an admin

An admin or a control plane component can taint devices without having to tell the DRA driver to include taints in its device information in ResourceSlices. They do that by creating DeviceTaintRules. Each DeviceTaintRule adds one taint to devices which match the device selector. Without such a selector, no devices are tainted. This makes it harder to accidentally evict all pods using ResourceClaims when leaving out the selector by mistake.

Devices can be selected by giving the name of a DeviceClass, driver, pool, and/or device. The DeviceClass selects all devices that are selected by the selectors in that DeviceClass. With just the driver name, an admin can taint all devices managed by that driver, for example while doing some kind of maintenance of that driver across the entire cluster. Adding a pool name can limit the taint to a single node, if the driver manages node-local devices.

Finally, adding the device name can select one specific device. The device name and pool name can also be used alone, if desired. For example, drivers for node-local devices are encouraged to use the node name as their pool name. Then tainting with that pool name automatically taints all devices on a node.

Drivers might use stable names like "gpu-0" that hide which specific device is currently assigned to that name. To support tainting a specific hardware instance, CEL selectors can be used in a DeviceTaintRule to match a vendor-specific unique ID attribute, if the driver supports one for its hardware.

The taint applies as long as the DeviceTaintRule exists. It can be modified and and removed at any time. Here is one example of a DeviceTaintRule for a fictional DRA driver:

apiVersion: resource.k8s.io/v1
kind: DeviceTaintRule
metadata:
  name: example
spec:
  # The entire hardware installation for this
  # particular driver is broken.
  # Evict all pods and don't schedule new ones.
  deviceSelector:
    driver: dra.example.com
  taint:
    key: dra.example.com/unhealthy
    value: Broken
    effect: NoExecute

The kube-apiserver automatically tracks when this taint was created by setting the timeAdded field in the spec. The toleration period starts at that time stamp. During updates which change the effect (see simulated eviction flow below), the kube-apiserver automatically updates the time stamp. Users can control the time stamp explicitly by setting the field when creating a DeviceTaintRule and by changing it to some different value when updating.

The status contains a condition added by the eviction controller:

kubectl describe devicetaintrules

Name:         example
...
Spec:
  Device Selector:
    Driver:  dra.example.com
  Taint:
    Effect:      NoExecute
    Key:         dra.example.com/unhealthy
    Time Added:  2025-11-05T18:15:37Z
    Value:       Broken
Status:
  Conditions:
    Last Transition Time:  2025-11-05T18:15:37Z
    Message:               1 pod evicted since starting the controller.
    Observed Generation:   1
    Reason:                Completed
    Status:                False
    Type:                  EvictionInProgress
Events:                    <none>

Pods get evicted by deleting them. Usually this happens very quickly, except when a toleration for the taint delays it for a certain period or when there are very many pods which need to be evicted. When it takes longer, the message provides information about the current status:

2 pods need to be evicted in 2 different namespaces. 1 pod evicted since starting the controller.

The condition can be used to check whether an eviction is currently active:

kubectl wait --for=condition=EvictionInProgress=false DeviceTaintRule/example

Beware of the potential race between scheduler and controller observing the new taint at different times, which can lead to pods still being scheduled at a time when the controller thinks that there are none which need to be evicted and thus sets this condition to False. In practice, this race is made very unlikely by updating the status only after an intentional delay of a few seconds.

For effect: None, the message provides information about the number of affected devices, how many of those are allocated, and how many pods would be evicted if the effect was NoExecute. This can be used to do a dry-run before actually triggering eviction:

• Create a DeviceTaintRule with the desired selectors and effect: None.

• Review the message:

  3 published devices selected. 1 allocated device selected.
  1 pod would be evicted in 1 namespace if the effect was NoExecute.
  This information will not be updated again. Recreate the DeviceTaintRule to trigger an update.
  

  Published devices are those listed in ResourceSlices. Tainting them prevents allocation for new pods. Only allocated devices cause eviction of the pods using them.

• Edit the DeviceTaintRule and change the effect into NoExecute.

3 - Pod-level resource managers

Pod-level resource support for the existing resource managers (Topology, CPU, and Memory) extends them to handle pod-level resource specifications. When enabled (via the PodLevelResources and PodLevelResourceManagers feature gates), the resource managers can use .spec.resources directly as the basis for their allocation decisions, evolving from a strictly per-container allocation model to a Pod-centric one. This partitioning scheme introduces a more flexible and powerful resource management model, particularly for performance-sensitive workloads. It allows you to define hybrid allocation models where some containers in a Pod receive exclusive, NUMA-aligned resources, while others share the remaining resources from a pod-level shared pool.

To practice setting up kubelet resource managers with pod-level resources and observe allocation behaviors hands-on, follow the Use pod-level resources with kubelet resource managers tutorial.

To understand pod-level resource managers, it is helpful to contrast them with the traditional container-focused model. Previously, kubelet resource allocations were strictly all or nothing: to receive exclusive NUMA-aligned resources for your workload, every container in the Pod had to be Guaranteed (specifying requests equal to limits for both CPU and memory).

Pod-level resource managers use .spec.resources to enable flexible partitioning based on the configured Topology Manager scope:

• pod scope: The kubelet allocates and NUMA-aligns a single Pod bubble for the entire Pod based on .spec.resources. Containers requesting exclusive allocations carve out dedicated slices from within this Pod bubble, while all other containers share the remaining bubble capacity in a pod-isolated shared pool.
• container scope: Enables a hybrid allocation model. The kubelet allows individual containers to receive exclusive, NUMA-aligned resources directly from the Node's allocatable pool, while using the Pod's .spec.resources ceiling to cap collective consumption—allowing sidecars to run in the general Node shared pool without requiring every container in the Pod to be Guaranteed.

Both standard init containers and restartable init containers (sidecars) are fully supported. They can receive exclusive resource slices or use the Pod's shared pool, and the pod-level resource managers respect their lifecycle rules (for example, reusable resources for standard init containers vs. persistent reservations for sidecars).

Glossary

Pod level resources specification

The resource budget defined at the Pod level in .spec.resources, that specifies the collective requests and limits for the entire Pod.

Guaranteed container

A container that specifies resource requests equal to its limits for both CPU (exclusive CPU allocation requires a positive integer value) and Memory. Consistent with existing kubelet behavior, this makes the container eligible for exclusive resource allocation from the resource managers.

Exclusive slice

A dedicated portion of resources (for example: specific CPUs or memory pages) allocated solely to a single container, ensuring isolation from other containers.

Pod shared pool

The subset of a Pod's allocated resources that remains after all exclusive slices have been reserved. These resources are shared by all containers in the Pod that do not receive an exclusive allocation. While containers in this pool share resources with each other, they are strictly isolated from the exclusive slices and the general node-wide shared pool.

How pod-level resource managers work

The CPU and Memory resource managers operate differently depending on the configured Topology Manager scope.

Topology manager's pod scope and pod-level resources

When the Topology Manager scope is set to pod, the kubelet performs a single NUMA alignment for the entire Pod based on the resource budget defined in .spec.resources.

The resulting NUMA-aligned resource pool is then partitioned:

1. Exclusive slices: Containers that specify Guaranteed resources (requests equal to limits for both CPU and memory, and the CPU request is a positive integer) receive exclusive slices from the Pod's total allocation.
2. Pod shared pool: The remaining resources form a shared pool for all other containers in the Pod that do not receive an exclusive allocation. While containers in this pool share resources with each other, they are strictly isolated from the exclusive slices and the general node-wide shared pool.

Note that when standard init containers run to completion, their resources enter a per-Pod reusable set rather than returning to the Node's resource pool. Because they run sequentially, subsequent app containers can reuse these resources (either for their own exclusive slices or for the shared pool).

This allows you to co-locate containers that require exclusive resources (for example, a high-performance primary application) with those that do not (for example, sidecars for logging or monitoring), all within a single NUMA-aligned Pod.

Consider the containers in the following Pod spec, where the Topology Manager scope is pod and the Pod has a total budget of 4 CPUs. main-app requests an exclusive 2 CPU slice, while the sidecars share the remaining 2 CPUs in the Pod's shared pool:

pods/resource/pod-level-resource-managers-pod-scope-mixed.yaml

apiVersion: v1
kind: Pod
metadata:
  name: pod-scope-mixed
  annotations:
    kubernetes.io/description: "A pod demonstrating pod-level scope where one container gets exclusive resources and others share the remaining pod resources in a shared pool."
spec:
  # At Pod level, the Pod has CPU request equal to limits and memory request
  # also equal to memory limits. The main-app container meets the requirements
  # for the Guaranteed QoS class at container level, and the sidecar containers
  # don't specify any resource request. Under pod scope, this means that the
  # kubelet could statically assign 4 CPUs to the overall Pod, of which 2 are
  # assigned exclusively to the main-app container, and the remaining 2 are
  # shared by the sidecars in the pod's shared pool.
  resources:
    requests:
      cpu: "4"
      memory: "4Gi"
    limits:
      cpu: "4"
      memory: "4Gi"
  initContainers:
  - name: metrics-sidecar
    # Note: This is a placeholder image for demonstration purposes, not an
    #actual metrics helper.
    image: registry.k8s.io/pause:3.9
    restartPolicy: Always
  - name: logging-sidecar
    # Note: This is a placeholder image for demonstration purposes, not an
    # actual logging agent.
    image: registry.k8s.io/pause:3.9
    restartPolicy: Always
  containers:
  - name: main-app
    # Note: This is a placeholder image for demonstration purposes.
    image: registry.k8s.io/pause:3.9
    resources:
      requests:
        cpu: "2"
        memory: "2Gi"
      limits:
        cpu: "2"
        memory: "2Gi"

Important considerations:

When using pod-level resources with the Topology manager's pod scope, there are some important considerations:

• Empty shared pool restriction: This configuration does not allow Pod specifications that would produce an empty Pod shared pool if there are containers that require one. If the sum of resource requests from all containers that are Guaranteed exactly equals the total resource budget, and there is at least one other container that requires a shared pool, the kubelet rejects the Pod at admission.

  For example, the following Pod asks for a pod-level budget of 4 CPUs. main-app requires an exclusive 3 CPUs and metrics-sidecar requires an exclusive 1 CPU. Because there are 0 CPUs left in the shared pool for logging-sidecar, the kubelet rejects this Pod (the same validation applies for memory):

  pods/resource/pod-level-resource-managers-empty-shared-pool.yaml

  apiVersion: v1
  kind: Pod
  metadata:
    name: empty-shared-pool
    annotations:
      kubernetes.io/description: "A pod demonstrating a configuration that is rejected because exclusive containers consume the entire pod resource budget, leaving no resources for the remaining container in the shared pool."
  spec:
    # At Pod level, the Pod has CPU request equal to limits and memory request
    # also equal to memory limits. The main-app and metrics-sidecar containers
    # meet the requirements for the Guaranteed QoS class at container level, and
    # the logging-sidecar container doesn't specify any resource request. Because
    # the Guaranteed containers consume the entire pod resource budget,
    # leaving 0 CPUs for the shared pool required by logging-sidecar, this pod
    # will be rejected at admission.
    resources:
      requests:
        cpu: "4"
        memory: "4Gi"
      limits:
        cpu: "4"
        memory: "4Gi"
    initContainers:
    - name: metrics-sidecar
      # Note: This is a placeholder image for demonstration purposes, not an
      # actual metrics helper.
      image: registry.k8s.io/pause:3.9
      restartPolicy: Always
      resources:
        requests:
          cpu: "1"
          memory: "1Gi"
        limits:
          cpu: "1"
          memory: "1Gi"
    - name: logging-sidecar
      # Note: This is a placeholder image for demonstration purposes, not an
      # actual logging agent.
      image: registry.k8s.io/pause:3.9
      restartPolicy: Always
    containers:
    - name: main-app
      # Note: This is a placeholder image for demonstration purposes.
      image: registry.k8s.io/pause:3.9
      resources:
        requests:
          cpu: "3"
          memory: "3Gi"
        limits:
          cpu: "3"
          memory: "3Gi"
  

• Wasted resources: Any resources overallocated when using the pod scope (the total container requests sum to less than the pod-level budget and there are no shared pool containers, or the shared pool containers don't fully use the remaining amount) remain assigned and reserved for the Pod, effectively going to waste during the entire Pod execution.

• Persistent pool: The Pod's total resource pool (the NUMA alignment and total reserved capacity) is persistent. If a shared-pool container crashes and restarts, the Pod's overall resource reservation remains safely anchored on the Node. The Node releases the resources back to its general pool only when the entire Pod terminates.

Topology manager's container scope and pod-level resources

When the Topology Manager scope is set to container, the kubelet evaluates each container individually for exclusive allocation.

If the overall Pod achieves a Guaranteed QoS class (by specifying appropriate values in the Pod-level .spec.resources), you can mix and match containers:

• Containers with their own Guaranteed requests receive exclusive NUMA-aligned resources.
• Other containers in the Pod that do not specify Guaranteed requests run in the Node's shared pool.
• The collective resource consumption of all containers is still enforced by the Pod's .spec.resources limits.

This scope is useful when you have an infrastructure sidecar that needs to be aligned to a specific NUMA Node for device access, while the main workload can run in the general Node shared pool.

Consider the containers in the following Pod spec, where the Topology Manager scope is container and the Pod represents a workload with an infrastructure sidecar and two application workers, with a total budget of 4 CPUs. The infrastructure-sidecar gets an exclusive, NUMA-aligned 2 CPU slice. The two application workers (worker-1 and worker-2) run in the general, node-wide shared pool:

pods/resource/pod-level-resource-managers-container-scope-mixed.yaml

apiVersion: v1
kind: Pod
metadata:
  name: container-scope-mixed
  annotations:
    kubernetes.io/description: "A pod demonstrating container-level scope where one container gets exclusive resources and others run in the node's shared pool."
spec:
  # At Pod level, the Pod has CPU request equal to limits and memory request
  # also equal to memory limits. The infrastructure-sidecar container meets the
  # requirements for the Guaranteed QoS class at container level, and the worker
  # containers don't specify any resource request. Under container scope, the
  # kubelet evaluates containers individually for exclusive allocation. This
  # means the infrastructure-sidecar gets an exclusive 2 CPU slice, while the
  # worker containers run in the node's general shared pool, all while bounded
  # by the overall pod limits.
  resources:
    requests:
      cpu: "4"
      memory: "4Gi"
    limits:
      cpu: "4"
      memory: "4Gi"
  initContainers:
  - name: infrastructure-sidecar
    # Note: This is a placeholder image for demonstration purposes, not an
    # actual infrastructure helper.
    image: registry.k8s.io/pause:3.9
    restartPolicy: Always
    resources:
      requests:
        cpu: "2"
        memory: "2Gi"
      limits:
        cpu: "2"
        memory: "2Gi"
  containers:
  - name: worker-1
    # Note: This is a placeholder image for demonstration purposes.
    image: registry.k8s.io/pause:3.9
  - name: worker-2
    # Note: This is a placeholder image for demonstration purposes.
    image: registry.k8s.io/pause:3.9

CPU quota (CFS)

When running mixed workloads within a Pod, the kubelet enforces isolation differently depending on the allocation:

• Exclusive containers: Containers with exclusive CPU slices have their CPU CFS quota enforcement disabled, allowing them to run without throttling by the Linux scheduler.
• Pod shared pool containers: Containers in the Pod shared pool have CPU CFS quotas enabled, ensuring they do not consume more than the leftover Pod budget and preventing them from interfering with the exclusive containers.

Persistent pool and restarts

The Pod's total resource pool (the NUMA alignment and total reserved capacity) is persistent. If a container in the Pod's shared pool crashes and restarts, the Pod's overall resource reservation remains safely anchored on the Node. The Node releases the resources back to its general pool only when the entire Pod terminates.

kubelet downgrades and state checkpoints

In Kubernetes 1.36, enabling PodLevelResourceManagers updated internal kubelet state checkpoint files (cpu_manager_state and memory_manager_state) to a format that older kubelet versions cannot load. If you downgrade a 1.36 kubelet after active use, the older kubelet fails to start; you must drain the Node, delete these checkpoint files, and restart the kubelet.

In Kubernetes 1.37, checkpoint files use a forward-compatible format to prevent start-up failures during downgrades, though 1.36 kubelet versions do not restore active pod-level resource assignments. For complete details on checkpoint formats and recovery, see the Pod-level resource managers reference.

Observability and metrics

You can monitor the behavior and health of the resource managers across both container-level and pod-level allocations using the following kubelet metrics (enabled via the PodLevelResourceManagers feature gate):

• resource_manager_allocations_total: Counts the total number of exclusive resource allocations performed by a manager. The source label ("pod" or "node") distinguishes between allocations drawn from the node-level pool versus a pre-allocated pod-level pool.
• resource_manager_allocation_errors_total: Counts errors encountered during exclusive resource allocation, distinguished by the intended allocation source ("pod" or "node").
• resource_manager_container_assignments: Tracks the cumulative number of containers that will be granted a specific type of resource assignment. The assignment_type label ("node_exclusive", "pod_exclusive", "pod_shared") provides visibility into how many containers are running with exclusive resources (from the node or pod pool) versus the pod-level shared pool.

PodResources API

In Kubernetes 1.37, the kubelet's node-local PodResources gRPC API includes pod-level resource allocations when PodLevelResourceManagers is enabled. Node-local monitoring agents and device plugins can query top-level Pod assignments (cpu_ids and memory) while avoiding double-counting container-level allocations.

For complete API schemas, field masks, and scope-by-scope reporting tables, see the Pod-level resource managers reference.

Limitations and caveats

• The functionality is only implemented for the static CPU Manager policy and the Static Memory Manager policy. Note that the BestEffort policy is not supported for the Memory Manager.
• This feature is only supported on Linux nodes. On Windows nodes, the resource managers will act as a no-op for pod-level allocations.

What's next

• Learn how to Assign Pod-level CPU and memory resources.
• Follow the Use pod-level resources with kubelet resource managers tutorial.
• Read about Node Resource Managers.