This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

CompositePodGroup API

FEATURE STATE: Kubernetes v1.37 [alpha](disabled by default)

A CompositePodGroup is a runtime object that represents a non-leaf node in a multi-level workload hierarchy. While the Workload API defines static scheduling policy templates, CompositePodGroup and PodGroup objects are the runtime counterparts that carry policies and hierarchy references for a specific workload instance.

What is a CompositePodGroup?

The CompositePodGroup API resource is part of the scheduling.k8s.io/v1alpha3 API group. Your cluster must have that API group enabled, as well as the CompositePodGroup feature gate, before you can use this API.

A CompositePodGroup represents a grouping of child groups (which can be CompositePodGroup or PodGroup objects). It carries scheduling policies, disruption modes, priority settings, and optional topology constraints that apply collectively across its child groups.

API structure

A CompositePodGroup consists of a spec that defines the desired scheduling behavior for its child groups, and a status subresource.

Scheduling policy

Each CompositePodGroup carries a scheduling policy (basic or gang) in spec.schedulingPolicy. When a workload controller creates a CompositePodGroup, this policy is copied from the Workload's CompositePodGroupTemplate at creation time.

For a gang policy on a CompositePodGroup, the minGroupCount field specifies the minimum number of child groups that must be schedulable simultaneously:

spec:
  schedulingPolicy:
    gang:
      minGroupCount: 2

Parent group reference

Non-root CompositePodGroup resources specify their parent group using spec.parentCompositePodGroupName. Root CompositePodGroup objects leave this field unset.

spec:
  parentCompositePodGroupName: root-group-0

Workload reference

The spec.workloadRef field links the CompositePodGroup back to the CompositePodGroupTemplate in the Workload object it was derived from.

spec:
  workloadRef:
    workloadName: hierarchical-workload
    templateName: replica-group

Status

The CompositePodGroup API schema includes a status subresource. In the alpha release, the status field is present in the API type, but kube-scheduler does not update or populate status conditions for CompositePodGroup objects. Status tracking for composite groups will be implemented in future releases.

Creating a CompositePodGroup

Workload controllers create CompositePodGroup objects automatically from Workload templates at runtime.

The following manifest creates a root CompositePodGroup with a gang scheduling policy that requires at least 2 child groups to be schedulable simultaneously:

apiVersion: scheduling.k8s.io/v1alpha3
kind: CompositePodGroup
metadata:
  name: root-group-0
  namespace: default
spec:
  workloadRef:
    workloadName: hierarchical-workload
    templateName: root
  schedulingPolicy:
    gang:
      minGroupCount: 2

You can inspect CompositePodGroup resources in your cluster:

kubectl get compositepodgroups

To view details for a specific composite group:

kubectl describe compositepodgroup root-group-0

How it fits together

The relationship between controllers, Workloads, CompositePodGroups, PodGroups, and Pods follows this pattern:

  1. The workload controller creates a Workload defining a tree of CompositePodGroupTemplates and leaf PodGroupTemplates.
  2. For each runtime instance, the controller creates a root CompositePodGroup, descendant CompositePodGroup objects, and leaf PodGroup objects in a top-down manner.
  3. The controller creates Pods that reference their leaf PodGroup via spec.schedulingGroup.podGroupName.

The following example illustrates a complete manifest hierarchy for a two-level workload:

apiVersion: scheduling.k8s.io/v1alpha3
kind: Workload
metadata:
  name: hierarchical-workload
  namespace: default
spec:
  compositePodGroupTemplates:
  - name: root
    schedulingPolicy:
      gang:
        minGroupCount: 2
    podGroupTemplates:
    - name: workers-a
      schedulingPolicy:
        gang:
          minCount: 4
    - name: workers-b
      schedulingPolicy:
        gang:
          minCount: 4
---
apiVersion: scheduling.k8s.io/v1alpha3
kind: CompositePodGroup
metadata:
  name: root-group-0
  namespace: default
spec:
  workloadRef:
    workloadName: hierarchical-workload
    templateName: root
  schedulingPolicy:
    gang:
      minGroupCount: 2
---
apiVersion: scheduling.k8s.io/v1alpha3
kind: PodGroup
metadata:
  name: workers-a-0
  namespace: default
spec:
  parentCompositePodGroupName: root-group-0
  workloadRef:
    workloadName: hierarchical-workload
    templateName: workers-a
  schedulingPolicy:
    gang:
      minCount: 4
---
apiVersion: scheduling.k8s.io/v1alpha3
kind: PodGroup
metadata:
  name: workers-b-0
  namespace: default
spec:
  parentCompositePodGroupName: root-group-0
  workloadRef:
    workloadName: hierarchical-workload
    templateName: workers-b
  schedulingPolicy:
    gang:
      minCount: 4
---
apiVersion: v1
kind: Pod
metadata:
  name: worker-a-0
  namespace: default
spec:
  schedulingGroup:
    podGroupName: workers-a-0
  containers:
  - name: worker
    image: registry.k8s.io/pause:3.9

The Workload acts as a long-lived policy template, while CompositePodGroup and PodGroup resources handle per-instance runtime scheduling state.

What's next

1 - CompositePodGroup Lifecycle

FEATURE STATE: Kubernetes v1.37 [alpha](disabled by default)

A CompositePodGroup represents a non-leaf node in a multi-level PodGroup hierarchy. Unlike PodGroup resources, CompositePodGroup resources do not directly contain Pods. Instead, they maintain a hierarchy of descendant CompositePodGroup and PodGroup objects and carry scheduling policies that apply to its children groups.

Ownership and garbage collection

A CompositePodGroup object, together with its descendant CompositePodGroup and PodGroup resources, is owned by the workload controller that created it via Kubernetes ownerReferences. When the owning workload object gets deleted, cascading garbage collection automatically deletes the associated group hierarchy.

CompositePodGroup names must be unique within a namespace and must be valid DNS subdomains.

Creation ordering

To ensure proper hierarchy resolution and scheduling, workload controllers create resources in a top-down order:

  1. Workload: Defines static templates (CompositePodGroupTemplates and PodGroupTemplates).
  2. Root CompositePodGroup: Created with spec.workloadRef pointing to the root template in the Workload.
  3. Descendant CompositePodGroups and PodGroups: Created top-down. Each child group specifies its parent by using spec.parentCompositePodGroupName and its template using spec.workloadRef.
  4. Pods: Created with spec.schedulingGroup.podGroupName pointing to their leaf PodGroup.

If a group references a parent CompositePodGroup that does not exist, or if a Pod references a PodGroup that has not yet been created, the scheduler holds off scheduling until all parent resources in the hierarchy exist.

Limitations and validation rules

  • Consistent scheduler name: All Pods across an entire CompositePodGroup hierarchy must use the same spec.schedulerName. If a mismatch is detected, the scheduler rejects the hierarchy as unschedulable.
  • Consistent priority: All Pods across an entire CompositePodGroup hierarchy must specify the same value of spec.priority which must be equal to the priority specified by the root group. If a mismatch is detected, the scheduler rejects the hierarchy as unschedulable.
  • Consistent preemption policy: All Pods across an entire CompositePodGroup hierarchy must use the same spec.preemptionPolicy. In addition, when the PodGroupPreemptionPolicy feature gate is enabled, the root group's preemption policy must be equal to the one specified by the Pods. If a mismatch is detected, the scheduler rejects the hierarchy as unschedulable.
  • Maximum nesting depth: The group-template hierarchy supports a maximum depth of 4 levels.
  • List item limit: The maximum number of child CompositePodGroupTemplates and PodGroupTemplates at any level of a Workload is 8.
  • Immutable gang group count: The spec.schedulingPolicy.gang.minGroupCount field on a CompositePodGroup is immutable after creation.
  • Immutable hierarchy references: spec.parentCompositePodGroupName on groups and spec.schedulingGroup on Pods are immutable once set.

What's next