Scheduling Building Block APIs and the workloadbuilder Library

Workload-aware Scheduling defines a set of reusable API building blocks under the scheduling.k8s.io API group. Controller authors embed these primitives into their own APIs so that users express scheduling intent (gang scheduling, topology, disruption behavior) with a consistent schema across the ecosystem, and the shared workloadbuilder library compiles that intent into the scheduler-facing Workload, PodGroup, and CompositePodGroup objects.

The built-in consumer of these building blocks today is the Job controller, gated by the WorkloadWithJob feature gate.

Reusable building blocks

The building blocks are strongly-typed Go structs in the scheduling.k8s.io/v1alpha3 API group. They are meant for controller authors: a controller embeds these structs into its own API type as fields, and the workloadbuilder library compiles them. The type names follow two conventions: leaf-level types are prefixed WorkloadPodGroup... (for example, WorkloadPodGroupSchedulingPolicy) and the multi-level variants - WorkloadCompositePodGroup....

Each controller chooses the field names and structure that are idiomatic for its own API, so these blocks impose no fixed top-level shape. Although not strictly enforced, reusing the standard field names and structure is recommended, so that a user who has configured gang scheduling on one controller's resource recognizes the same options on another's.

To adopt the building blocks, a controller adds the ones it wants to support as fields on its own types. For example, the Job API groups all four of them into a single type reachable at spec.scheduling:

type JobSchedulingConfiguration struct {
    SchedulingPolicy      *schedulingv1alpha3.WorkloadPodGroupSchedulingPolicy
    SchedulingConstraints *schedulingv1alpha3.WorkloadPodGroupSchedulingConstraints
    DisruptionMode        *schedulingv1alpha3.WorkloadPodGroupDisruptionMode
    ResourceClaims        []schedulingv1alpha3.WorkloadPodGroupResourceClaim
}

A different controller might nest the same blocks per component of a multi-part workload, or support only a subset of them.

Scheduling policy

The scheduling policy block carries the same basic and gang policies as a PodGroup's spec.schedulingPolicy. See PodGroup scheduling policies for what each policy means and how the scheduler applies it.

The one difference is that the block's gang minimum count is optional. Users may leave minCount unset, in which case the controller supplies a default that makes sense for its own domain; the Job controller, for example, uses the Job's parallelism.

Scheduling constraints

The scheduling constraints block carries the topology constraints documented in Topology-aware workload scheduling: a node label key naming the domain (such as a rack or a zone) that every Pod in the group must share, with at most one topology constraint per group. Controllers should freeze the field after creation, since constraints are immutable in the compiled Workload.

Disruption mode

The disruption mode block selects whether the group's Pods may be disrupted individually (single) or only as a unit (all), corresponding to the Pod and PodGroup disruption modes documented in Pod group disruption and priority.

The library rejects combinations that are not meaningful. For example, prevent all disruption mode for PodGroups with BasicSchedulingPolicy, because the preemption unit must not be larger than the scheduling unit - a group scheduled pod-by-pod has no group-level unit to preempt or disrupt.

Resource claims

The resource claims block expresses which dynamic resource allocation claims are shared by every Pod in the group rather than allocated per Pod. Each entry names the claim within the group and points at either an existing ResourceClaim or a ResourceClaimTemplate from which one is generated. A group may declare at most four claims.

Pods consume the devices allocated to the group by declaring a matching claim in their own spec, using the same name and referring to the same object.

Composite building blocks

Multi-level controllers that orchestrate other controllers (for example, JobSet creating Jobs) coordinate a group of groups. For that layer, the API provides an analogous set of primitives prefixed with WorkloadCompositePodGroup... (for example WorkloadCompositePodGroupSchedulingPolicy). They follow the same shapes as the leaf-level blocks, except the composite gang policy uses minGroupCount (the minimum number of child groups that must be schedulable together) in place of the leaf's minCount. Keeping leaf and composite types distinct lets each hierarchy level evolve independently.

Example: the Job integration

The Job controller is the built-in example of these blocks in use. A user fills in a Job's spec.scheduling, and the controller compiles it into a Workload and PodGroup. See Integrate with Workload APIs for a complete Job manifest, the defaults that apply when spec.scheduling is omitted, and which fields you can change after creation.

The workloadbuilder library

workloadbuilder is a shared Go library that turns a controller's scheduling intent into the scheduler-facing Workload and its runtime PodGroup/CompositePodGroup objects, so each controller does not reimplement defaulting, validation, and template compilation. It is designed for both in-tree controllers (such as the Job controller) and out-of-tree controllers (such as JobSet or Kubeflow TrainJob), which vendor it like any other Go dependency. It ships from k8s.io/component-helpers/scheduling/schedulingv1/workloadbuilder.

The library consumes the scheduling.k8s.io/v1alpha3 building blocks and compiles them into scheduling.k8s.io/v1beta1 Workload and PodGroup objects, while CompositePodGroup objects remain scheduling.k8s.io/v1alpha3.

How a controller uses it

A controller describes its workload as a tree of WorkloadItem nodes, one per logical component. A node with no children becomes a single PodGroupTemplate, while a node with children becomes a CompositePodGroupTemplate over them, which is how a multi-level controller represents a group of groups. Each node carries:

• a default config, the controller's own defaults for anything the user leaves unset. This is where a controller decides, for example, that an unconfigured Job stays on basic scheduling.
• an input, the user's intent taken from the controller's API. The controller records each building block together with the field path it lives at, so validation errors point at the exact field the user set.
• optional callbacks, which adjust the merged configuration. This is how a controller supplies a context-specific default, such as filling in an unset gang minimum count from the Job's parallelism.

The controller then hands that tree to a Builder and works through four calls:

1. NewBuilder constructs the builder from the tree, along with the name, namespace, and owner reference for the object to be produced. The owner becomes the Workload's controller reference, which is used for discovery and garbage collection.
2. Validate resolves the tree and reports any problems as a list of field errors, which a controller returns from its own API validation.
3. BuildWorkload compiles the tree into a Workload. The result is cached, so several PodGroups can be created from one compiled result.
4. NewPodGroup creates a runtime PodGroup from one of the compiled templates, naming the template it should be built from.

builder := workloadbuilder.NewBuilder(item, opts)
if errs := builder.Validate(ctx, workloadbuilder.ValidationInput{}); len(errs) > 0 {
    // reject the request
}
workload, err := builder.BuildWorkload()
podGroup, err := builder.NewPodGroup("trainer-pg", item.Name)

For complete, runnable versions of this flow, including how the Job controller wires it up, see the examples in the package reference.

Validating a scheduling configuration

Validate checks a configuration in two layers:

• Structural validation of the building blocks themselves: required fields, value ranges, the rule that exactly one member of a union is set, and immutability. These checks come from declarative validation rules generated from the API types.
• Controller-policy checks that declarative validation cannot express: the allow-lists described below, and cross-field rules such as rejecting the all disruption mode alongside the basic policy.

Validation also differs between creating and updating an object. On an update, the library additionally enforces the fields that are frozen after creation, which means the controller has to supply the previously stored configuration along with the new one. On a create there is nothing to compare against, so those checks do not apply.

Opting out of declarative validation

Whether you want the first layer depends on where your controller runs, and one option controls it:

• Out-of-tree controllers leave declarative validation enabled, which is the default. Nothing else applies those structural rules to a custom resource, so a single Validate call covers both layers.
• In-tree controllers set DisableDeclarativeValidation, because the API server already runs declarative validation on the embedded blocks while validating the parent object. Skipping the first layer avoids checking the same fields twice, leaving Validate to run only the controller-policy checks.

Opting in to scheduling options

Because the building-block types are shared across controllers, future releases may add scheduling options that do not make sense for every controller. To keep new options from silently leaking in, workloadbuilder uses an allow-list model: a controller declares the policies and disruption modes it supports, and Validate rejects anything outside that set, reporting the error at the offending block's field path.

Options are therefore denied by default. When a new policy is introduced, an existing controller keeps rejecting it until its maintainers extend the allow-list, which for an out-of-tree controller means updating its vendored copy of the library as well.

builder := workloadbuilder.NewBuilder(item, workloadbuilder.BuildOptions{
    Owner:                  owner,
    AllowedPolicies:        []workloadbuilder.SchedulingPolicyOption{workloadbuilder.BasicPolicy, workloadbuilder.GangPolicy},
    AllowedDisruptionModes: []workloadbuilder.DisruptionModeOption{workloadbuilder.SingleMode, workloadbuilder.AllMode},
})
allErrs := builder.Validate(ctx, workloadbuilder.ValidationInput{})

Generating PodGroups from an existing Workload

When the Workload already exists, whether compiled by a parent controller or created manually, a child controller that only manages the runtime PodGroup uses NewBuilderFromExistingWorkload instead. That builder creates PodGroup objects from the supplied Workload using its own owner reference. It does not validate or compile anything, so the existing Workload is never recompiled.

builder := workloadbuilder.NewBuilderFromExistingWorkload(parentWorkload, workloadbuilder.BuildOptions{Owner: owner})
podGroup, err := builder.NewPodGroup("trainer-pg", "trainer-pgt-0")

What's next

• Learn about PodGroup scheduling policies.
• Learn about Pod group disruption and priority.
• Learn about Topology-aware workload scheduling.
• See the Workload API overview.
• See how the Job controller integrates with the Workload APIs.
• Read the workloadbuilder package reference.