Kubernetes News

The Kubernetes blog is used by the project to communicate new features, community reports, and any news that might be relevant to the Kubernetes community.
The Kubernetes project logo

  • How to Pretty-Print Your Kubernetes YAML as KYAML and Why You'd Want To

    YAML has been the standard way to write Kubernetes manifests for years. Every example, tutorial, and configuration file you come across is written in it. The problem isn't that YAML is a bad format. It's that YAML gives you a lot of choices, and not all of them are equally good for writing Kubernetes manifests. Some features make files harder to read, some are easy to misuse and others can lead to surprising behavior.

    The interesting part is that Kubernetes doesn't actually need most of those features. It only relies on a small subset of YAML. This led to a simple question: if Kubernetes only needs a small part of YAML, why not standardize on that part and avoid the rest? Instead of introducing a new configuration language, SIG CLI introduced KYAML, a stricter, more consistent way to write YAML.

    What is KYAML?

    KYAML is a strict subset (or "dialect") of standard YAML, designed to be parseable by the existing ecosystem without any changes, as proposed in KEP 5295. It does not introduce a new format or a new parser. It just narrows the scope of choices you make when writing YAML, so everyone ends up making the same ones.

    Think of it less like a new language and more like an agreed-upon style. Everything valid in KYAML is valid YAML.

    How KYAML solves it

    Standard YAML has a few well-known traps and JSON is not without its own.

    Whitespace sensitivity. Indentation defines structure in YAML, which means a wrongly indented file can remain syntactically valid while representing a different object than intended. This gets especially painful with templating tools like Helm, where you are manipulating indentation from outside the YAML context.

    Silent type coercion. String quoting is optional in YAML, which sounds convenient until it is not. Some values that look like strings get coerced into other types without warning. The classic example is the "Norway Bug".

    country:NO

    In standard YAML, NO is parsed as a boolean false, not the string "NO" and it has caught more than a few people off guard.

    JSON is not the answer either. It lacks comment support, is strict about trailing commas, and requires every key to be quoted, none of which makes for a good config writing experience.

    KYAML addresses all of these by making structure and types explicit:

    • Does not depend on whitespace for structure
    • Always quotes value strings so no silent type coercion
    • Always uses {} for maps and structs
    • Always uses [] for lists
    • Allows comments and trailing commas, unlike JSON
    • Includes a --- header to distinguish it from JSON at a glance, since both start with {

    YAML calls this flow style, as opposed to the conventional block style most people use. KYAML sits halfway between JSON and YAML, more explicit than default YAML, friendlier than JSON.

    Here is the same Pod manifest written in both formats for comparison.

    Standard YAML

    apiVersion:v1
kind:Pod
metadata:
name:my-pod
labels:
app:demo
spec:
containers:
- name:nginx
image:nginx:1.20

    KYAML

    ---
{
apiVersion:"v1",
kind:"Pod",
metadata:{
name:"my-pod",
labels:{
app:"demo",
},
},
spec:{
containers:[{
name:"nginx",
image:"nginx:1.20",
}],
},
}

    Notice the double-quoted string values, the braces around every mapping, the brackets around the list and the trailing commas. The additional syntax makes the document structure explicit instead of relying on indentation.

    How to pretty print YAML as KYAML

    There are different ways to get KYAML output.

    Option 1: kubectl -o kyaml

    Since Kubernetes 1.34, kubectl supports KYAML as a native output format.

    # Kubernetes 1.35+ (beta; feature enabled by default, still requires -o kyaml CLI param)
kubectl get deployment my-app -o kyaml

# Kubernetes 1.34 (alpha, opt-in)
export KUBECTL_KYAML=true
kubectl get deployment my-app -o kyaml

    To save the output to a file:

    kubectl get deployment my-app -o kyaml > my-app.yaml

    There are currently no plans to make KYAML the default output format. If you prefer using KYAML by default, you can configure your preferred default with kuberc. For more details, see the kuberc documentation.

    # Kubernetes 1.36+
kubectl kuberc set --section defaults --command get --option output=kyaml

# Kubernetes 1.33–1.35 (alpha prefix still required)
kubectl alpha kuberc set --section defaults --command get --option output=kyaml

    Option 2: Kubernetes' yamlfmt

    sigs.k8s.io/yaml ships a yamlfmt tool that can convert files to KYAML.

    Install via Go:

    go install sigs.k8s.io/yaml/yamlfmt@latest

    Running it against a file prints the KYAML version to stdout. It also accepts a directory, in which case it converts and prints every file in that directory. So you'll need to redirect the output to a file (or files) if you want the conversion to stick.

    yamlfmt -o=kyaml my-deployment.yaml

    It can also show you a diff instead of a full conversion:

    yamlfmt -o=kyaml -d my-deployment.yaml

    Option 3: Google's yamlfmt

    For converting existing files, Google's yamlfmt added a dedicated kyaml formatter in v0.21.0.

    Install via Go, or grab a binary from the releases page:

    go install github.com/google/yamlfmt/cmd/yamlfmt@latest

    It is also available as a pre-commit hook and as a Docker image for CI pipelines.

    Add a .yamlfmt config to your project root:

    formatter:
type:kyaml

    Preview the output without modifying your file:

    yamlfmt -dry my-deployment.yaml

    then apply:

    yamlfmt my-deployment.yaml

    To convert an entire directory:

    yamlfmt ./k8s/

    The kyaml formatter takes no additional configuration and does not share options with the default formatter so mixing them will cause an error.

    For more on the available modes and flags, check the command usage docs.

    Is KYAML worth adopting?

    Every valid KYAML file is a valid YAML file. So whatever you write in KYAML, your existing tools, your kubectl, your CI pipelines, none of them need to change. You can even pass KYAML as input to any version of kubectl, not just 1.34+, because at the end of the day it is just YAML.

    KYAML is not strictly necessary. You can keep writing block-style YAML and things will work. But it is a deliberate choice to make your configs less error-prone and more consistent especially across a team or a larger repo.

    It is less of a migration and more of a better habit.

  • Gateway API v1.6: TCPRoute and UDPRoute Graduate to Standard

    Gateway API logo

    The Kubernetes SIG Network community is thrilled to share the release of Gateway API v1.6.0, which was released on June 30th of this year!

    Gateway API has become the standard for modern, role-oriented, and expressive service networking in Kubernetes. In previous releases, Gateway API established a production-grade foundation for HTTP and TLS layer 7 traffic. With version 1.6.0, Gateway API takes a major step forward by expanding standard layer 4 protocol routing and introducing cleaner API boundaries for experimental innovation.

    Here is a quick summary of what's new in Gateway API v1.6.0:

    • TCPRoute and UDPRoute Graduate to Standard: Raw L4 TCP and UDP traffic routing reach GA stability in the v1 API version.
    • Experimental API Group Separation: Experimental resources transition to a distinct API group (gateway.networking.x-k8s.io) with an X prefix to make experimental vs. standard boundaries crystal clear.

    Let's dive into the details!

    TCPRoute and UDPRoute graduate to Standard

    Leads: Nick Young, Ricardo Katz and Zac Nixon

    Until now, Gateway API only offered a stable routing model for HTTP and TLS traffic. Workloads that speak a raw protocol over TCP or UDP - databases, DNS, VoIP, gaming, IoT telemetry - had no portable way to plug into a Gateway. Users either fell back to a plain Kubernetes Service, or to an implementation-specific CRD that doesn't travel between Gateway controllers.

    TCPRoute and UDPRoute close that gap: they route traffic to backends based on protocol and port alone, no L7 awareness required. With this release, both have graduated from the Experimental channel to Standard, and moved to the v1 API version. The v1alpha2 version of each was deprecated as of the v1.6 release, and will be removed in a future release.

    How it works

    A Gateway needs a listener that allows TCPRoute attachment:

    apiVersion:gateway.networking.k8s.io/v1
kind:Gateway
metadata:
name:example-gateway
spec:
gatewayClassName:example-gateway-class
listeners:
- name:foo
protocol:TCP
port:12345
allowedRoutes:
kinds:
- kind:TCPRoute

    A TCPRoute then attaches to that listener and forwards traffic to a backend:

    apiVersion:gateway.networking.k8s.io/v1
kind:TCPRoute
metadata:
name:tcp-app
spec:
parentRefs:
- name:example-gateway
sectionName:foo
rules:
- backendRefs:
- name:my-foo-service
port:6000

    Traffic arriving on the Gateway's port 12345 is proxied to the endpoints of my-foo-service on port 6000. Omitting sectionName and port from parentRefs attaches the route to every TCP listener on the Gateway instead of a single one.

    UDPRoute follows the same pattern; swap the listener protocol and the route kind:

    apiVersion:gateway.networking.k8s.io/v1
kind:Gateway
metadata:
name:example-gateway
spec:
gatewayClassName:example-gateway-class
listeners:
- name:foo
protocol:UDP
port:12345
allowedRoutes:
kinds:
- kind:UDPRoute
---
apiVersion:gateway.networking.k8s.io/v1
kind:UDPRoute
metadata:
name:udp-app
spec:
parentRefs:
- name:example-gateway
sectionName:foo
rules:
- backendRefs:
- name:my-foo-service
port:6000

    XBackend arrives in Experimental

    Leads: Keith Mattix II

    Gateway API v1.6 introduces the new XBackend resource, which is a general-purpose decorator for Service (and other backend types) within Gateway API.

    The Service resource is an amazing, stable, and flexible object, but that comes with some costs: The flexibility creates a lot of edge cases that Gateway API needs to handle, and the stability makes it impossible to add new concepts to Service.

    The XBackend resource builds on the ideas in the upstream EndpointSelector KEP, to add a Gateway API-native object that still targets the backend app, while allowing the community to extend it to handle use cases that are difficult or dangerous to handle with Service.

    The first version of XBackend includes support for ExternalHostname destinations, which are ruled out from Service support in Gateway API because of the possibility of confused deputy attacks.

    For XBackend, this support is an Extended/Optional feature, allowing implementations and users to opt in once they understand the security tradeoffs.

    This support is very useful for egress use cases (which are most commonly used for cluster-hosted agentic workloads), which the community is also working towards formalizing in GEPs about Gateways for Egress (work in progress, stay tuned!)

    The XBackend API is experimental and its behavior can change, do not assume it is ready for production

    An example of a Gateway with an ExternalName backend that can be used for egress to a cloud AI API is as follows:

    
# Gateway-level TLS remains authoritative for incoming connections
apiVersion:gateway.networking.k8s.io/v1
kind:Gateway
spec:
listeners:
- name:https
protocol:HTTPS
tls:
certificateRefs:
- name:gateway-cert
---
# Backend resource for external destination
apiVersion:gateway.networking.x-k8s.io/v1alpha1
kind:XBackend
metadata:
name:ai-provider-api
namespace:ai-apps
spec:
type:ExternalHostname
externalHostname:
hostname:api.ai-provider.com

---
# HTTPRoute referencing XBackend
apiVersion:gateway.networking.k8s.io/v1
kind:HTTPRoute
spec:
rules:
- backendRefs:
- name:ai-provider-api
kind:XBackend
group:gateway.networking.x-k8s.io

    The community is also working on moving Session Persistence config from XBackendTrafficPolicy into XBackend, along with other use cases like retries, TLS origination and similar config that is useful to be able to configure per-application rather than per-Route.

    Experimental resources move off the standard API group

    Previously, experimental resources shared the same API group as standard ones - gateway.networking.k8s.io - distinguished only by a v1alpha2-style version. TCPRoute and UDPRoute were the last resources to graduate under that scheme.

    Going forward, new experimental resources are defined in a separate group, gateway.networking.x-k8s.io, and the names of their API types get an X prefix - for example XBackend and XMesh. When one of these graduates to Standard, it's renamed into the gateway.networking.k8s.io group and drops the X prefix, the same way XMesh is expected to become Mesh.

    This separation makes the experimental/standard boundary explicit at the API group level, rather than relying on version strings alone.

    What's next & getting involved

    The graduation of TCPRoute and UDPRoute to Standard marks an essential milestone in making Gateway API a complete, universal ingress and mesh networking API for Kubernetes workloads across layer 4 and layer 7 protocols.

    Try it out

    You can start using Gateway API v1.6.0 today with your favorite Gateway controller implementation:

    Gateway API relies on an extensive conformance test suite to ensure consistent, portable behavior across all implementations. Here is a list of the implementations that are conforment with v1.6 on the day we published the article:

    Get involved

    Gateway API is an open, community-driven project built under Kubernetes SIG Network. We welcome contributions, feedback, and participation from everyone!

    Acknowledgments

    A huge thank you to all the contributors, reviewers, maintainers, and implementation authors whose hard work made Gateway API v1.6.0 possible!

  • Kubernetes v1.37 Sneak Peek

    As we get closer to the release date for Kubernetes v1.37, the project develops and matures, features may be deprecated, removed, or replaced with better ones for the project's overall health. This blog outlines some of the planned changes for the Kubernetes v1.37 release that the release team feels you should be aware of for the continued maintenance of your Kubernetes environment and keeping up to date with the latest changes. The information below reflects the current status of the v1.37 release and may change before the actual release date.

    Deprecations and removals for Kubernetes v1.37

    Kubectl: kubectl run --filename/-f to be deprecated

    The --filename (or -f) flag for kubectl run is being deprecated as the generated pod is always built purely from CLI arguments like NAME and --image.

    See kubernetes/kubernetes#138671 for the original issue and discussion.

    Kubelet: Static Pods can no longer reference Secrets or ConfigMaps

    Static Pods were never meant to read API resources directly, since they aren't created through the API server — but a bug let them reference Secrets or ConfigMaps via fields like configMapRef or secretRef. That bug is now fixed: as of v1.37 these references are strictly prohibited, and the PreventStaticPodAPIReferences feature gate that previously let you opt out of the restriction has been removed.

    See kubernetes/kubernetes#140226 for the original issue and discussion.

    Deprecating kube-proxy's support for ipvs mode

    kube-proxy support for ipvs mode was introduced in v1.8 to resolve iptables performance bottlenecks. However, since the kernel ipvs API alone cannot fully implement Kubernetes Services, ipvs mode continues to use iptables underneath (KEP-3866, "The ipvs mode of kube-proxy will not save us").

    Clusters running kube-proxy in ipvs mode (or mode: ipvs in KubeProxyConfiguration) would now be logging a deprecation warning on startup. The deprecation timeline looks like this:

    • By v1.40, ipvs mode for kube-proxy is expected to be disabled by default (still selectable via the feature gate)
    • By v1.43, support for ipvs mode would be removed entirely KEP-5495, Graduation Criteria. To confirm which mode you’re currently running, use:
    kubectl -n kube-system get configmap kube-proxy -o jsonpath='{.data.config\.conf}' | grep 'mode:'

    To understand the rationale behind this deprecation, see KEP-5495: Deprecate ipvs mode in kube-proxy.

    Ongoing major changes

    Future removal of cgroup v1 support

    As modern Linux distributions and container runtimes use cgroup v2 as the default, support for the legacy cgroup v1 is officially being phased out. Since the v1.35 release, the failCgroupV1 setting has defaulted to true. Consequently, the kubelet will fail to initialize on any nodes that still rely on cgroup v1 unless an explicit configuration override is applied.

    apiVersion:kubelet.config.k8s.io/v1beta1
kind:KubeletConfiguration
failCgroupV1:false# temporary override

    Using this override should be considered a short-term fix. Advanced resource management capabilities, such as In-Place Pod Resizing and Tiered Memory Protection, depend entirely on cgroup v2. While the override remains available in Kubernetes v1.37, users are encouraged to migrate to cgroup v2, as support for cgroup v1 is planned to be removed in a future release.

    To learn more about this deprecation, refer to KEP-5573: Remove cgroup v1 support.

    Breaking changes in Kubernetes v1.37

    SELinux volume relabeling ("SELinuxMount") graduates to GA

    SELinuxMount is expected to reach GA and be enabled by default in v1.37. Volumes would then be mounted with -o context=<label> (the mount option default) instead of being recursively relabeled, but only when the volume's CSI driver opts in via a CSIDriver that sets .spec seLinuxMount: true.

    Because a single mount can only hold one SELinux context, pods with different SELinux labels sharing a volume on the same node (which previously coexisted under recursive relabeling) may now fail to start. To retain the previous recursive behavior for a specific workload, set seLinuxChangePolicy: Recursive in the Pod spec.

    Clusters without SELinux enabled see no effect at all. To learn more, check SELinux Volume Label Changes goes GA (and likely implications in v1.37)

    Metrics API goes GA

    The metrics.k8s.io API is expected to graduate to Stable (GA) in Kubernetes v1.37 after spending nearly nine years in Beta. The API provides a standard way to retrieve CPU and memory usage for pods and nodes, powering widely used Kubernetes features such as the Horizontal Pod Autoscaler (HPA) and commands like kubectl top.

    This graduation recognizes the API's stability and widespread adoption, with no functional changes expected. Both v1 and v1beta1 will remain usable during the transition, enabling developers to adopt the stable API at their own pace without breaking existing workflows.

    To learn more about this enhancement, refer to KEP-5207: metrics.k8s.io API definition.

    Kubelet in UserNS a.k.a. Rootless Mode

    Traditionally, Kubernetes node components such as the kubelet run with root privileges on the host. While necessary for many deployments, this also means that a vulnerability in one of these components could potentially have a greater impact on the underlying system.

    With Kubernetes v1.37, kubelet in User Namespace (Rootless Mode) is expected to graduate to Beta. This enhancement allows Kubernetes node components to run inside a Linux user namespace as an unprivileged user on the host while still behaving as root within the namespace. By reducing the need for host-level root privileges, it adds an extra layer of isolation and helps limit the impact of potential vulnerabilities affecting node components.

    To learn more about this enhancement, refer to KEP-2033: Kubelet in UserNS(aka Rootless Mode).

    Volume health monitor

    Historically, Kubernetes has lacked an API for CSI drivers to report storage failures, which become evident only through failed mounts or hung I/O. Since remediation controllers had nothing machine-readable to act upon, the only way to figure out the root cause behind this failure was to cross-reference Kubernetes objects alongside external vendor dashboards.

    In Kubernetes v1.37, this KEP resets graduation to Alpha after an initial implementation in v1.21 and introduces four new CSI RPCs. The controller plugin reports the health of storage volumes using ControllerListVolumeHealth (lists unhealthy volumes) and ControllerGetVolumeHealth (checks a specific volume). A controller-side health monitor polls these CSI controllers and stores the results in PersistentVolumeClaim.status.healthStatus.

    On the node side, the kubelet calls NodeGetVolumeHealth to obtain the health of individual volumes on that node and records it in Pod.status.volumeHealth, while NodeGetStorageHealth reports the health of the drivers registered to a node in CSINode.status.storageHealth.

    The error vocabulary is kept simple, extensible, and machine-parsable (Inaccessible, Degraded, etc.), with further driver-specific elaboration available via reason and message. Finally, the controller-side and node-side reports are kept independent and are hence displayed separately, providing a more holistic view of storage health to consumers.

    To learn more about this enhancement, refer to KEP-1432: Volume Health Monitor.

    Want to know more?

    New features and deprecations are also announced in the Kubernetes release notes. We will formally announce what's new in Kubernetes v1.37 as part of the CHANGELOG for that release.

    Kubernetes v1.37 release is planned for Wednesday, August 26th, 2026. Stay tuned for updates!

    You can see the announcements of changes in the release notes for:

    Get involved

    The simplest way to get involved with Kubernetes is by joining one of the many Special Interest Groups (SIGs) that align with your interests.

    If you don't know where to start, join our monthly New Contributor Orientations where we teach the community how the project is structured, and we'll guide you on how to make your first contribution to the project.

  • How the controller-runtime Cache Actually Works, and Why Your Controller Does Not Crash the API Server

    This article has been revised since it was first published, to correct several significant technical inaccuracies in the original text.

    Kubernetes has long been the default platform for distributed workloads, and writing your own controller for it is now a matter of a few hours. The common path — Golang, using kubebuilder on top of controller-runtime — gives you a project scaffold, types, and a reconciler. For typical scenarios that is more than enough. But as soon as load grows or the controller starts behaving in ways you did not expect, a whole class of edge cases shows up. Most of them trace back to the same root cause: a fuzzy mental model of how controller-runtime works inside. If you write Kubernetes controllers in Go, this article should help you build a coherent picture and avoid expensive surprises in production.

    This article walks through the internals of controller-runtime and, along the way, shows which architectural decisions are baked into Kubernetes itself. The starting point is how controllers actually read objects from the Kubernetes API.

    A common misconception goes like this: r.Get() inside Reconcile queries kube-apiserver directly; r.List() returns a fresh, live view of the world; and after r.Update() you can re-read the object and immediately see the new state. In practice the model is the opposite: controller-runtime operates against a local copy of the data populated through list + watch. Reads inside a reconciler cost almost nothing and do not load the control plane even at hundreds of calls per second — but the price of this design is that a controller can quietly consume gigabytes of memory, perform hidden O(n) scans, and regularly trip over stale reads.

    This post is aimed at engineers who already write controllers in Go with controller-runtime but want to consolidate the pieces into a single mental model rather than carry around a bag of isolated observations. The focus is the practical impact on production clusters: memory, network traffic, read consistency, and reconciler behavior.

    TL;DR

    If you take only one idea from this article, take this:

    r.Get() and r.List() inside a reconciler typically do not read from the API server. They read from a local in-memory cache, which the manager warms up with list and then keeps current through watch.

    Almost every other property of the system follows from that one fact:

    • Reads are cheap, but not strongly consistent immediately after a write.
    • Writes go straight to the API server, not through the cache.
    • The size of the local cache and the set of indexes directly drive memory consumption.
    • An incorrectly written List() can silently turn into a linear scan over tens of thousands of objects.
    • APIReader is rarely needed — but in some places you really cannot do without it.

    The rest of the article unpacks why this is so and how the model is wired underneath.

    A bit of context: what a reconciliation loop is

    To avoid arguments about terminology, start with the basic model.

    A controller in Kubernetes lives inside a reconciliation loop: it continuously compares the desired state of an object with the actual state and tries to bring one in line with the other. The idea is described in the original architectural notes on Kubernetes. In practice it looks like this:

    • A user or another controller mutates an object.
    • An event lands in a queue.
    • Reconcile reads the current state.
    • The controller decides what to create, update, or delete.
    • The system produces a new event and the loop repeats.

    What matters here is not that the controller "does something" — it is where it learns about changes from and where it reads state from. That is exactly where the cache comes in.

    On a live cluster, the easiest way to see this in action is:

    kubectl get pods --watch

    In watch mode, kubectl subscribes to the same event stream that controllers consume. You create or delete a Pod and you see not a single "final" object but a chain of states: the scheduler assigns a node, the kubelet updates status, other controllers contribute their changes. Kubernetes controllers do not poll continuously — they consume an event stream and maintain a local state that is kept current.

    For a visual walkthrough, see Reconciliation loop pattern in visual representation, a talk that shows how the reconciliation loop plays out on a real Pod and the states it passes through.

    Why the cache exists in controller-runtime at all

    Imagine the simplest possible controller:

    func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
 var pod corev1.Pod
 if err := r.Get(ctx, req.NamespacedName, &pod); err != nil {
 return ctrl.Result{}, err
 }
 //... meaningful logic...
}

    Looks straightforward. But what happens when you call r.Get? Does it fire an HTTP request at the API server? If it did, picture the scene: a dozen controllers, each issuing a get and a list per reconcile, with hundreds of reconciles per second. The API server and etcd would be writing each other farewell letters within minutes.

    To prevent that, Kubernetes was built around a watch model rather than polling from the very beginning. The standard mechanism works like this: a client takes a snapshot of the slice of the world it cares about once, then subscribes to a stream of changes and keeps a local copy current. This is the list + watch pattern, and there is no "what is in the world right now?" loop anywhere in it.

    This idea has lived in client-go since the very first controllers in kube-controller-manager. controller-runtime wraps it in a friendly framework so that you do not have to glue the Reflector, the delta queue, and the Indexer together yourself (more on those below).

    So when people talk about "the controller-runtime cache", they are not talking about a clever optimization. They are describing the foundation of the entire model: you read from memory, you write to the API server, and you receive feedback through a watch.

    The rest of this article walks through how each piece is wired up.

    Glossary

    A few terms collected up front, so you do not have to jump back and forth later. Skim or skip if any of them are already familiar.

    • GVK (GroupVersionKind) — the triple that uniquely identifies an API type in Kubernetes: group, version, and kind, for example apps/v1/Deployment. Almost every API in controller-runtime works in terms of GVK rather than the name you would type in kubectl (such as deployments).

    • resourceVersion — a monotonic counter that the API server tracks automatically, and that changes every time the object is written. Although it's a decimal number, the field is represented as a string. Resource versions serve two main purposes. The first one, you can use them for optimistic concurrency control (for example: on update, the API server checks that the resourceVersion you provide matches the one in etcd, otherwise it returns 409 Conflict). The second place you are going to see resource versions in your controller is to resume a watch. Read watch bookmarks to learn more.

    • Manager — the ctrl.Manager object in controller-runtime. This is what your controller constructs in main.go and runs through mgr.Start(ctx). It orchestrates everything: it owns the shared cache, builds the client, starts controllers, webhooks, the healthz endpoint, and other runnables. A single process usually has exactly one manager, with many controllers living inside it.

    • Informer — an entity from client-go that maintains a watch on a single GVK, keeps an indexed local store, and dispatches events to subscribers. In controller-runtime an informer is created automatically when you register Watches(...) or perform the first Get/List on a given type.

    • Store — the in-memory backing store of an informer, where the objects themselves live. Each informer in controller-runtime has its own store.

    • ResourceEventHandler — an interface with three methods: OnAdd, OnUpdate, OnDelete. The informer calls them for every event delivered through the delta queue. The store is always updated before the handler runs, so a handler never sees an indexer that is behind its own event. Subscribers (your controllers) register handlers like this and learn about changes through them.

    • workqueue — a queue of reconcile requests (namespace/name) with deduplication and rate limiting. On every event the controller enqueues a request; workers pop them one at a time and hand them to Reconcile as a ctrl.Request.

    • Predicate — a controller-side filter. A predicate decides whether an event should be enqueued at all (for example, "react only to changes in spec, ignore status").

    With those in hand, you can dive in.

    Anatomy: what lives under the cache package

    If you peek into sigs.k8s.io/controller-runtime/pkg/cache, you will see that it is a thin wrapper over k8s.io/client-go/tools/cache. The same primitives that power the rest of Kubernetes live underneath:

    • Reflector — keeps a watch open against the API server and writes incoming changes into a queue as deltas. A delta is a record of the form "object X received an Added / Updated / Deleted event, and here is its new version". Effectively a single line in a change log.

    • The delta queue (RealFIFO, or DeltaFIFO on older client-go) — holds those deltas in arrival order until the informer processes them.

    • Indexer (Store) — the in-memory object store, plus the indexes built over it.

    • SharedIndexInformer — the conductor that ties everything together and dispatches events to subscribers — your controllers and any other observers.

    At a glance the pipeline looks like this:

    Pipeline diagram: API server to Reflector to the delta queue to Indexer to Event handlers

    Now walk through each link.

    Reflector and resourceVersion

    Within the cache, the Reflector is the only component that talks to the API server. It has exactly two jobs: fetch the initial snapshot at startup, then keep a watch open from there on. (Writes and APIReader reads bypass the cache entirely and reach the API server on their own — more on those later.)

    This is where the resourceVersion earns its keep. Along with the objects, the API server reports the version at which the snapshot was produced. The Reflector then says to the API server, "open a watch from version X", and receives a stream of events for everything that happened after that version. That is the basis of consistency: there is no gap between the snapshot and the stream, because the stream resumes exactly where the snapshot ended.

    That snapshot no longer arrives as a separate list call by default. Current versions use a streaming list instead: the Reflector opens the watch with sendInitialEvents=true, and the API server begins the stream with synthetic ADDED events for the whole current state before switching to live changes. One request instead of two, and a plain list as the fallback. The pattern is unchanged — snapshot, then stream — which is why this article keeps saying list + watch, the way the Kubernetes documentation does.

    If the connection drops, the Reflector reconnects with the last known resourceVersion. If the API server replies with 410 Gone ("that version is no longer in the history, you are too far behind"), the Reflector fetches a fresh snapshot and starts over. This is called a relist, and it does not happen on a schedule — only in those failure scenarios.

    The delta queue

    This piece is worth pausing on, and it is also the piece that changed most recently.

    Historically the buffer between the Reflector and the rest of the informer was DeltaFIFO, which held deltas in a map keyed by namespace/name: deltas for one object accumulated in a slot, Pop() handed back the whole slice for that key at once, and a dedupDeltas helper collapsed consecutive Deleted entries. If you have read about informer internals before, that is probably the picture you are carrying.

    That is no longer the default. Shared informers now use RealFIFO, and since client-go 1.36 DeltaFIFO cannot be switched back on at all. Which version you compile against is what decides this, not the version of the cluster you point at. The new queue is deliberately simpler — a flat, strictly ordered slice of deltas:

    type RealFIFO struct {
 //...
 items []Delta
}

    Its own documentation states the design goal plainly: every notification from the Reflector is passed, in order, through Pop. Which means:

    1. Order is preserved globally, not just per object. Deltas come out in exactly the sequence they arrived.
    2. One Pop, one delta. There is no per-key slot and no slice — Pop takes items[0]. (There is also a PopBatch for processing several deltas in one pass, on by default since client-go 1.35, but it is a batching optimization, not a merge: each delta is still delivered.)
    3. No deduplication whatsoever. RealFIFO has no dedupDeltas equivalent. Nothing is collapsed — including consecutive deletes, and including intermediate states.

    So the worked example gets simpler than it used to be. Suppose three events for default/my-deploy arrive in quick succession:

    1. Added — the Deployment is created (say, with spec.replicas=1).
    2. Updated — somebody bumps spec.replicas to 2.
    3. Updated — and immediately to 3.

    All three are appended to the queue and popped one at a time, and the informer dispatches them in order: first OnAdd, then two OnUpdate calls (the intermediate 1→2, then the final 2→3). The event handler runs three times, no shortcuts.

    The store is written before the handlers are notified, and handler delivery is asynchronous — the informer writes the indexer, then hands the notification to a per-subscriber buffer that the subscriber's own goroutine drains later. So your handler never sees an indexer lagging behind its own event, but it can see one that has moved well past it. Handling the 1→2 update, a Get from the cache can legitimately return 3 — or NotFound, if the object has since been deleted. Never treat the object in the store as "the state at the time of my event".

    Deduplication does exist — but it lives one layer up, in the controller's workqueue, and with RealFIFO that is now the only place it happens. The mechanic is straightforward: for each delta, the controller's event handler extracts the namespace/name key from the object and enqueues it. Re-inserting the same key silently coalesces with the existing entry; the workqueue does not care about the object itself.

    A concrete picture: you create a Pod. Within a second or two a flurry of Updated deltas arrives — the scheduler assigns a node, the kubelet sets Pending, then ContainerCreating, Running, Ready. Five deltas in a row, and the event handler fires on every one of them — but throughout this window the workqueue holds a single entry with the key default/my-pod. By the time Reconcile pops it, the cache already holds the final state, and Reconcile runs once.

    So you get two layers with cleanly separated responsibilities:

    • The delta queue — an ordered stream of change facts, delivered one at a time and without merging. Its job is to tell consumers everything that happened, in the right order.
    • workqueue — a queue of keys with deduplication. This is the layer that collapses "ten updates in a row → a reconcile or two".

    If you keep that two-layer picture in your head, it becomes clear why a flood of events against a single object barely affects controller throughput — the workqueue absorbs them.

    Indexer: the local copy of the cluster

    The Indexer, backed by a ThreadSafeStore, is the local copy of the cluster. Underneath it is a plain map[string]interface{} keyed by namespace/name, plus a single sync.RWMutex, plus a dictionary of registered indexes (covered in their own section below).

    An uncontended r.Get is cheap: a map lookup followed by a DeepCopy of the object. The part of that structure that matters most at scale, though, is not the map — it is the one sync.RWMutex, which guards the store and every index at once. Readers hold it for shared access, the informer needs it exclusively to write, so the two genuinely compete: a List holds the read lock while it walks every object of that kind, and the next store write waits behind that walk. This was a real bottleneck in kube-controller-manager at scale (kubernetes#130767); recent client-go releases hold the write lock for much less time.

    SharedIndexInformer and subscriptions

    A SharedIndexInformer fuses the Reflector, the delta queue, and the Indexer together and exposes two interfaces to the rest of the world:

    • Read objects directly from the indexer.
    • Register a ResourceEventHandler and receive notifications for every event coming out of the queue — OnAdd, OnUpdate, OnDelete. The store is written before the handler call, so by the time your handler runs the indexer already reflects that event, and possibly later ones too.

    "Outside" here means your controllers. When a controller registers Watches(...), under the hood it asks the informer: "add a handler that, on every change, enqueues the key into my workqueue". The controller's workers then pop keys one at a time and call your Reconcile(ctx, ctrl.Request{NamespacedName:...}).

    The keyword in the name is Shared. The manager creates one informer per GVK, and every controller, webhook, and event source within that manager subscribes to it:

    Shared informer diagram: a single list / watch per GVK, feeding multiple subscribers

    In other words: an informer is the thing that subscribed to Pods once, holds them locally, and serves every interested party in the process. From the API server's perspective, that is one snapshot and one watch per GVK, regardless of how many reconcilers live inside your process.

    What happens at startup and on the very first r.Get

    Step by step, here is what happens between the moment the manager starts and the first r.Get inside your reconciler:

    1. The manager's mgr.Start(ctx) brings up every registered informer.
    2. For each GVK, the Reflector fetches a full snapshot: every object of that type that falls within your scope.
    3. The snapshot is loaded into the informer's store, registered indexes are rebuilt, and the informer is marked as synced.
    4. The same stream then continues as an ordinary watch from the resourceVersion the snapshot synced to.
    5. Only then does the controller start invoking Reconcile — specifically, once every source it owns reports synced, which includes its event handlers having processed the initial snapshot. Until that point, workers do not drain the workqueue, even if events have already started piling up.

    So "the reconciler is running but the cache is still empty" is not a state you can observe — the warm-up happens before the first Reconcile. (The one exception: a Get for a type nothing registered a watch for starts a new informer on the spot, and blocks until it is warm.)

    What happens during the first r.Get? Suppose your reconciler contains:

    var obj appsv1.Deployment
err := r.Get(ctx, req.NamespacedName, &obj)

    Under the hood it boils down to roughly this:

    item, exists, err := indexer.GetByKey("default/my-deploy")
if !exists {
 return apierrors.NewNotFound(...)
}
// DeepCopy into obj

    No HTTP, no TLS, no protobuf serialization, no etcd. A map lookup plus a deep copy of the object, and no I/O at all.

    To repeat, because it matters: even the very first Get for a registered type reads from a fully warmed-up, fully indexed snapshot. There is no "first time slow, then fast".

    Note: This applies specifically to mgr.GetClient(). If for some reason you need to read objects before mgr.Start() (for example, during initialization), use mgr.GetAPIReader(), which goes straight to the API server. The regular client does not hand you an empty result at that point — it fails fast with ErrCacheNotStarted. More on this later.

    Client ≠ Cache: read from memory, write to the API server

    Another point that often gets lost. client.Client in controller-runtime is a composite object:

    • Reads (Get, List) go through the cache.
    • Writes (Create, Update, Patch, Apply, Delete, DeleteAllOf) go straight to the API server.

    This is not a hack — it is a deliberate design choice:

    • Reads are frequent; they should be cheap.
    • Writes are rare; they should be exact.
    • Writing through the cache would invite split-brain — the local copy thinks the change went through, while the API server has already rejected it.

    It is worth dwelling on "should be exact". This is where resourceVersion shows up again.

    When you read an object from the cache, you do not get its current state in etcd — you get the state as the Reflector last observed it. That state carries a resourceVersion. You then mutate the object and call r.Update(ctx, &obj). The request goes to the API server right now, and the API server checks:

    • Does the resourceVersion in your PUT match the resourceVersion in etcd? Yes — write it.
    • No, etcd already has a newer one? Reply with 409 Conflict — somebody beat you to it.

    This is optimistic concurrency control. No real locks are taken; everybody writes in parallel; but only one of the racing Update calls wins — the one that arrives with the current version. Everyone else gets a 409 and is expected to re-read and try again.

    Why does this matter for the cache? If you naively send a PUT with "your" resourceVersion from the cache and somebody has updated the object since you read it, you will get 409. That is not a bug. It is exactly the protection the system is supposed to give you. Writing without the resourceVersion check (via Patch without an optimistic lock, or via Server-Side Apply) is also possible, but that is a separate conversation.

    The "write → visibility" cycle now looks like this:

    Write visibility diagram: client.Update to API server to watch event to cache

    Between "you executed Update" and "the cache reflects the new state" there is a window — usually milliseconds, but with no guaranteed upper bound. Inside that window, an r.Get for the same object returns the previous version. The next section is essentially a list of mistakes that grow out of that window.

    Common mistakes that everyone makes

    Mistake 1: expecting read-after-write

    A familiar pattern:

    obj.Spec.Replicas = ptr.To(int32(5))
if err := r.Update(ctx, &obj); err != nil {
 return ctrl.Result{}, err
}

// re-read and confirm it is now 5
var fresh appsv1.Deployment
_ = r.Get(ctx, key, &fresh)
fmt.Println(*fresh.Spec.Replicas) // surprise: 3

    This is not a controller-runtime bug. It is a property of an eventually consistent system: the cache catches up asynchronously, through the watch.

    The right pattern is to never rely on instant freshness. Reconcile must be idempotent and must always look at the current state. If it does not match the desired state, the next reconcile fixes it. You do not need to "wait 100ms" or "re-trigger". You need to write the logic so that one or two extra invocations break nothing.

    If a stale read is a genuine correctness problem for you, a live read does not fix it — a concurrent write can be mid-commit anyway. See the controller-runtime FAQ for patterns that do.

    Mistake 2: DeepCopy and who owns the memory

    To make sense of this, a quick word on event mechanics inside a controller. When you register a source via Watches(...), two layers sit between the indexer and your Reconcile:

    • Predicate — the filter. It looks at an event (CreateEvent, UpdateEvent, DeleteEvent, GenericEvent) and decides whether to pass it through.
    • EventHandler — the transformer. It receives the object and turns it into one or more ctrl.Request values that go into the workqueue. The classic EnqueueRequestForObject enqueues the namespace/name of the current object.

    Here is the critical part. Predicates and handlers receive the same objects that live in the informer's shared store. The same *corev1.Pod is seen by every controller subscribed to Pods.

    Because Go has no immutable structs, nothing prevents you from doing pod.Labels["foo"] = "bar" directly inside a handler — and that Pod is the one in the store. Raw client-go listers have always worked this way; ThreadSafeStore's own documentation puts it bluntly: you must not modify anything returned by Get or List as it will break the indexing feature. Patch a status "for convenience" in a handler and you break the world view of an unrelated controller next door.

    The cache-backed client from controller-runtime shields you from that on the read path: Get and List deep-copy by default, and have since its earliest releases. You can opt out with UnsafeDisableDeepCopy, which is named that way on purpose. The event path is not shielded — there is no DeepCopy anywhere between the informer and your predicate. The simple rule:

    • Anything you receive from r.Get / r.List is yours; mutate freely.
    • Anything you receive in a Predicate or an EventHandler is shared, not yours. If you must mutate it, call obj.DeepCopy() first; otherwise you are silently corrupting the cache for every other controller subscribed to the same type.

    A concrete review heuristic: if predicate.Funcs{UpdateFunc:...} or handler.EnqueueRequestsFromMapFunc(...) contains expressions like e.ObjectNew.SetLabels(...) or obj.Status.X = Y, stop and ask whether a DeepCopy is missing before that mutation.

    Mistake 3: resync is not relist

    An informer has a resync period (cache.Options.SyncPeriod, 10 hours by default in controller-runtime), and many people read it as meaning: rebuild the cache from the API server every n hours, fetching every resource once again.

    It does not. A resync does not perform a list. It re-emits everything currently in the indexer back through the delta queue, and the informer dispatches an update per object, calling OnUpdate(old, old) for each one. This is for controllers that manage state outside the Kubernetes API (a cloud provider resource, for example): out-of-band changes produce no watch event, and a periodic resync is the only way to notice them. It generates no traffic to the API server.

    One caveat before you rely on resync as a safety net: because both sides of the synthetic update are the same object, predicates that compare old and new — such as GenerationChangedPredicate — will drop it.

    A real relist happens only in two cases: when the watch died with 410 Gone, and when you explicitly recreate the informer.

    Mistake 4: do not confuse RequeueAfter with a timer

    A small note that often saves time. Sometimes you want to wait inside a reconciler — "we just called the provider's API; if it is not ready yet, retry in a minute". The temptation is to spin up time.Sleep or your own goroutine.

    Resist it. controller-runtime already provides a built-in mechanism:

    return ctrl.Result{RequeueAfter: 30 * time.Second}, nil

    The controller puts your req back into the workqueue with a delayed trigger 30 seconds out. If a real event for the same object arrives within that window, the reconcile fires immediately, without waiting for the timer (the key is deduplicated in the queue). This is both cheaper and more correct than a hand-rolled timer: you do not hold a worker, and you do not risk missing a real event.

    cache + index = almost SQL

    Now you get to what is, arguably, the most useful capability of the cache — and the one most controllers leave unused.

    By default, a List from the cache looks like this:

    var pods corev1.PodList
_ = r.List(ctx, &pods)
for _, p := range pods.Items {
 if p.Spec.NodeName == "node-1" {
 // do something
 }
}

    It works — until the cluster has 50,000 Pods and reconciles run hundreds of times per second. Then the loop turns slow: every trigger walks all 50,000 Pods under the store's read lock, then deep-copies each one after the lock is released, doing O(n) work per reconcile, and it is the walk, not the copying, that blocks writers into the store.

    The Indexer in client-go can do much better. You declare up front which field you want to index on:

    // Index by spec.nodeName for Pods
if err := mgr.GetFieldIndexer().IndexField(
 ctx,
 &corev1.Pod{},
 "spec.nodeName",
 func(obj client.Object) []string {
 pod := obj.(*corev1.Pod)
 if pod.Spec.NodeName == "" {
 return nil
 }
 return []string{pod.Spec.NodeName}
 },
); err != nil {
 return err
}

    Two things about that call are worth making explicit, because the tidy example hides them behind a convention.

    The index name is arbitrary. That second argument, "spec.nodeName", is only a string key the index is registered under. controller-runtime does not parse it as JSONPath and does not check it against the object's schema — you could write "by-node" or "xyzzy" and it would behave identically. The only rule is that the exact same string comes back in MatchingFields at query time. Naming the index after the field it happens to read is a readability convention, nothing more.

    But remember: this only works for reads served from the cache.

    The indexed value is computed, not read. The function returns whatever strings you build; they need not be the verbatim contents of any single field. You can lowercase a value, join several fields into one composite key, bucket a timestamp (the time-bucket trick below does exactly this), or emit a string that appears nowhere in the object literally. Whatever the function returns becomes a key in the inverted dictionary, and a MatchingFields lookup for that exact key is what finds the objects again. The only constraint is that the value has to be derivable from the object you are indexing.

    What is an inverted index? The term comes from search engines. Normally you have documents and each document has a list of words in it. "Inverted" means the relationship is flipped: a dictionary in which the key is a word and the value is the list of documents that contain it. Same idea here: the key is the value of a field (for example, node-1), and the value is the list of object keys whose field has that value:

    map["node-1"] = {"default/pod-a", "kube-system/pod-b",...}
map["node-2"] = {"default/pod-c",...}

    What the indexer does:

    • On every incoming event (ADDED, MODIFIED, DELETED), the indexer runs the object through your indexing function, gets back the set of index keys, and updates the inverted dictionary. If a Pod migrates from node-1 to node-2, the node-1 key loses its reference to it and the node-2 key gains one.
    • By the time you call List, the index is already current. You do not pay for a rebuild at query time — no scan over all objects, no dictionary reconstruction. All the work was done up front, at the moment the object changed.

    And now you can write:

    var pods corev1.PodList
_ = r.List(ctx, &pods,
 client.MatchingFields{"spec.nodeName": "node-1"},
)

    This is not "fetch the full list, then filter". It is a lookup in the inverted index → a ready set of keys → a fetch of the corresponding objects. A different code path entirely.

    The comparison to SQL is more accurate than it might look at first:

    SQL controller-runtime
    CREATE INDEX idx_node ON pods(node_name) IndexField(&Pod{}, "spec.nodeName", fn)
    SELECT * FROM pods WHERE node_name = 'node-1' List(&pods, MatchingFields{"spec.nodeName": "node-1"})
    SELECT * FROM obj WHERE owner_uid = $1 List(&list, MatchingFields{"metadata.ownerReferences.uid": uid}) (requires an IndexField for that field)

    Note the last row: MatchingFields does not make magic out of thin air. For every field you want to look up via MatchingFields you need a corresponding IndexField registered during manager setup. Without one, controller-runtime rejects the query and returns an error.

    A few things worth keeping in mind:

    • Equality only. No range queries, no LIKE, no sorts, no aggregates. If you need "everything older than five minutes", either do a regular List and filter in code, or use a time-bucket trick: instead of indexing the precise time.Time, index a rounded value (for example, now.Truncate(5*time.Minute).Format(...)). You can then select objects by a specific window.

    • MatchingLabels is not an index. Many people assume that since label-based lookups are so common, there must be an optimization for them. There is not: ThreadSafeStore keeps no separate label dictionary. The only indexes the cache has are the namespace index and the field indexes you register yourself.

      Two separate things follow, and they get conflated. The walk really is O(n): with no label index, List(..., MatchingLabels{...}) still visits every cached object of that kind, or whatever subset the namespace or a field index already narrowed it to. But the selector is evaluated before the deep copy, so objects that do not match are skipped without ever being copied. Against 50,000 Pods with ten matches, that is 50,000 cheap comparisons and ten expensive copies — not 50,000 copies. Which is why a label-filtered List is still much better than fetching everything and filtering in your own code afterwards: same walk either way, and the filtered version skips the copies you were going to throw away.

      So MatchingLabels is fine to use — just do not expect it to make the traversal cheaper. To shrink the candidate set itself, use a namespace or a field selector backed by a registered IndexField. To avoid holding the objects at all, filter at cache population time via cache.ByObject{Label:...} or DefaultLabelSelector, which pushes the selector down to the watch itself — covered in the next section on selective caches.

    • An index costs memory. An index maps each indexed value to a set of namespace/name keys — not to copies of the objects. Cheap per object, not free, and still not a reason to index everything in sight speculatively.

    • You can only index data that is in the object itself. You cannot index a Pod by "has a related PVC with such-and-such flag". Either store that bit in the Pod itself, or index the PVC, not the Pod.

    Note: An index is built at registration time and is populated as part of the initial snapshot. By the time the first Reconcile runs, List with MatchingFields already works — the index is not built lazily. (Get never consults a field index; it is a direct lookup by store key.)

    Selective cache: do not pull the whole cluster into your controller

    By default, an informer pulls every object of its type from every namespace. For Pod, Secret, ConfigMap, and Event in a large cluster, that is a multi-gigabyte surprise delivered on the first list at startup.

    It hurts especially with:

    • Secrets, because Helm stores release state in them (helm.sh/release.v1.*), and those secrets are often a hundred kilobytes each.
    • v1.Node objects, whose status.images field carries a list of every image that has ever landed on the node — tens of kilobytes per node in busy clusters.
    • Events, which can be very numerous and which you almost certainly do not need cached at all.

    In controller-runtime, caching policy lives in cache.Options, passed when constructing the manager:

    mgr, err := ctrl.NewManager(cfg, ctrl.Options{
 Cache: cache.Options{
 ByObject: map[client.Object]cache.ByObject{
 // Cache Secrets only from your own namespace, and only by label
 &corev1.Secret{}: {
 Namespaces: map[string]cache.Config{
 "my-controller": {},
 },
 Label: labels.SelectorFromSet(labels.Set{
 "app.kubernetes.io/managed-by": "my-controller",
 }),
 },
 // Cache all Pods, but trim noise on the way into the store
 &corev1.Pod{}: {
 Transform: func(obj any) (any, error) {
 pod := obj.(*corev1.Pod)
 pod.ManagedFields = nil
 return pod, nil
 },
 },
 },
 },
})

    A subtle point: this is a manager-level setting and it affects every controller in the process that reads the corresponding type. If you narrow the cache for Secrets to a single namespace and another controller in the same binary needs all secrets in the cluster, that controller will not see them. Before you tighten the scope, audit who else is reading the type.

    A short tour of the options:

    • Namespaces restricts the visible scope. If your controller only manages its own namespace, there is no reason to keep other people's objects in memory.
    • Label / Field become parameters of the watch itself. The API server only sends matching objects, saving network and memory.
    • Transform is invoked before the object lands in the store. It is the perfect place to drop managedFields, oversized annotations, or the binary data of ConfigMaps that you do not need.
    • DefaultLabelSelector / DefaultNamespaces apply the same restriction globally, when every type needs the same scope.

    Caveat: A selector limits what is cached, not what exists. If an object does not match your selector, then as far as your controller is concerned, it does not exist in either Get or List. This bites people: somebody mislabels a single Secret and then spends half a day figuring out why their controller "cannot see it".

    Metadata-only: when spec and data are not needed

    A separate pattern: you need to know that an object exists, but you do not need its spec or data. Typical examples: a controller that waits for a Secret with a particular name to appear but never reads it; one that counts PersistentVolume objects by the topology.kubernetes.io/zone label; one that reacts to ConfigMap objects in a namespace by name and does not care about contents.

    Caveat: PartialObjectMetadata by definition gives you nothing from spec or status — only ObjectMeta. So you cannot filter through it on spec fields (such as a PersistentVolume's storageClassName or a Pod's nodeName); those fields do not exist in the local copy. Everything covered by metadata-only is labels, annotations, ownerReferences, finalizers, creationTimestamp, and the rest of metadata.

    For this case there is PartialObjectMetadata:

    var list metav1.PartialObjectMetadataList
// controller-runtime infers the list shape from the variable type.
list.SetGroupVersionKind(schema.GroupVersionKind{
 Group: "",
 Version: "v1",
 Kind: "Secret",
})
if err := r.List(ctx, &list, client.InNamespace("my-ns")); err != nil {
 return err
}

    Under the hood this is a separate watch that asks the API server for metadata only. The store keeps such objects without Data, Spec, or Status — only ObjectMeta. For Secrets the memory difference can reach an order of magnitude.

    APIReader: when the cache is not enough

    mgr.GetAPIReader() returns a client.Reader that goes straight to the API server, around the cache. When you actually need it:

    • A one-off read of a resource for which you do not maintain an informer. Spinning up a watch for a single operation is expensive.
    • Reads before mgr.Start(), for instance during initialization. At that point the cache-backed client fails with ErrCacheNotStarted rather than returning data.
    • Paginated traversal of large result sets through client.Continue. The cache-backed client rejects Continue with an explicit error, and honors Limit only as truncation — an arbitrary N objects, not a stable "first N". For genuine pagination you need APIReader or a direct client of your own.

    The price is a real network request, and it is easy to underestimate: on top of the round trip you pay to deserialize whatever comes back, which for a large collection is not cheap. So the trade is less obvious than it looks — reading from the API server is not automatically cheaper just because it avoids keeping objects in memory. Measure before you "optimize" a cached read into a live one. One thing to avoid outright: do not build "look in the cache, and if missing, fall back to the API" logic. That is exactly the split-brain pattern the cache is meant to protect you from.

    Disabling the cache for a type entirely

    If you do not need a local cache for a given type at all — say, the type is "fat", read rarely, and the list + watch overhead is not worth paying — you can tell the manager not to cache it. This is configured through client.Options.Cache.DisableFor:

    mgr, err := ctrl.NewManager(cfg, ctrl.Options{
 Client: client.Options{
 Cache: &client.CacheOptions{
 DisableFor: []client.Object{
 &corev1.Secret{},
 },
 },
 },
})

    With this configuration, mgr.GetClient().Get(...) and List(...) for Secret go straight to the API server, bypassing the cache. No informer is started for that type, which means no list at startup and no permanent memory pressure from a store. That also means no events: nothing will trigger your controller when such an object changes. If you need those triggers, pair the direct reads with a metadata-only watch. This is a more radical alternative to APIReader: where APIReader is reached for ad hoc, individual requests, DisableFor turns the cache off for the type wholesale.

    Real-world projects use this: external-secrets, for instance, has flags that disable caching for Secrets and ConfigMaps, trading memory for API traffic.

    Aside: If you want to avoid a watch on the API server entirely, you can feed the controller events from a source of your own design, bypassing list + watch. In controller-runtime this is done with WatchesRawSource / source.Channel: you can wire the controller to events from any place — an internal queue, a kubelet, a custom watch. Niche, but a perfectly valid pattern when the API server should not be touched.

    Good practices

    A short checklist worth running through before you ship a controller into a live cluster:

    • Constrain cache scope (Namespaces, Label, Field selectors), especially for "fat" types: Secret, ConfigMap, Event, Pod, Node.
    • Remember that a constrained cache acts as if everything outside its scope does not exist. A mislabeled object "disappears" with no error anywhere.
    • Add a Transform for objects whose heavy fields you do not need — ManagedFields alone consume a noticeable share of memory.
    • Add an IndexField for every List that uses MatchingFields. Without a matching index the query does not silently degrade — it fails with an error.
    • Do not mutate objects you receive in an EventHandler or a Predicate without a prior DeepCopy. Mutations to the store break neighboring controllers silently and persistently.
    • Make Reconcile idempotent. It must behave correctly even if it is invoked five times in a row with no real change.
    • Do not expect read-after-write from the cache immediately after Update. The cache lags during that window.
    • When the cache cannot serve a read (initialization, deliberately uncached types), use APIReader, not the regular client.
    • Use PartialObjectMetadata for types where you only need metadata. It can save gigabytes.
    • Do not call mgr.GetClient() before mgr.Start(). The cache is not running yet, so reads fail with ErrCacheNotStarted instead of returning data. Use mgr.GetAPIReader() when you genuinely need to read during initialization.
    • For deferred actions, use RequeueAfter, not time.Sleep and not your own goroutines.

    Wrapping up

    In one breath:

    • The cache in controller-runtime is not an optimization, it is the operating model. Under the hood it is Reflector + a delta queue + Indexer — exactly the same primitives that power Kubernetes itself.
    • r.Get and r.List go to memory; Create, Update, Patch, and Delete go straight to the API server. Feedback flows in through the watch.
    • IndexField plus MatchingFields turn the cache into a near-complete query engine with inverted indexes.
    • Namespaces, selectors, PartialObjectMetadata, and Transform are the levers that control how much memory and traffic you actually consume.
    • APIReader bypasses the cache for the rare read the cache cannot serve — but it is not a fix for staleness races; see the controller-runtime FAQ.

    And the single sentence to remember: r.Get inside a reconciler reads from memory, not from the API server — not even the first time. The exceptions are the ones you opt into yourself: APIReader, Cache.DisableFor, and unstructured reads. Once that becomes a reflex, half the questions on controller code reviews answer themselves.

  • Building a Custom Metrics Exporter for Kubernetes

    Kubernetes ships with built-in awareness of CPU and memory, but most real-world scaling decisions depend on signals that live entirely outside that narrow window: how many messages are waiting in a queue, how long the last batch job took, how many active WebSocket connections a pod is holding. When the built-in metrics are not enough, a metrics exporter bridges that gap.

    This post walks through writing one from scratch, packaging it as a container, and wiring it into a cluster so that Prometheus — and ultimately the HorizontalPodAutoscaler — can consume it.

    What a metrics exporter actually does

    An exporter is a small HTTP server with a single responsibility: expose application state as text on a /metrics endpoint. Prometheus scrapes that endpoint on a regular interval, stores the time-series data, and makes it available for queries, alerts, and autoscaling rules.

    In some cases you can instrument your application directly — embedding the Prometheus client library and exposing /metrics from within the same process — rather than running a separate exporter. A standalone exporter makes more sense when the data source is external to your application or when you do not control the application code.

    The format Prometheus expects is plain text — one metric per line, with a name, optional labels, and a numeric value. Client libraries handle the serialization for you, so in practice you only need to decide what to measure and call the right function when that value changes.

    Choosing what to measure

    Before writing any code, it helps to decide what kind of signal you are dealing with. The Prometheus data model has three main types:

    • Counters only ever increase. They are the right tool for totals: requests served, jobs processed, errors encountered. Never use a counter for a value that can go down.

    • Gauges represent a current snapshot of a value that can rise and fall freely. Queue depth, active connections, and cache size are all gauges.

    • Histograms record the distribution of observed values, such as request latency. They let you calculate percentiles (p99, p50) rather than just averages.

    Once you know which type fits your signal, choose a name that follows the convention <namespace>_<name>_<unit> in snake_case. A job processor might expose worker_jobs_processed_total (counter), worker_queue_depth (gauge), and worker_job_duration_seconds (histogram). Clear names save everyone debugging time later.

    Setting up the project

    The Go Prometheus client is the most common choice for exporters in the Kubernetes ecosystem, largely because the same library powers most of the official Kubernetes components. Start by creating a module and pulling in the dependency:

    mkdir my-exporter && cd my-exporter
go mod init example.com/my-exporter
go get github.com/prometheus/client_golang/prometheus
go get github.com/prometheus/client_golang/prometheus/promhttp

    Registering metrics

    Create main.go. The first thing to do is declare the metrics and register them with Prometheus's default registry. Registration tells the library that these metrics exist so they appear in the output even before the first observation is recorded:

    package main

import (
 "log"
 "net/http"

 "github.com/prometheus/client_golang/prometheus"
 "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
 jobsProcessed = prometheus.NewCounterVec(
 prometheus.CounterOpts{
 Name: "worker_jobs_processed_total",
 Help: "Total number of jobs processed, partitioned by status.",
 },
 []string{"status"},
 )

 queueDepth = prometheus.NewGauge(prometheus.GaugeOpts{
 Name: "worker_queue_depth",
 Help: "Current number of jobs waiting in the queue.",
 })

 jobDuration = prometheus.NewHistogram(prometheus.HistogramOpts{
 Name: "worker_job_duration_seconds",
 Help: "Time spent processing a single job.",
 Buckets: prometheus.DefBuckets,
 })
)

func init() {
 prometheus.MustRegister(jobsProcessed, queueDepth, jobDuration)
}

    prometheus.MustRegister panics on a duplicate registration, which makes misconfigurations obvious at startup rather than silently at runtime. If you are embedding this exporter inside a library that other packages will also instrument, prefer prometheus.Register and handle the error yourself.

    Collecting real values

    With the metrics registered, the next step is to keep them current. You can either continually update the data as the data change, or run your own internal refresh loop. The pattern below shows a polling loop — a goroutine that periodically reads from whatever data source your application owns and updates the registered metrics. Replace the simulated values with real calls to your database, internal API, or message broker:

    import (
 "math/rand"
 "time"
)

func collectMetrics() {
 for {
 // Replace these with real reads from your application.
 depth := float64(rand.Intn(50))
 queueDepth.Set(depth)

 start := time.Now()
 time.Sleep(time.Duration(rand.Intn(200)) * time.Millisecond)
 jobDuration.Observe(time.Since(start).Seconds())
 jobsProcessed.WithLabelValues("success").Inc()

 time.Sleep(5 * time.Second)
 }
}

    The polling interval (here five seconds) should be shorter than Prometheus's scrape interval so that each scrape sees a fresh value. The default scrape interval in most cluster deployments is fifteen seconds, which gives you comfortable headroom.

    Exposing the endpoint

    Wire the collection loop and the HTTP handler together in main. A /healthz path alongside /metrics gives Kubernetes a liveness probe target without exposing metric data on the health route:

    func main() {
 go collectMetrics()

 http.Handle("/metrics", promhttp.Handler())
 http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
 w.WriteHeader(http.StatusOK)
 })

 log.Println("Listening on :8080")
 if err := http.ListenAndServe(":8080", nil); err != nil {
 log.Fatalf("server error: %v", err)
 }
}

    Verify the output locally before building the image:

    go run .
curl http://localhost:8080/metrics | grep worker_

    You should see three # HELP and # TYPE blocks followed by the current metric values. If those lines appear, the exporter is working correctly and is ready to be containerized.

    Build a container image

    A multi-stage build keeps the final image small and avoids shipping a Go toolchain to production. The first stage compiles a statically linked binary; the second stage copies only that binary into a minimal base. The example below uses Docker, but the same pattern works with any OCI-compatible build tool such as Buildah or Podman:

    FROM golang:1.21-alpine AS builder
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /exporter .

FROM gcr.io/distroless/static:nonroot
COPY --from=builder /exporter /exporter
EXPOSE 8080
ENTRYPOINT ["/exporter"]

    distroless/static:nonroot contains no shell, no package manager, and runs as a non-root user by default, which satisfies most cluster security policies without extra configuration.

    Build and push the image, replacing <registry> with your own registry address:

    docker build -t <registry>/my-exporter:v1.0.0 .
docker push <registry>/my-exporter:v1.0.0

    (Note: Using a CI/CD pipeline to automate this is generally a better pattern than running these commands manually.)

    Deploying to the cluster

    Two manifests are enough to run the exporter: a Deployment that manages the pod lifecycle, and a Service that gives Prometheus a stable address to scrape. (You might prefer to have Prometheus scrape from every Pod; if that makes sense for your use case, then it's OK to configure instead).

    The examples below use the monitoring namespace, which is a common convention when running Prometheus and related components together. Adjust the namespace to match your own cluster setup.

    The Deployment sets conservative resource limits appropriate for a lightweight sidecar-style process, and uses the /healthz route for its liveness probe:

    apiVersion:apps/v1
kind:Deployment
metadata:
name:my-exporter
namespace:monitoring
labels:
app.kubernetes.io/name:my-exporter
spec:
replicas:1
selector:
matchLabels:
app.kubernetes.io/name:my-exporter
template:
metadata:
labels:
app.kubernetes.io/name:my-exporter
spec:
containers:
- name:exporter
image:<registry>/my-exporter:v1.0.0
ports:
- name:metrics
containerPort:8080
livenessProbe:
httpGet:
path:/healthz
port:8080
initialDelaySeconds:5
periodSeconds:10
resources:
requests:
cpu:50m
memory:32Mi
limits:
cpu:100m
memory:64Mi

    The Service names the port metrics, which the ServiceMonitor in the next section will reference by that name:

    apiVersion:v1
kind:Service
metadata:
name:my-exporter
namespace:monitoring
labels:
app.kubernetes.io/name:my-exporter
spec:
selector:
app.kubernetes.io/name:my-exporter
ports:
- name:metrics
port:8080
targetPort:metrics

    Apply both:

    kubectl apply -f deployment.yaml -f service.yaml

    Telling Prometheus where to look

    How you configure scraping depends on how Prometheus was installed.

    Option 1: Prometheus Operator (ServiceMonitor)

    If you installed Prometheus using the Prometheus Operator or the kube-prometheus-stack Helm chart, the operator must be running in your cluster before you create a ServiceMonitor. The release label must match the label selector configured on your Prometheus resource — kube-prometheus-stack is the default for a standard Helm install:

    apiVersion:monitoring.coreos.com/v1
kind:ServiceMonitor
metadata:
name:my-exporter
namespace:monitoring
labels:
release:kube-prometheus-stack
spec:
selector:
matchLabels:
app.kubernetes.io/name:my-exporter
endpoints:
- port:metrics
interval:15s
path:/metrics

    Option 2: Annotation-based discovery

    If your Prometheus uses annotation-based pod discovery instead, you will need a matching scrape_config rule in your Prometheus configuration — check with whoever manages your Prometheus installation to confirm it is in place.

    You can add the following three annotations to the Pod template regardless of which scraping method you use. They are ignored by the Prometheus Operator but picked up automatically by annotation-based setups:

    annotations:
prometheus.io/scrape:"true"
prometheus.io/port:"8080"# omit if not using annotation-based discovery
prometheus.io/path:"/metrics"# omit if not using annotation-based discovery

    If you are unsure which setup your cluster uses, the ServiceMonitor approach is more explicit and easier to debug.

    Verifying the scrape

    Port-forward to the Prometheus service and open the targets page to confirm the exporter has been discovered:

    kubectl port-forward svc/prometheus-operated 9090 -n monitoring

    Navigate to http://localhost:9090/targets. The my-exporter target should appear with state UP. If it shows DOWN, check that the ServiceMonitor's release label matches and that the pod is running:

    kubectl get pods -n monitoring -l app.kubernetes.io/name=my-exporter
kubectl describe servicemonitor my-exporter -n monitoring

    Once the target is healthy, run a quick query in the expression browser to confirm data is flowing:

    rate(worker_jobs_processed_total{status="success"}[2m])

    A non-zero result here means the full pipeline is working: your application is producing data, Prometheus is scraping it, and the time-series are stored and queryable.

    What comes next

    A working exporter is the foundation, not the destination. The natural next step is surfacing these metrics to the HorizontalPodAutoscaler so that your workload scales on the signals that actually drive load, not just CPU. That requires a metrics adapter — the Prometheus Adapter is the most widely deployed option — which registers your custom metrics with the Kubernetes Custom Metrics API. Once registered, any HorizontalPodAutoscaler in the cluster can reference worker_queue_depth or worker_jobs_processed_total directly in its metrics block.

    For a walkthrough of that setup, see Autoscaling on multiple metrics and custom metrics. For a catalog of ready-made exporters covering databases, message brokers, and cloud services, the Prometheus exporters and integrations page is a good starting point.