Kubernetes News

The Kubernetes blog is used by the project to communicate new features, community reports, and any news that might be relevant to the Kubernetes community.
The Kubernetes project logo

  • Building a Custom Metrics Exporter for Kubernetes

    Kubernetes ships with built-in awareness of CPU and memory, but most real-world scaling decisions depend on signals that live entirely outside that narrow window: how many messages are waiting in a queue, how long the last batch job took, how many active WebSocket connections a pod is holding. When the built-in metrics are not enough, a metrics exporter bridges that gap.

    This post walks through writing one from scratch, packaging it as a container, and wiring it into a cluster so that Prometheus — and ultimately the HorizontalPodAutoscaler — can consume it.

    What a metrics exporter actually does

    An exporter is a small HTTP server with a single responsibility: expose application state as text on a /metrics endpoint. Prometheus scrapes that endpoint on a regular interval, stores the time-series data, and makes it available for queries, alerts, and autoscaling rules.

    In some cases you can instrument your application directly — embedding the Prometheus client library and exposing /metrics from within the same process — rather than running a separate exporter. A standalone exporter makes more sense when the data source is external to your application or when you do not control the application code.

    The format Prometheus expects is plain text — one metric per line, with a name, optional labels, and a numeric value. Client libraries handle the serialization for you, so in practice you only need to decide what to measure and call the right function when that value changes.

    Choosing what to measure

    Before writing any code, it helps to decide what kind of signal you are dealing with. The Prometheus data model has three main types:

    • Counters only ever increase. They are the right tool for totals: requests served, jobs processed, errors encountered. Never use a counter for a value that can go down.

    • Gauges represent a current snapshot of a value that can rise and fall freely. Queue depth, active connections, and cache size are all gauges.

    • Histograms record the distribution of observed values, such as request latency. They let you calculate percentiles (p99, p50) rather than just averages.

    Once you know which type fits your signal, choose a name that follows the convention <namespace>_<name>_<unit> in snake_case. A job processor might expose worker_jobs_processed_total (counter), worker_queue_depth (gauge), and worker_job_duration_seconds (histogram). Clear names save everyone debugging time later.

    Setting up the project

    The Go Prometheus client is the most common choice for exporters in the Kubernetes ecosystem, largely because the same library powers most of the official Kubernetes components. Start by creating a module and pulling in the dependency:

    mkdir my-exporter && cd my-exporter
go mod init example.com/my-exporter
go get github.com/prometheus/client_golang/prometheus
go get github.com/prometheus/client_golang/prometheus/promhttp

    Registering metrics

    Create main.go. The first thing to do is declare the metrics and register them with Prometheus's default registry. Registration tells the library that these metrics exist so they appear in the output even before the first observation is recorded:

    package main

import (
 "log"
 "net/http"

 "github.com/prometheus/client_golang/prometheus"
 "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
 jobsProcessed = prometheus.NewCounterVec(
 prometheus.CounterOpts{
 Name: "worker_jobs_processed_total",
 Help: "Total number of jobs processed, partitioned by status.",
 },
 []string{"status"},
 )

 queueDepth = prometheus.NewGauge(prometheus.GaugeOpts{
 Name: "worker_queue_depth",
 Help: "Current number of jobs waiting in the queue.",
 })

 jobDuration = prometheus.NewHistogram(prometheus.HistogramOpts{
 Name: "worker_job_duration_seconds",
 Help: "Time spent processing a single job.",
 Buckets: prometheus.DefBuckets,
 })
)

func init() {
 prometheus.MustRegister(jobsProcessed, queueDepth, jobDuration)
}

    prometheus.MustRegister panics on a duplicate registration, which makes misconfigurations obvious at startup rather than silently at runtime. If you are embedding this exporter inside a library that other packages will also instrument, prefer prometheus.Register and handle the error yourself.

    Collecting real values

    With the metrics registered, the next step is to keep them current. You can either continually update the data as the data change, or run your own internal refresh loop. The pattern below shows a polling loop — a goroutine that periodically reads from whatever data source your application owns and updates the registered metrics. Replace the simulated values with real calls to your database, internal API, or message broker:

    import (
 "math/rand"
 "time"
)

func collectMetrics() {
 for {
 // Replace these with real reads from your application.
 depth := float64(rand.Intn(50))
 queueDepth.Set(depth)

 start := time.Now()
 time.Sleep(time.Duration(rand.Intn(200)) * time.Millisecond)
 jobDuration.Observe(time.Since(start).Seconds())
 jobsProcessed.WithLabelValues("success").Inc()

 time.Sleep(5 * time.Second)
 }
}

    The polling interval (here five seconds) should be shorter than Prometheus's scrape interval so that each scrape sees a fresh value. The default scrape interval in most cluster deployments is fifteen seconds, which gives you comfortable headroom.

    Exposing the endpoint

    Wire the collection loop and the HTTP handler together in main. A /healthz path alongside /metrics gives Kubernetes a liveness probe target without exposing metric data on the health route:

    func main() {
 go collectMetrics()

 http.Handle("/metrics", promhttp.Handler())
 http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
 w.WriteHeader(http.StatusOK)
 })

 log.Println("Listening on :8080")
 if err := http.ListenAndServe(":8080", nil); err != nil {
 log.Fatalf("server error: %v", err)
 }
}

    Verify the output locally before building the image:

    go run .
curl http://localhost:8080/metrics | grep worker_

    You should see three # HELP and # TYPE blocks followed by the current metric values. If those lines appear, the exporter is working correctly and is ready to be containerized.

    Build a container image

    A multi-stage build keeps the final image small and avoids shipping a Go toolchain to production. The first stage compiles a statically linked binary; the second stage copies only that binary into a minimal base. The example below uses Docker, but the same pattern works with any OCI-compatible build tool such as Buildah or Podman:

    FROM golang:1.21-alpine AS builder
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /exporter .

FROM gcr.io/distroless/static:nonroot
COPY --from=builder /exporter /exporter
EXPOSE 8080
ENTRYPOINT ["/exporter"]

    distroless/static:nonroot contains no shell, no package manager, and runs as a non-root user by default, which satisfies most cluster security policies without extra configuration.

    Build and push the image, replacing <registry> with your own registry address:

    docker build -t <registry>/my-exporter:v1.0.0 .
docker push <registry>/my-exporter:v1.0.0

    (Note: Using a CI/CD pipeline to automate this is generally a better pattern than running these commands manually.)

    Deploying to the cluster

    Two manifests are enough to run the exporter: a Deployment that manages the pod lifecycle, and a Service that gives Prometheus a stable address to scrape. (You might prefer to have Prometheus scrape from every Pod; if that makes sense for your use case, then it's OK to configure instead).

    The examples below use the monitoring namespace, which is a common convention when running Prometheus and related components together. Adjust the namespace to match your own cluster setup.

    The Deployment sets conservative resource limits appropriate for a lightweight sidecar-style process, and uses the /healthz route for its liveness probe:

    apiVersion:apps/v1
kind:Deployment
metadata:
name:my-exporter
namespace:monitoring
labels:
app.kubernetes.io/name:my-exporter
spec:
replicas:1
selector:
matchLabels:
app.kubernetes.io/name:my-exporter
template:
metadata:
labels:
app.kubernetes.io/name:my-exporter
spec:
containers:
- name:exporter
image:<registry>/my-exporter:v1.0.0
ports:
- name:metrics
containerPort:8080
livenessProbe:
httpGet:
path:/healthz
port:8080
initialDelaySeconds:5
periodSeconds:10
resources:
requests:
cpu:50m
memory:32Mi
limits:
cpu:100m
memory:64Mi

    The Service names the port metrics, which the ServiceMonitor in the next section will reference by that name:

    apiVersion:v1
kind:Service
metadata:
name:my-exporter
namespace:monitoring
labels:
app.kubernetes.io/name:my-exporter
spec:
selector:
app.kubernetes.io/name:my-exporter
ports:
- name:metrics
port:8080
targetPort:metrics

    Apply both:

    kubectl apply -f deployment.yaml -f service.yaml

    Telling Prometheus where to look

    How you configure scraping depends on how Prometheus was installed.

    Option 1: Prometheus Operator (ServiceMonitor)

    If you installed Prometheus using the Prometheus Operator or the kube-prometheus-stack Helm chart, the operator must be running in your cluster before you create a ServiceMonitor. The release label must match the label selector configured on your Prometheus resource — kube-prometheus-stack is the default for a standard Helm install:

    apiVersion:monitoring.coreos.com/v1
kind:ServiceMonitor
metadata:
name:my-exporter
namespace:monitoring
labels:
release:kube-prometheus-stack
spec:
selector:
matchLabels:
app.kubernetes.io/name:my-exporter
endpoints:
- port:metrics
interval:15s
path:/metrics

    Option 2: Annotation-based discovery

    If your Prometheus uses annotation-based pod discovery instead, you will need a matching scrape_config rule in your Prometheus configuration — check with whoever manages your Prometheus installation to confirm it is in place.

    You can add the following two annotations to the Pod template regardless of which scraping method you use. They are ignored by the Prometheus Operator but picked up automatically by annotation-based setups:

    annotations:
prometheus.io/scrape:"true"
prometheus.io/port:"8080"# omit if not using annotation-based discovery
prometheus.io/path:"/metrics"# omit if not using annotation-based discovery

    If you are unsure which setup your cluster uses, the ServiceMonitor approach is more explicit and easier to debug.

    Verifying the scrape

    Port-forward to the Prometheus service and open the targets page to confirm the exporter has been discovered:

    kubectl port-forward svc/prometheus-operated 9090 -n monitoring

    Navigate to http://localhost:9090/targets. The my-exporter target should appear with state UP. If it shows DOWN, check that the ServiceMonitor's release label matches and that the pod is running:

    kubectl get pods -n monitoring -l app.kubernetes.io/name=my-exporter
kubectl describe servicemonitor my-exporter -n monitoring

    Once the target is healthy, run a quick query in the expression browser to confirm data is flowing:

    rate(worker_jobs_processed_total{status="success"}[2m])

    A non-zero result here means the full pipeline is working: your application is producing data, Prometheus is scraping it, and the time-series are stored and queryable.

    What comes next

    A working exporter is the foundation, not the destination. The natural next step is surfacing these metrics to the HorizontalPodAutoscaler so that your workload scales on the signals that actually drive load, not just CPU. That requires a metrics adapter — the Prometheus Adapter is the most widely deployed option — which registers your custom metrics with the Kubernetes Custom Metrics API. Once registered, any HorizontalPodAutoscaler in the cluster can reference worker_queue_depth or worker_jobs_processed_total directly in its metrics block.

    For a walkthrough of that setup, see Autoscaling on multiple metrics and custom metrics. For a catalog of ready-made exporters covering databases, message brokers, and cloud services, the Prometheus exporters and integrations page is a good starting point.

  • Operating AI/ML Workloads on Kubernetes: A Headlamp Plugin for Kubeflow

    Kubernetes has quietly become the default platform for AI and machine learning. Whether you run notebook servers for data scientists, schedule distributed training jobs, tune hyperparameters, or orchestrate multi-step ML pipelines, those workloads increasingly land on a Kubernetes cluster. Kubeflow is one of the most popular ways to assemble that stack, and it does so the Kubernetes-native way: every capability is exposed as a Custom Resource Definition (CRD).

    That design is a gift to cluster operators, because it means ML workloads can be observed and managed with the same primitives as everything else in the cluster. But in practice the specialized ML dashboards that ship with these platforms hide the Kubernetes layer underneath. When a notebook is stuck or a training run fails, the operator is often left dropping back to kubectl to find out what actually happened at the Pod level.

    This post introduces the Headlamp Kubeflow plugin, which closes that gap by surfacing Kubeflow's custom resources directly inside a general-purpose Kubernetes UI. It is a worked example of a pattern any CRD-heavy platform can follow: meet operators where they already work, and show them the cluster-level truth.

    Headlamp itself is an extensible Kubernetes web UI maintained under Kubernetes SIG UI and licensed under Apache 2.0. It runs as a desktop app or in-cluster, and its plugin system lets anyone add first-class views for custom resources.

    Why operators need a different view

    Purpose-built ML dashboards help data scientists submit experiments, pipelines, and notebooks. Cluster operators and site reliability engineers (SREs) troubleshoot the Kubernetes resources underneath, and they ask different questions:

    • Why is a notebook stuck? Is it ImagePullBackOff, OOMKilled, or a Pod waiting on a PersistentVolumeClaim?
    • Which Run resources failed recently across namespaces?
    • Which parameter set does a Katib Experiment report as optimal?
    • Do TrainJob resources reference the expected TrainingRuntime resources?
    • Which batch workloads are running, and what state does Kubernetes report?

    The Headlamp Kubeflow plugin helps answer these questions by reading directly from the Kubernetes API server. It shows Pod conditions, Kubernetes failure reasons, and resources across namespaces without requiring an intermediary ML service or database.

    What the plugin covers

    Kubeflow is modular, and teams often install only the components they need. The plugin discovers the Kubeflow API groups on a cluster and displays only the corresponding sections.

    The plugin supports the following component families and API resources:

    Kubeflow components and API resources supported by the Headlamp plugin
    Component Purpose API resources
    Notebooks Provides development environments such as Jupyter, VS Code, and RStudio Notebook, Profile, PodDefault
    Pipelines Defines and tracks pipelines, versions, experiments, runs, and schedules Pipeline, PipelineVersion, Run, RecurringRun, Experiment
    Katib Automates hyperparameter tuning and neural architecture search Experiment, Trial, Suggestion
    Training Runs distributed training workloads such as PyTorch and TensorFlow jobs TrainJob, TrainingRuntime, ClusterTrainingRuntime
    Spark Runs large-scale data processing with Apache Spark SparkApplication, ScheduledSparkApplication

    What you can see

    Inspect notebook Pods

    The Notebook detail view shows Pod conditions and their reason and message fields. It also shows CPU, memory, and GPU requests and limits; volume mounts and their backing types, such as PersistentVolumeClaim, ConfigMap, Secret, or emptyDir; environment variables that reference Secret or ConfigMap objects; sidecar containers; and node tolerations. This view consolidates information that would otherwise require several kubectl describe commands.

    Inspect hyperparameter tuning

    The Katib views show the tuning algorithm, search space, every Trial with its live status, and the current best Trial with its metric values and parameter assignments. They also show the early-stopping configuration and the number of Trial resources that stopped early, so you can follow the search without leaving the cluster UI.

    Inspect pipeline state without the backend database

    The Pipelines views read Kubernetes API resources directly and do not query the Kubeflow Pipelines API service or backend database. You can inspect stored pipeline state even when that service is unavailable. The Pipeline detail view compares the latest and previous PipelineVersion specifications in a side-by-side YAML diff. Run views show state and duration, RecurringRun views show human-readable schedules, and the artifacts view aggregates pipelineRoot values from recent Run resources.

    Map ML resources

    The plugin registers a Headlamp map source that renders Notebook, Profile, PodDefault, Experiment, Pipeline, SparkApplication, and TrainJob resources as graph nodes. It draws edges between supported resources based on .metadata.ownerReferences. Headlamp also shows inline summaries for these resource types when you hover over them.

    Try it

    The Kubeflow plugin README explains installation and local-cluster setup, including a lightweight CRD-only path for evaluation. Because the plugin discovers installed API groups, you can use it with an existing modular Kubeflow installation or create an evaluation cluster with only the CRDs and sample resources.

    Apply the pattern to other platforms

    Kubeflow illustrates a broader pattern. Platforms often model domain-specific workflows with custom resources. Their dashboards focus on those workflows, while Kubernetes operators also need the state of the underlying API resources and Pods. A CRD-driven plugin in a general Kubernetes UI can expose that state without making operators switch between unrelated tools.

    The plugin uses the Apache 2.0 license and is developed under Kubernetes SIG UI. To report a problem or contribute an improvement, use the Headlamp plugins repository's issue tracker or pull requests.

  • Kubernetes Dashboard to Headlamp: A Step-by-Step Guide

    1. Before you start: know what is changing

    Kubernetes Dashboard and Headlamp both show what is running in a cluster, but they work differently. When Headlamp runs on the desktop, it uses your existing kubeconfig to connect to one or more clusters and can be extended with plugins. When Headlamp runs inside a cluster, it uses a Kubernetes ServiceAccount to access the API and follow RBAC rules. Kubernetes Dashboard, in contrast, only runs in-cluster and always relies on service account tokens. Understanding these models early helps you choose the right setup and permissions.

    1.1 How Kubernetes Dashboard works

    Dashboard is a web app that runs inside your cluster.

    • You install it in the cluster, often with Helm.
    • You usually run one Dashboard per cluster.
    • You often reach it with kubectl port-forward or an ingress.
    • You log in with a Bearer token. That token is often from a service account.
    • It includes forms that help you create resources.
    • It leans on tables and lists for navigation.

    It feels like this: a UI that lives with the cluster.

    1.2 How Headlamp works

    Headlamp acts more like a Kubernetes client with a UI.

    • It can run on your desktop or in a cluster.
    • It reads your kubeconfig, like kubectl does.
    • It can show more than one cluster in one place.
    • It favors YAML when you create or change resources.
    • It includes list views and a visual map.
    • You can add features with plugins.

    Headlamp is a UI that follows your identity, not your cluster.

    1.3 What stays the same

    Many workflows will feel familiar:

    • Browse workloads and resources
    • Filter by namespace
    • Inspect YAML, events, and status
    • View logs
    • Take actions your RBAC allows

    1.4 What changes

    A few things will feel different:

    • Login shifts from pasted tokens to kubeconfig (and sometimes SSO).
    • Creation shifts from forms to "apply YAML."
    • Multi-cluster becomes normal, not a special case.
    • The map view helps you see how resources connect.

    2. Pre-migration checklist

    This checklist helps you avoid surprises during the switch. It makes sure Headlamp can use the same identity and permissions you already trust in Kubernetes. It also gives you a quick way to prove the migration worked before you turn off Dashboard.

    2.1 Write down what you use today

    List the basics:

    • Which clusters you use (dev, staging, prod)
    • Which namespaces you touch most
    • What you do most often (view, edit, scale, delete, debug)
    • How you access Dashboard today (port-forward or ingress)
    • How you log in (service account token, and which RBAC bindings)

    This is your baseline.

    2.2 Check that kubeconfig works

    Headlamp uses kubeconfig, especially on desktop. Make sure yours works before you install anything.

    Run:

    kubectl config current-context

    Then try:

    kubectl get nodes

    If you cannot list nodes, test in a namespace you can access:

    kubectl get pods -n <namespace>

    If these work, Headlamp can use the same identity and RBAC.

    2.3 Pick a rollout plan

    There is no need to rush. Most teams choose one of these:

    Parallel rollout (recommended)

    • Install Headlamp
    • Let people try it
    • Keep Dashboard for a short time
    • Remove Dashboard after the team is ready

    Cutover

    • Install Headlamp
    • Switch docs and links
    • Remove Dashboard soon after

    Parallel rollout is safer for shared clusters.

    2.4 Decide where Headlamp will run

    You can use either option. Many teams use both.

    Desktop

    • Uses your kubeconfig
    • Uses no cluster resources
    • No port-forward needed
    • Multi-cluster works out of the box

    In-cluster

    • Works well for shared, browser access
    • Can be managed like other cluster apps
    • Often paired with ingress and SSO

    2.5 Note optional dependencies

    These are common. You can handle them later.

    • metrics-server (for CPU and memory graphs)
    • ingress (for an in-cluster URL)
    • OIDC / SSO (for browser sign-in)
    • cleanup of old Dashboard service accounts and RBAC

    3. Choose where Headlamp will run (desktop or in-cluster)

    Headlamp can run on your desktop or inside a cluster. Both work well, but they fit different needs. Desktop is the fastest way to start because it uses your kubeconfig and does not run in the cluster. In-cluster is best when you need a shared URL and want the platform team to manage upgrades and access.

    Option A: Desktop (user-managed)

    Desktop Headlamp runs on each user's machine. It reads the same kubeconfig you use with kubectl. This keeps access tied to each user's identity and RBAC.

    Why teams pick it

    • No in-cluster service to deploy or expose.
    • It uses no cluster CPU or memory.
    • It uses your kubeconfig and RBAC.
    • It works with many clusters in one app.
    • You do not need port-forward for day-to-day use.

    Option B: In-cluster (best for shared access)

    In-cluster Headlamp is installed as a Kubernetes workload (often via Helm). This lets cluster admins manage it like other in-cluster apps.

    • Cluster admins manage install, upgrades, and configuration through the Helm chart and standard Kubernetes tooling.
    • Admins control ingress and can set up OIDC login for shared access.
    • It supports shared use in team environments.

    4. Install Headlamp (desktop and in-cluster)

    This section gets Headlamp running. Follow the path you chose in Section 3.

    4.1 Desktop install (fastest way to start)

    Install Headlamp on your machine. Then open it like any other app. Headlamp reads your kubeconfig and uses the same identity and RBAC rules as kubectl.

    Windows

    Install with WinGet:

    winget install headlamp

    Or with Chocolatey:

    choco install headlamp

    macOS

    Install with Homebrew:

    brew install --cask headlamp

    Linux

    Install with Flatpak (Flathub):

    flatpak install flathub io.kinvolk.Headlamp

    Quick check

    1. Launch Headlamp.
    2. Confirm you can see a cluster context.
    3. Open a namespace you can access and confirm you can list workloads. Headlamp will only show actions your RBAC allows.

    4.2 In-cluster install (shared access)

    Use this path when you want a shared UI that the platform team can manage. Headlamp supports in-cluster deployment with Helm or a YAML manifest.

    Install with Helm

    Add the repo and update:

    helm repo add headlamp https://kubernetes-sigs.github.io/headlamp/
helm repo update

    Create a namespace (example):

    kubectl create namespace headlamp

    Install the chart:

    helm install headlamp headlamp/headlamp --namespace headlamp

    Install with a YAML manifest (optional)

    Headlamp also provides a YAML manifest you can apply and then adjust to your needs.

    Check the install

    Confirm the pod is running:

    kubectl get pods -n headlamp

    Confirm the service exists:

    kubectl get svc -n headlamp

    Access it (two common ways)

    Quick test with port-forward

    This is the fastest way to verify the service works:

    kubectl port-forward -n headlamp svc/headlamp 8080:80

    Then open: http://localhost:8080

    Shared access with ingress

    If you want a stable URL, expose the service through your ingress controller. Your exact ingress YAML depends on your setup. Headlamp's OIDC callback URL is your public URL plus /oidc-callback, so ingress and TLS settings matter.

    4.3 Updating Headlamp

    Updates depend on how you installed Headlamp. Package managers upgrade in place. DMG or EXE installs update by reinstalling the newer download.

    macOS

    If you installed with Homebrew, run:

    brew upgrade headlamp

    If you installed from a DMG, download the newest DMG and drag Headlamp into /Applications, replacing the old version. DMG installs do not auto upgrade.

    Windows

    If you installed with WinGet, run:

    winget upgrade headlamp

    If you installed with Chocolatey, run:

    choco upgrade headlamp

    If you installed from the EXE, download the newest installer and run it again. EXE installs do not auto upgrade.

    Linux

    If you installed with Flatpak, run:

    flatpak update io.kinvolk.Headlamp

    If you installed with AppImage, download the newest AppImage and run that file instead.

    If you installed with a tarball, download the newest tarball, extract it, and run the new headlamp binary.

    4.4 Notes for in-cluster access (keep it safe)

    Treat an in-cluster UI like any other cluster-facing service. Use TLS, lock down who can reach it, and rely on Kubernetes auth and RBAC to control what users can do.

    5. Authentication and RBAC

    Headlamp uses the Kubernetes API the same way kubectl does. Your cluster still decides who can do what. Headlamp only shows actions your identity is allowed to take.

    This section covers two setups: desktop and in-cluster.

    5.1 Desktop: use kubeconfig

    On desktop, Headlamp reads your kubeconfig and uses the same credentials you use with kubectl. There is no separate token login flow to manage.

    Step 1: Confirm your kubeconfig works

    Run:

    kubectl config current-context

    Then test access:

    kubectl get nodes

    If you cannot list nodes, test a namespace you can access:

    kubectl get pods -n <namespace>

    If these commands work, your kubeconfig and credentials are valid for Headlamp too.

    Step 2: Point Headlamp at the right kubeconfig (if needed)

    Headlamp can use the default kubeconfig path. It can also use a custom file path. You can set KUBECONFIG to choose a specific file.

    Example:

    KUBECONFIG=/path/to/config headlamp

    You can also use more than one kubeconfig file at once. On Unix systems, separate paths with :. On Windows, separate paths with ;.

    What to expect in the UI

    Headlamp adapts to your RBAC permissions. If you do not have permission to edit or delete a resource, Headlamp will not offer those actions.

    5.2 In-cluster: shared access needs a sign-in plan

    In-cluster Headlamp is shared by many users. You need a clear plan for sign-in and access. Headlamp supports OpenID Connect (OIDC) for a "Sign in" flow.

    You will usually choose one of these patterns:

    • A. Configure Headlamp with OIDC (built-in).
    • B. Put an auth layer in front of Headlamp (common in enterprises).

    A. Built-in OIDC (Headlamp)

    To use OIDC, Headlamp needs:

    • Client ID
    • Client secret
    • Issuer URL
    • (Optional) scopes

    Your OIDC provider must also allow Headlamp's callback URL. The callback is your Headlamp URL plus:

    • /oidc-callback

    Example:

    • https://headlamp.example.com/oidc-callback

    Ingress note

    If Headlamp is behind an ingress or load balancer, make sure it forwards X-Forwarded-Proto. If it does not, Headlamp may generate an http callback URL instead of https. That can break login.

    B. Auth layer in front of Headlamp

    Some teams protect Headlamp with an identity-aware proxy or a platform auth system. This keeps sign-in consistent across tools. Headlamp docs include an example using OpenUnison, which can deploy Headlamp with hardened defaults and integrate with identity providers.

    5.3 RBAC: keep it least privilege

    Kubernetes security starts with API authentication and authorization (RBAC). Headlamp respects those rules.

    Practical guidance:

    • Start with the lowest permissions that still let users do their job.
    • If Dashboard used a high-privilege service account token, plan to remove or tighten that access after the move.
    • For in-cluster, treat the UI like any other endpoint. Use TLS and limit network access.

    5.4 Quick troubleshooting

    Desktop: "I do not see my cluster"

    Your kubeconfig may not be in the default location. Point Headlamp to the file with KUBECONFIG or a file path.

    In-cluster: "OIDC login fails after redirect"

    Confirm your provider allows https://YOUR_URL/oidc-callback. If you use ingress, make sure it forwards X-Forwarded-Proto.

    6. Manage multiple clusters

    Kubernetes Dashboard is usually tied to one cluster at a time. Headlamp is built for multi-cluster work. It is a client that follows your kubeconfig, not a single cluster install. That means you can keep one UI open and switch clusters as you work.

    Clusters come from your kubeconfig

    Headlamp reads clusters from your kubeconfig files. That means the clusters you can access with kubectl can also show up in Headlamp.

    Switch clusters in the UI

    Once Headlamp loads your kubeconfig, you can switch clusters using the cluster selector. This makes it easier to move between dev, staging, and prod without changing tools.

    Optional: use more than one kubeconfig file

    If you keep separate kubeconfig files, you can load them together. Headlamp supports multiple kubeconfig paths in KUBECONFIG.

    Unix/macOS/Linux (: separator):

    KUBECONFIG=~/.kube/dev:~/.kube/prod headlamp

    Windows (; separator):

    $env:KUBECONFIG="$HOME\.kube\dev;$HOME\.kube\prod"

    Optional: add a cluster from inside Headlamp

    You can also add clusters by loading additional kubeconfig files from the UI.

    Permissions stay the same

    Multi-cluster does not change security rules. Each cluster still enforces its own RBAC. Headlamp shows only what your identity can do in the selected cluster.

    7. Navigate and understand resources

    If you used Kubernetes Dashboard, this part will feel familiar. Headlamp keeps the same core resource views, but makes it easier to move around and understand what is connected.

    Find resources in familiar places

    Headlamp groups resources in a way that maps closely to Dashboard:

    • Workloads for Pods, Deployments, StatefulSets, and Jobs
    • Network for Services and Ingress
    • Storage for PersistentVolumes and Claims
    • Configuration for ConfigMaps and Secrets
    • Nodes for cluster infrastructure

    You can filter by namespace at the top of the UI, just like in Dashboard.

    Inspect and edit resources

    From any list, you can click into a resource to see details:

    • Status and conditions
    • Events
    • Labels and annotations
    • The full YAML definition

    If your RBAC allows it, you can edit YAML directly from the UI. If it does not, Headlamp shows the resource as read-only. This matches how kubectl behaves.

    Use search and filters to move faster

    Headlamp adds faster search and filtering across lists. This helps when clusters or namespaces get large. You can narrow views without jumping between pages.

    Understand relationships with Map View

    Dashboard mostly shows resources as lists. Headlamp also includes a Map View.

    Map View shows how resources relate to each other:

    • Deployments
    • ReplicaSets
    • Pods
    • Services

    This helps when you are troubleshooting. Instead of clicking through several pages, you can see the connections at once. You can spot missing links or broken relationships faster.

    When to use lists vs Map View

    • Use lists when you know what resource you are looking for.
    • Use Map View when you are trying to understand why something is not working.

    Both views work on the same data. You are just choosing how much context you want at that moment.

    8. Deploy applications with YAML

    This is the biggest change for most Kubernetes Dashboard users. Dashboard relied on forms. Headlamp relies on manifests. The goal is not to slow you down. It is to align the UI with how Kubernetes is usually run in practice.

    From forms to manifests

    In Kubernetes Dashboard, you often deployed an app by filling in a form:

    • container image
    • replicas
    • service type

    Headlamp does not include the same wizard. Instead, it lets you apply YAML directly from the UI.

    This matches how most teams deploy today:

    • manifests live in Git
    • CI/CD applies them
    • Helm or GitOps tools manage changes

    Headlamp fits into that flow rather than replacing it.

    Create resources using YAML

    To deploy an application in Headlamp:

    1. Select a cluster and namespace.
    2. Click Create.
    3. Paste or upload a YAML manifest.
    4. Review it.
    5. Click Apply.
    Create button highlight

    The resource appears immediately in the UI.

    If the manifest is not valid, Headlamp shows the same errors you would see from the Kubernetes API.

    Generate YAML the easy way

    If you miss the Dashboard wizard, you can still generate YAML quickly.

    For example:

    kubectl create deployment nginx \
 --image=nginx \
 --dry-run=client \
 -o yaml > nginx.yaml

    You can edit the file if needed, then paste it into Headlamp and apply it.

    This gives you a repeatable manifest instead of an object created only through a UI.

    What if you use Helm or GitOps?

    That works well with Headlamp.

    • Install with Helm as usual.
    • Deploy with GitOps pipelines as usual.
    • Use Headlamp to view, inspect, and debug what is running.

    Headlamp does not replace those tools. It gives you visibility into what they create.

    What to expect compared to Dashboard

    • You will not see a multi-step deploy form.
    • You will work more with YAML.
    • You gain clarity about what is actually applied to the cluster.
    • The same manifest can be reused in CI, Git, or other tools.

    9. Deploy and debug workloads

    One of the main reasons people used Kubernetes Dashboard was day-to-day debugging. Headlamp covers the same tasks and adds a few useful upgrades.

    View logs

    You can view pod logs directly in the UI.

    To check logs:

    1. Open Workloads.
    2. Select Pods.
    3. Click a pod.
    4. Open the Logs tab.
    Workloads view

    If the pod has more than one container, you can switch between containers. Logs stream live, which helps during rollouts or active incidents.

    Exec into running pods

    Headlamp also lets you open a shell inside a container.

    From a pod view:

    • Open the pod actions menu.
    • Choose Terminal or Exec.

    This opens an interactive session inside the container. It replaces the need to switch back to the terminal for quick checks.

    This action follows RBAC rules. If you cannot run kubectl exec, Headlamp will not allow it either.

    Check metrics and resource usage

    Headlamp can show CPU and memory usage for pods and nodes. This works the same way it did in Dashboard.

    A few things to know:

    • Metrics require metrics-server to be installed in the cluster.
    • If metrics are missing, Headlamp shows a clear notice.
    • Once metrics are available, usage appears on pod and node views.

    This makes it easy to answer simple questions:

    • Is this pod using too much memory?
    • Is a node under pressure?

    View events when something goes wrong

    Events are often the fastest way to understand failures.

    In Headlamp, you can:

    • View events on resource detail pages.
    • See warnings and errors tied to pods, nodes, or deployments.

    This is often the first place to look when a workload is stuck or crashes.

    How this compares to Dashboard

    What stays the same:

    • Log viewing
    • Event inspection
    • RBAC-aware actions

    What improves:

    • Built-in exec sessions
    • Clearer layout and filtering
    • Fewer context switches between UI and CLI

    10. Remove Kubernetes Dashboard

    After Headlamp is working and your team is comfortable using it, you can remove Kubernetes Dashboard. This is the final cleanup step.

    Removing Dashboard reduces clutter and avoids keeping unused access paths around.

    Confirm Headlamp covers your needs

    Before uninstalling anything, make sure:

    • Users can access the clusters they need in Headlamp.
    • Common tasks work:
      • browse resources
      • deploy with YAML
      • view logs and events
      • exec into pods (if allowed)
    • RBAC behaves as expected for different roles.

    Once these checks pass, you are ready to remove Dashboard.

    Uninstall the Dashboard

    If you installed Kubernetes Dashboard with Helm, remove it with:

    helm uninstall kubernetes-dashboard -n kubernetes-dashboard

    If Dashboard was installed by a manifest or addon, remove it using the same method you used to install it.

    After removal, confirm the resources are gone:

    kubectl get pods -n kubernetes-dashboard

    Many Dashboard setups used dedicated service accounts and cluster-wide roles.

    Review and remove anything that was created only for Dashboard access, such as:

    • service accounts
    • role bindings or cluster role bindings
    • old documentation that points users to Dashboard URLs or port-forward commands

    This reduces long-lived credentials and unused permissions.

    Communicate the change

    Make sure your team knows:

    • Headlamp is now the primary Kubernetes UI.
    • How to access it (desktop or URL).
    • Where to go for help if something feels different.

    11. Post-migration checklist

    This final checklist helps you confirm the migration is complete. It gives you confidence that Headlamp is working as expected and that nothing important was left behind.

    Access and visibility

    • Headlamp opens without errors.
    • Users can access the correct clusters.
    • Namespace filtering works as expected.
    • Multi-cluster switching behaves correctly.

    Authentication and RBAC

    • Desktop users access clusters using kubeconfig.
    • In-cluster users can sign in using the chosen auth method.
    • Users only see actions their RBAC allows.
    • No unexpected permission errors appear during normal use.

    Core workflows

    • Resources load under Workloads, Network, and Configuration.
    • YAML can be viewed and edited where permissions allow.
    • Applications can be deployed using Create and YAML.
    • Logs load correctly for running pods.
    • Exec works for users who are allowed to use it.
    • Metrics appear if metrics-server is installed.

    Operational confidence

    • Teams can troubleshoot without switching tools.
    • Map View helps explain relationships during debugging.
    • Platform or DevOps teams know how Headlamp is installed and managed.

    Cleanup confirmation

    • Kubernetes Dashboard is no longer running.
    • Dashboard-only service accounts and RBAC bindings are removed.
    • Internal docs no longer reference Dashboard URLs or port-forward commands.

    Team alignment

    • The team knows Headlamp is the default Kubernetes UI.
    • Onboarding docs point new users to Headlamp.
    • There is a clear path for feedback or questions.

    You've now completed the move from Kubernetes Dashboard to Headlamp. Your team can use the same Kubernetes access model, work across clusters, and rely on workflows that match how Kubernetes is used today. From here, Headlamp becomes your default UI, whether on the desktop or in shared environments. As your needs grow, you can keep using it as-is or extend it with plugins and new views over time.

    If you want to help shape what comes next, join the Headlamp community and contribute at headlamp.dev.

  • Announcing etcd v3.7.0

    This article is a mirror of the original announcement

    Today, SIG etcd is releasing etcd v3.7.0, the latest minor release of the popular distributed key-value store and core Kubernetes component. v3.7 ships the long-requested RangeStream feature, delivers several other performance improvements, removes the last remnants of the legacy v2store, and completes a major protobuf overhaul.

    You can download etcd v3.7.0 here:

    This release also includes new versions of the two core etcd dependencies, bbolt v1.5.0 and raft v3.7.0.

    For instructions on installing etcd, see the install documentation. For the full list of changes, see the etcd v3.7 changelog.

    A heartfelt thank you to all the contributors who made this release possible!

    Major features

    The most significant changes in v3.7.0 include:

    Features

    RangeStream

    In etcd v3.6 and earlier, it is challenging to work with requests that return large result sets. The database would buffer the full result set before sending, leading to unpredictable latency and memory usage, both on the server and the client. The RangeStream RPC lets calling applications accept result sets in chunks, reducing latency and making buffering memory usage more predictable.

    Instructions on how to use RangeStream in gRPC calls and in etcdctl can be found in the etcd documentation. Users should try it out for their own applications.

    In coordinated releases, the RangeStream feature will become available to users running the upcoming v1.37 of Kubernetes by enabling the EtcdRangeStream feature gate. This early and planned adoption is possible thanks to the merger of etcd and Kubernetes development in 2023.

    Performance improvements

    v3.7 delivers multiple specific performance improvements, both for the Kubernetes control plane and for other use cases. Kubernetes users should see a significant decrease in overall CPU usage by the etcd members, compared with v3.6.

    Keys-only range optimization

    etcd v3.7.0 includes a keys-only Range optimization (#21791: keys-only Range optimization). When processing a keys_only Range request or etcdctl get --keys-only, etcd reads solely from its in-memory index. It returns the matched keys without loading all serialized values from bbolt as it did previously. The only exception where loading from bbolt is still required is when keys_only Range requests must be sorted by value (i.e., when SortTarget is set to VALUE).

    This reduces unnecessary backend reads and memory use for workloads that only need key names, making large keys-only range requests more efficient.

    Faster, more reliable etcd leases

    v3.7 improves lease expiration and renewal:

    Faster find() operations

    etcd 3.7 improves the performance of concurrent watches on keys by making find() operations faster (#19768: adt: split interval tree by right endpoint on matched left endpoints).

    Other features

    Protobuf overhaul

    v3.7 migrates and replaces multiple outdated protobuf libraries with fully supported dependencies. This includes replacing github.com/golang/protobuf and github.com/gogo/protobuf with the fully-supported google.golang.org/protobuf (#14533: Protobuf: cleanup both golang/protobuf and gogo/protobuf), and migrating grpc-logging to grpc-middleware v2 (#20420: Migrate grpc-logging to grpc-middleware v2).

    As well as improving security and maintainability, this refactor has been shown to reduce CPU usage by etcd components.

    While these changes are not expected to directly affect users running etcd via official binaries or container images, they may affect users who depend on etcd Go modules, such as the client SDK or packages under api/ or pkg/. These consumers may need to update their code or dependencies due to protobuf and related API changes introduced in this release. More detailed information is available from the API change tracking issue.

    Unix socket support

    etcd now supports Unix socket endpoints (#19760: Add Support for Unix Socket endpoints), enabling local communication without a TCP port. Since this is restricted to single-member clusters, it is mainly aimed at development, testing, and edge device use-cases.

    Bootstrap from v3store

    One of the major changes in etcd v3.7 is that the server now bootstraps entirely from the v3 store (#20187 Bootstrap etcdserver from v3store), eliminating its dependency on the legacy v2 store during startup.

    This milestone is the result of a long-term effort spanning multiple releases, from v3.4 through v3.7. It resolves a long-standing technical debt, significantly simplifies the bootstrap workflow, and lays the foundation for future improvements to etcd.

    To maintain backward compatibility, etcd v3.7 continues to generate v2 snapshots. As a result, the --snapshot-count flag is also retained in v3.7. This is the last remaining dependency on the legacy v2 store, and both the v2 snapshot generation and the --snapshot-count flag will be removed in v3.8.

    etcdutl timeouts

    All etcdutl commands now have a timeout command line argument (#20708: etcdutl: enable timeout functionality for all commands), so offline utility commands no longer block indefinitely when holding a lock.

    Setting the authentication token directly

    Client v3 now allows users to set the JWT directly, offering more flexibility in authentication options (#16803: clientv3: allow setting JWT directly, #20747: clientv3: disable auth retry when token is set),

    Retrieve AuthStatus without authenticating

    Clients can check their AuthStatus without attempting to authenticate first, eliminating some application overhead (#20802: etcdserver: remove permission check on AuthStatus api).

    New watch metrics

    v3.7 adds optional watch send-loop metrics (#21030: Instrument watchstream send loop) for better observability of the watch path:

    • etcd_debugging_server_watch_send_loop_watch_stream_duration_seconds
    • etcd_debugging_server_watch_send_loop_watch_stream_duration_per_event_seconds
    • etcd_debugging_server_watch_send_loop_control_stream_duration_seconds
    • etcd_debugging_server_watch_send_loop_progress_duration_seconds

    There is also a new etcd_server_request_duration_seconds metric (#21038: Add metric etcd_server_request_duration_seconds).

    etcdctl command cleanup

    etcdctl commands were reorganized for clarity (#20162: etcdctl: organize etcdctl subcommand) and global command line arguments are now hidden to streamline help output (#20493: etcdctl: hide global flags).

    Upgrading

    This release contains breaking changes, particularly around the removal of legacy v2 components. Users should review the upgrade guide before upgrading their nodes. As with all minor releases, perform a rolling upgrade one member at a time and confirm cluster health between steps.

    Experimental flags removed

    All deprecated experimental flags have been removed (#19959: Cleanup the deprecated experimental flags). Features in etcd now follow the Kubernetes-style feature-gate lifecycle (Alpha → Beta → GA) introduced in v3.6, rather than the old --experimental prefix. If your configuration still relies on --experimental-* command line arguments, migrate to using the corresponding feature gates or stable command line arguments before you upgrade to etcd 3.7.

    Legacy V2 API packages and code cleanup

    To remove the dependencies on v2store, the following components have been removed:

    These changes may create some breakage for users, particularly those who have not already updated to v3.6.11 or later. Users should report any blockers encountered, or cases that need better upgrade documentation.

    Non-blocking client creation

    etcd no longer honors the deprecated grpc.WithBlock dial option ( #21942: Make the etcd client creation non-blocking). To preserve the previous blocking behavior when needed, follow the guidance in grpc-go's anti-patterns documentation.

    Multiarch container images only

    For users relying on the official etcd container images, v3.7 will be distributed only as multiarch containers. Architecture-tagged images will not be available, so adjust deployments accordingly.

    API changes

    As with every etcd release, there are a number of API changes. These are designed to be backwards-compatible to the extent possible, but may require adjustment by some users. See our API documentation page for full information.

    bbolt v1.5.1

    etcd v3.7 depends on, and includes, v1.5.1 of the bbolt storage engine. v1.5 includes several improvements to functionality and performance, including:

    raft v3.7.0

    etcd 3.7 depends on, and includes, v3.7.0 of the raft consensus engine. v3.7 includes several improvements, including:

    raft v3.7.0 also includes the same protobuf library updates and refactoring as etcd does.

    Dependency updates

    Other dependency updates include a bump to golang.org/x/crypto v0.52.0 for CVE resolution (#21903: [release-3.7] Bump golang.org/x/crypto to v0.52.0), an OpenTelemetry contrib update to v0.61.0 (#20017: Update otelgrpc to v0.61.0), and compilation with Go 1.26.4 (#21891: [release-3.7] Update Go to 1.26.4).

    Contributors

    etcd v3.7.0 is the product of more than a hundred contributors across the community. Thank you to everyone who wrote code, reviewed PRs, filed and triaged issues, and helped test the alpha, beta, and release candidates.

    Leads

    The SIG etcd leads for the v3.7 release are ivanvc, serathius, ahrtr, fuweid, siyuanfoundation, and jberkus. Ivan leads our release team.

    Other contributors

    ah8ad3, ajaysundark, aladesawe, amosehiguese, ArkaSaha30, ashikjm, AwesomePatrol, dims, Elbehery, gangli113, henrybear327, Jille, jmhbnz, joshuazh-x, kishen-v, lavishpal, liggitt, marcelfranca, miancheng7, mmorel-35, MrDXY, mrueg, purpleidea, qsyqian, redwrasse, ronaldngounou, skitt, spzala, tcchawla, tjungblu, vivekpatani, wenjiaswe

    New contributors

    A special welcome to the contributors who made their first etcd contribution in this cycle — including Jeffrey Ying, whose work drove the RangeStream feature. New contributors can have a substantial impact on etcd; if you’d like to get involved, see the contributor guide.

    1911860538, 4rivappa, aaronjzhang, abdurrehman107, ABin-Huang, adeptvin1, aditya7880900936, AHBICJ, akstron, alliasgher, aman4433, aojea, apullo777, AR21SM, arturmelanchyk, AshrafAhmed9, asttool, asutorufa, BBQing, beforetech, boqishan, caltechustc, carsontham, christophsj, chuanye-gao, cnuss, cuiweixie, dmvolod, Dogacel, dongjiang1989, EduardoVega, evertrain, eyupcanakman, gaganhr94, goingforstudying-ctrl, greenblade29, Himanshu-370, HossamSaberX, huajianxiaowanzi, hwdef, ishan-gupta2005, ishan16696, ivangsm, JasonLove-Coding, Jefftree, jihogh, jonathan-albrecht-ibm, joshjms, kairosci, kei01234kei, kjgorman, kovan, kstrifonoff, Kunalbehbud, letreturn, lorenz, m4l1c1ou5, madhav-murali, madvimer, majiayu000, marcus-hodgson-antithesis, mattsains, mcrute, mingl1, MohanadKh03, mstrYoda, NAM-MAN, neeraj542, nicknikolakakis, nihalmaddala, niuyueyang1996, notandruu, ntdkhiem, nwnt, olamilekan000, pigeio, pjsharath28, progmem, Qian-Cheng-nju, quocvibui, ravisastryk, robin-vidal, robinkb, rockswe, roman-khimov, rsafonseca, sahilpatel09, SalehBorhani, SebTardif, seshachalam-yv, shashwat010, shivamgcodes, shuan1026, silentred, sneaky-potato, socketpair, srri, subrajeet-maharana, sxllwx, tchap, tsujiri, tzfun, upamanyus, uzairhameed, varunu28, vihasmakwana, wendy-ha18, xiaoxiangirl, xigang, xUser5000, yagikota, yajianggroup, yedou37, Zanda256, zechariahkasina, zhijun42, zhoujiaweii

    Feedback can be shared through:

  • Open source maintainership in the age of AI

    AI has really changed the game around software development. More people are leveraging AI than ever to contribute patches to projects they use. To me, this is a good thing as more folks will contribute patches rather than fork or not fix them. The main problem is that AI has made generating code fast but there has been very little improvement in maintaining code bases. In this post, we will highlight the ways the Kubernetes community is adapting to the world of AI assisted coding.

    The first step of this journey was to develop an AI policy. This seems mundane and bureaucratic but there were many PRs that derailed into discussions around AI usage. The AI policy helps steer the conversation around the project's stance on AI and provides a clear signal to contributors on how to use these tools responsibly.

    Kubernetes AI policy

    The Kubernetes project has established clear guidelines for AI-assisted contributions that balance innovation with accountability. These policies are designed to maintain code quality and ensure human oversight while acknowledging that AI tools can be valuable aids in the development process.

    Transparency first

    Contributors must disclose when AI tools have been used to assist with a pull request. A simple statement in the PR description such as "This PR was written in part with the assistance of generative AI" is sufficient. This transparency helps reviewers understand the context and apply appropriate scrutiny.

    Human accountability

    While AI tools can assist, the human contributor remains fully responsible for every change. The policy explicitly prohibits:

    • Listing AI as a co-author on commits
    • Using AI co-signing on commits
    • Adding trailers like "assisted-by" or "co-developed" that attribute work to AI

    This isn't about diminishing AI's role as a tool—it's about maintaining clear accountability. If something breaks, there needs to be a human who understands why and can fix it.

    CLA enforcement for co-authors

    The CNCF provides a tool for verifying the contributor license agreements on each pull request. AI agents are not able to solve these contributor license agreements so one enforcement the project made is to enable the CLA check for co-authors. This provides a flag to reviewers that the PR is not ready to merge.

    Human engagement required

    Perhaps the most critical aspect of the policy: reviewers expect to engage with humans, not with AI. Contributors cannot rely on AI to respond to review comments. If you cannot personally explain changes that AI helped generate, your PR will be closed. This requirement ensures that knowledge transfer happens and that contributors genuinely understand the code they're submitting.

    Verification obligations

    Contributors must verify AI-generated changes through code review, testing, and personal understanding. It's not enough for the code to work—you need to know why it works and be able to maintain it.

    These policies reflect a mature approach to AI: embrace it as a tool, but never let it replace human judgment, understanding, or responsibility.

    Automated AI reviews

    There exist many tools to aid in reviewing code. AI pull request tools introduce governance challenges so one of the first tasks the community took on was to document the process for what is needed to bring in new AI tools. One of the major evaluation criteria for these tools is to find maintainers willing to test drive them in kubernetes-sigs repositories. Kueue, JobSet and Agent-Sandbox have been experimenting with these tools to provide more support for maintainers.

    Copilot

    One tool that many maintainers started using was GitHub Copilot. The CNCF provides access for maintainers so this ended up being the first tool many started using. It provides some good experience on tuning reviews but there were some growing pains with this tool. The biggest blocker for community adoption is relying on contributors to have a copilot license. Only maintainers were able to request copilot reviews and automated reviews of pull requests was out of reach for the community. One of the goals of AI review tools is to provide an automated review tool that maintainers don't need to request. This demonstrated the need for organization control rather than relying on contributors having access.

    CodeRabbit

    In mid 2026, the Kubernetes community has rolled out CodeRabbit to a few projects. As with copilot, some tuning has been required to provide better reviews but the overall feedback has been positive. There is a lot of configuration available for this tool and one of the most interesting uses of this tool comes from agent-sandbox.

    AI pull request tools can be a quality gate. Contributors can at least get a quick spot check review without waiting for a maintainer. Agent-sandbox has added a label on PRs to reflect that there is still a need to resolve some of the comments from AI tools.

    Next steps

    The reality is that leveraging AI in open source projects is an area of active exploration. The community could use your help in tuning reviews tools, evaluating tools or evaluating emerging technologies in the AI space.

    Some areas we are exploring more:

    • The use of AI skills to reduce maintainer burnout.
    • AI assisted triage of failing tests.
    • Skills to aid the operational aspects of Kubernetes.