Beyond CRDs: Kubernetes API Governance for Platform Engineers

Remember the early days of Kubernetes, when a simple Pod or Deployment was enough to run your application? Fast forward to 2026, and the landscape is unrecognizable. Kubernetes isn’t just an orchestrator; it’s a programmable operating system for your entire cloud-native estate. At its heart lies the Kubernetes API – a powerful, extensible interface that platform engineers leverage daily to build sophisticated control planes, automated workflows, and self-service platforms.

The true enabler of this extensibility? Custom Resource Definitions (CRDs). They’ve moved from an experimental feature to the bedrock of modern Kubernetes operations, allowing us to define custom resources like DatabaseCluster, ManagedKafkaTopic, or ApplicationRollout. But with great power comes great complexity. What started as a clever way to extend Kubernetes has, for many organizations, devolved into a sprawling, undocumented, and often chaotic collection of custom APIs.

This “CRD explosion” is the silent killer of platform engineering productivity and operational excellence. It’s time to move beyond merely defining CRDs and start governing them. This post will delve into the critical aspects of Kubernetes API governance for platform engineers, offering actionable strategies to tame the custom resource wilderness and ensure your platform remains robust, performant, and delightful for developers.

The CRD Wild West: Why We Need Governance

Think about your internal developer platform or your core infrastructure services. Chances are, they’re built on a stack of custom controllers and their corresponding CRDs. While immensely powerful, an unmanaged custom API surface introduces significant challenges:

• Schema Drift and Incompatibility: Without strict versioning and validation, an innocent change to a CRD schema can break existing controllers, CI/CD pipelines, or even production workloads. The dreaded v1alpha1 that never matured often lingers indefinitely.
• Performance Bottlenecks: Every CRD introduces new objects that the API server must manage. Inefficient controllers watching too many objects, large custom resources, or frequent updates can put undue strain on the API server, impacting overall cluster performance and stability.
• Security Risks: Over-privileged custom controllers, lack of granular access control for custom resources, and unvalidated inputs can create critical security vulnerabilities, especially in multi-tenant environments.
• Developer Experience (DX) Degradation: Inconsistent API design, poor documentation, and the absence of clear lifecycle management make it hard for application teams to adopt and trust your platform. Developers spend more time deciphering obscure fields than building features.
• Operational Complexity: Troubleshooting becomes a nightmare when you’re dealing with hundreds of custom resources, each with its own implicit rules, interdependencies, and opaque states. Upgrades become perilous.

The goal of Kubernetes API governance is not to stifle innovation but to channel it, providing guardrails that ensure custom APIs are reliable, secure, and maintainable throughout their lifecycle.

Building Your API Governance Framework: Practical Solutions

Let’s explore concrete strategies and tools to bring order to your custom API landscape.

1. Robust API Versioning and Deprecation Strategies

Just like core Kubernetes APIs, your custom APIs need a clear versioning strategy. Don’t let v1alpha1 be the default for production-ready APIs.

Best Practices:

• Standardize Naming: Use v1alpha1, v1beta1, v1 progression. alpha for experimental, beta for testing/feedback, v1 for stable.
• Multi-Version Support: Design your CRDs to support multiple API versions concurrently during migration phases. Use the storage field to designate the authoritative version.
• Graceful Deprecation: Provide clear upgrade paths and communicate deprecations well in advance. Leverage admission webhooks for early warnings.

Example: Multi-version CRD Manifest

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.platform.example.com
spec:
  group: platform.example.com
  names:
    plural: databases
    singular: database
    kind: Database
    shortNames: ["db"]
  scope: Namespaced
  versions:
    - name: v1beta1
      served: true
      storage: false # Not the storage version, good for migration phase
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                engine: { type: string, description: "Database engine (e.g., postgres, mysql)" }
                version: { type: string, description: "Engine version" }
                size: { type: string, pattern: "^(small|medium|large)$" }
                # ... other v1beta1 fields
    - name: v1
      served: true
      storage: true # This is the storage version, new resources will be stored in this format
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: ["engine", "version", "instanceType"] # Now instanceType is required in v1
              properties:
                engine: { type: string, description: "Database engine (e.g., postgres, mysql)" }
                version: { type: string, description: "Engine version" }
                instanceType: { type: string, pattern: "^(db.t3.micro|db.m5.large)$", description: "Cloud provider instance type" }
                # ... other v1 fields
      subresources:
        status: {}
    # - name: v2alpha1 # Potentially a future version with breaking changes
    #   served: false # Not yet served to clients
    #   storage: false

When v1 becomes storage: true, existing v1beta1 resources are transparently converted by the API server when read or updated. kubectl convert can also be used for explicit client-side conversions.

2. Strict Schema Validation with OpenAPIv3 and CEL

Weak validation is a recipe for disaster. Leverage the full power of OpenAPIv3 schema validation, including advanced features like x-kubernetes-validations (Common Expression Language - CEL) for complex, cross-field validation rules.

Best Practices:

• Define Required Fields: Explicitly mark required fields.
• Enforce Types and Formats: Use type, format, pattern for basic validation.
• Advanced Validations (CEL): For business logic (e.g., “if engine is postgres, then version must be 14.x or higher”), CEL expressions are invaluable and performant.

Example: Advanced CRD Validation

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.platform.example.com
spec:
  # ... (group, names, scope)
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: ["engine", "version", "instanceType"]
              properties:
                engine:
                  type: string
                  enum: ["postgres", "mysql"] # Restrict allowed engines
                version:
                  type: string
                  # Use a regex for version format, e.g., '14.x', '8.0'
                  pattern: "^(1[0-6]\\.\\d+|8\\.\\d+)$"
                instanceType:
                  type: string
                  enum: ["db.t3.micro", "db.m5.large", "db.r5.xlarge"]
                storageGB:
                  type: integer
                  minimum: 20
                  maximum: 1000
              # x-kubernetes-validations using CEL for cross-field logic (Kubernetes 1.25+)
              x-kubernetes-validations:
                - rule: "self.engine == 'postgres' ? self.storageGB >= 50 : true"
                  message: "Postgres databases require at least 50GB storage."
                - rule: "self.instanceType.startsWith('db.t3') ? self.storageGB <= 200 : true"
                  message: "db.t3 instance types are limited to 200GB storage."
            status:
              type: object
              properties:
                phase: { type: string, enum: ["Provisioning", "Ready", "Failed"] }
                connectionString: { type: string }

For even more complex, dynamic, or external validations (e.g., checking against a service catalog), ValidatingAdmissionWebhooks are necessary.

3. Performance-Conscious Controller Design

Inefficient custom controllers can quickly overload the Kubernetes API server. Platform engineers must prioritize performance.

Best Practices:

• Efficient Watches: Use fieldSelector and labelSelector in your informers to narrow down the scope of watched resources.
• Event-Driven Reconciliation: Design controllers to react to specific events rather than constant polling.
• Debounce and Batching: Implement rate-limiting and event batching to prevent rapid, unnecessary reconciliation loops.
• Avoid Large Objects: Keep custom resource objects concise. Store large, volatile data elsewhere (e.g., S3, separate database) and reference it.
• Garbage Collection with OwnerReferences: Correctly use ownerReferences to ensure dependent resources are automatically cleaned up when the owner is deleted, preventing orphaned objects and reducing API server load.

Example: ownerReference in a Child Resource

apiVersion: apps.kubernetes.io/v1
kind: Deployment
metadata:
  name: my-app-backend
  labels:
    app: my-app
spec:
  # ...
---
apiVersion: v1
kind: Service
metadata:
  name: my-app-service
  labels:
    app: my-app
  ownerReferences: # This tells K8s that this service is "owned" by the Deployment
    - apiVersion: apps.kubernetes.io/v1
      kind: Deployment
      name: my-app-backend
      uid: <UID_OF_DEPLOYMENT> # K8s will fill this in automatically when created by a controller
      controller: true # Only one ownerReference can have controller: true
spec:
  # ...

While the above example shows core K8s resources, the principle applies directly to custom controllers managing child resources.

4. Granular RBAC for Custom Resources

Treat access to custom resources with the same scrutiny as core Kubernetes resources. Avoid giving broad * permissions to custom controllers or users.

Best Practices:

• Least Privilege: Grant only the minimum necessary permissions for controllers and users to interact with specific CRDs.
• Scoped Roles: Create ClusterRole or Role definitions that specifically target your custom resources and verbs (e.g., get, list, watch, create, update, delete).
• Audit Regularly: Periodically review ClusterRoleBindings and RoleBindings to ensure no over-privileged access exists.

Example: Specific RBAC for a Custom Resource

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: database-operator-role
rules:
  - apiGroups: ["platform.example.com"]
    resources: ["databases"]
    verbs: ["get", "list", "watch", "update", "patch"] # Operator needs to watch and update status
  - apiGroups: ["platform.example.com"]
    resources: ["databases/status"] # Separate permission for status updates
    verbs: ["update", "patch"]
  - apiGroups: [""] # For core K8s resources managed by the operator
    resources: ["secrets", "services", "deployments"]
    verbs: ["get", "list", "watch", "create", "update", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: bind-database-operator-role
subjects:
  - kind: ServiceAccount
    name: database-operator-sa
    namespace: platform-system
roleRef:
  kind: ClusterRole
  name: database-operator-role
  apiGroup: rbac.authorization.k8s.io

5. Centralized Policy Enforcement with Admission Controllers

For enforcing organization-wide policies that go beyond basic schema validation, Validating and Mutating Admission Webhooks are essential. Tools like Kyverno or OPA Gatekeeper provide a powerful, declarative way to manage these policies.

Best Practices:

• Policy-as-Code: Define policies in source control alongside your infrastructure.
• Auditing and Enforcement: Use policies to audit non-compliant resources (e.g., CRs without required labels) before enforcing stricter rules.
• Centralized Management: Deploy a single policy engine for all admission control needs.

Example: Kyverno Policy for Database CRD

Let’s imagine a policy requiring all Database resources to have an owner label.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-database-owner-label
spec:
  validationFailureAction: Enforce # Or Audit for testing
  rules:
    - name: check-for-owner-label
      match:
        any:
        - resources:
            kinds:
              - platform.example.com/v1/Database
      validate:
        message: "Database resources must have an 'owner' label specifying the responsible team."
        pattern:
          metadata:
            labels:
              owner: "?*" # Requires the owner label to exist and have a non-empty value

This policy would prevent the creation of Database resources that lack the owner label, providing immediate feedback to developers.

6. Documentation and Discoverability

A great API is useless if no one knows how to use it. Clear, accessible documentation is non-negotiable.

Best Practices:

• In-Schema Descriptions: Use the description field in your OpenAPIv3 schema extensively. Tools can often generate docs directly from this.
• Examples: Provide multiple, well-explained examples of your custom resources.
• Developer Portals: Integrate your custom API documentation into an internal developer portal (e.g., Backstage) for a unified experience.
• CRD-to-Doc Tools: Explore tools like crd-docs or custom scripts that generate human-readable documentation from your CRD definitions.

Troubleshooting and Common Pitfalls

• API Server Overload: If your API server metrics (e.g., apiserver_request_total, apiserver_request_duration_seconds) show spikes related to custom resource types, investigate your controllers for inefficient watches or high-frequency updates.
• Validation Errors: Use kubectl describe on your custom resource to see validation errors. If using admission webhooks, check the webhook controller logs for policy violations.
• Controller Reconciliation Loops: If a controller constantly updates a resource without converging, it might be due to a faulty reconciliation logic or an external dependency issue. Check controller logs and the custom resource’s status block.
• Breaking Changes: Always test CRD updates in non-production environments first. If a breaking change is deployed, ensure you have a rollback strategy or a conversion webhook.

Actionable Takeaways for Platform Engineers

Kubernetes API governance isn’t a luxury; it’s a necessity for scaling your cloud-native operations in 2026 and beyond.

1. Treat Custom APIs as First-Class Citizens: Apply the same rigor to your CRDs as you would to any public API product.
2. Standardize and Enforce: Establish clear guidelines for versioning, schema design, and security, then use tools like admission controllers to enforce them.
3. Prioritize Performance: Design controllers with efficiency in mind to prevent API server strain.
4. Invest in DX: Good documentation and consistent API design foster adoption and reduce cognitive load for developers.
5. Automate Policy: Leverage Policy-as-Code solutions to manage your API governance at scale.

By implementing these strategies, platform engineers can transform their custom Kubernetes API landscape from a source of chaos into a robust, secure, and performant foundation for innovation.

---

What are your biggest challenges in managing custom Kubernetes APIs today?

How do you balance the need for developer agility and innovation with strict API governance in your organization?