martelotte.com
← Back to blog

Running Agents on Kubernetes with Agent Sandbox

kubernetesk8sdevopsorchestrationcloud

The year is 2026, and Kubernetes has solidified its reign as the de facto operating system for the cloud-native era. From microservices to AI workloads, everything runs on Kubernetes. Yet, a peculiar class of workload continues to pose unique challenges: agents. These workhorses – be it for monitoring, security, data collection, or infrastructure management – are indispensable. But their often privileged nature, demanding access to host resources or sensitive network endpoints, makes them a prime target for security vulnerabilities and a potential source of instability if not managed with extreme care.

How do we reconcile the need for powerful agents with Kubernetes’ principles of least privilege and strict isolation? Enter the Agent Sandbox: a robust, multi-layered strategy for deploying and managing agents securely and efficiently within your Kubernetes clusters.

The Agent Conundrum: Why Special Treatment?

Agents are not your typical stateless microservices. They often need:

  • Elevated Permissions: Access to host filesystem (hostPath), network interfaces, process IDs (hostPID), or even full privileged mode to collect granular system metrics or enforce security policies.
  • Broad Network Access: Communication with external SaaS platforms, central data collectors, or specific internal services.
  • Resource Guarantees: Continuous operation, often with low latency requirements, meaning they can’t be easily evicted or starved.
  • Persistence: Though not always, some agents might require persistent storage or configuration that isn’t ephemeral.

Allowing such capabilities unchecked across a cluster is a recipe for disaster. A compromised agent can become an attacker’s gateway, granting access to the underlying node or sensitive data. A misconfigured agent can hog resources, impacting critical applications. The Agent Sandbox addresses these concerns head-on.

What is an Agent Sandbox?

An Agent Sandbox is not a single tool or a Kubernetes feature; it’s a methodology. It’s a comprehensive architecture that leverages a combination of Kubernetes native features and cloud-native best practices to create a secure, isolated, and resource-controlled environment specifically for agents. Its core tenets are:

  1. Strict Isolation: Logically separating agents from application workloads.
  2. Least Privilege: Granting only the minimum necessary permissions.
  3. Network Containment: Controlling ingress and egress traffic rigorously.
  4. Resource Governance: Ensuring agents consume appropriate resources without starving other workloads.
  5. Runtime Protection: Monitoring and enforcing security policies at execution.

Let’s build one, layer by layer, targeting Kubernetes 1.30+ environments.

Building Your Agent Sandbox: A Multi-Layered Approach

Layer 1: Dedicated Namespace and Fine-Grained RBAC

The first step is foundational: separate your agents into their own dedicated namespace. This provides the most basic form of logical isolation and simplifies applying policies.

kubectl create namespace agent-sandbox

Next, define a ServiceAccount, Role, and RoleBinding that grant only the necessary API permissions to your agents. Avoid using the default service account.

# agent-service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-agent-sa
  namespace: agent-sandbox
---
# agent-role.yaml
# Example: Role to read Pods and Nodes (e.g., for some network agents)
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-agent-reader
  namespace: agent-sandbox
rules:
- apiGroups: [""]
  resources: ["pods", "nodes"]
  verbs: ["get", "list", "watch"]
---
# agent-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-agent-read-binding
  namespace: agent-sandbox
subjects:
- kind: ServiceAccount
  name: my-agent-sa
  namespace: agent-sandbox
roleRef:
  kind: Role
  name: my-agent-reader
  apiGroup: rbac.authorization.k8s.io

Best Practice: Regularly audit these permissions. Tools like kube-audit or advanced RBAC analyzers (which are standard in 2026 platforms) can help identify over-privileged service accounts.

Layer 2: Core Security with Pod Security Standards (PSS) & Runtime Policies

Kubernetes 1.25+ made Pod Security Standards (PSS) the native way to enforce pod isolation. For agents, this is critical.

  • restricted: The most secure, but often too restrictive for many agents (e.g., disallows hostPath mounts).
  • baseline: A good starting point, disallowing known privilege escalations but allowing more flexibility.
  • privileged: Should be avoided at all costs, unless absolutely unavoidable.

For agents, you’ll likely start with baseline and then use a dynamic admission controller like OPA Gatekeeper or Kyverno to create fine-grained exceptions or enforce custom security contexts for specific agent deployments. This is where the “sandbox” truly takes shape.

Example: Enforcing baseline PSS and a custom security context

First, label your namespace to enforce PSS baseline:

kubectl label --overwrite ns agent-sandbox \
  pod-security.kubernetes.io/enforce=baseline \
  pod-security.kubernetes.io/warn=baseline \
  pod-security.kubernetes.io/audit=baseline

For agents requiring specific capabilities or host mounts (e.g., node-exporter), you’ll define a securityContext in your PodSpec. If it violates baseline, your admission controller (like Kyverno) can be configured to allow it only for specific images/labels within the agent-sandbox namespace.

# Partial DaemonSet for a (hypothetical) node-level agent needing host access
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: my-system-agent
  namespace: agent-sandbox
spec:
  selector:
    matchLabels:
      app: my-system-agent
  template:
    metadata:
      labels:
        app: my-system-agent
    spec:
      serviceAccountName: my-agent-sa
      securityContext:
        runAsNonRoot: true # Always prefer non-root
        runAsUser: 65534   # Use a non-privileged user ID (nobody)
        # SeccompProfile is crucial for restricting syscalls
        seccompProfile:
          type: RuntimeDefault # Use the containerd/docker default seccomp profile
      containers:
      - name: agent
        image: my-system-agent:1.2.0 # Replace with your agent image
        # ... other configurations
        volumeMounts:
        - name: host-root
          mountPath: /host # Mount specific host paths only if absolutely required
          readOnly: true
      volumes:
      - name: host-root
        hostPath:
          path: /
          type: DirectoryReadOnly

Gotcha: Running hostPath volumes or privileged containers will conflict with restricted PSS. Use baseline with an admission controller to carve out specific, tightly controlled exceptions. For the most demanding agents, you might even consider specialized runtime sandboxes.

Layer 3: Network Containment with Network Policies

Prevent agents from communicating unnecessarily with other namespaces or external services. Network Policies are your firewall within the cluster.

# agent-network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: agent-egress-ingress
  namespace: agent-sandbox
spec:
  podSelector: {} # Applies to all pods in the agent-sandbox namespace
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector: {} # Allow ingress from any pod within the agent-sandbox namespace
    # - namespaceSelector: # Optionally allow ingress from specific non-agent namespaces
    #     matchLabels:
    #       name: monitoring # Example: Allow ingress from a 'monitoring' namespace
  egress:
  - to:
    - ipBlock:
        cidr: 10.0.0.0/8 # Example: Allow egress to internal network CIDR
    - ipBlock:
        cidr: 192.168.0.0/16 # Another example internal CIDR
    # Allow egress to external monitoring/logging endpoint (e.g., SaaS provider IP)
    - ipBlock:
        cidr: 203.0.113.0/24 # Example: IP range of your SaaS observability platform
    ports:
    - protocol: TCP
      port: 443 # Only allow HTTPS
    # Allow DNS resolution
  - ports:
    - protocol: UDP
      port: 53
    to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system # Allow DNS to kube-system (coredns)
      podSelector:
        matchLabels:
          k8s-app: kube-dns # Specific pod selector for CoreDNS

Best Practice: Start with a “deny-all” egress policy for agent-sandbox and then explicitly whitelist only the necessary endpoints (internal services, external SaaS APIs). Regularly review and update these policies.

Layer 4: Resource Guardrails with Quotas and Limit Ranges

Agents, especially those with high data throughput or continuous processing, can be resource hogs. Use ResourceQuota and LimitRange to prevent resource starvation for other workloads.

# agent-resource-quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: agent-quota
  namespace: agent-sandbox
spec:
  hard:
    pods: "20"
    requests.cpu: "4" # Total CPU requests for all agents in the namespace
    requests.memory: "8Gi"
    limits.cpu: "8"   # Total CPU limits
    limits.memory: "16Gi"
---
# agent-limit-range.yaml
apiVersion: v1
kind: LimitRange
metadata:
  name: agent-limits
  namespace: agent-sandbox
spec:
  limits:
  - default:
      cpu: 500m
      memory: 512Mi
    defaultRequest:
      cpu: 100m
      memory: 128Mi
    type: Container

Performance Consideration: Setting requests and limits correctly is crucial. Under-provisioning can lead to CrashLoopBackoff or performance degradation for the agent. Over-provisioning wastes resources and can impact overall cluster stability. Monitor agent resource usage closely to fine-tune these values.

Layer 5: Enhanced Runtime Isolation (2026 Perspective)

For agents requiring truly deep access (e.g., next-gen eBPF-based security agents, deep network visibility tools) or if you’re running multi-tenant clusters where strong workload separation is paramount, consider hardened runtimes:

  • gVisor: An OCI runtime that intercepts syscalls, effectively creating a user-space kernel for the container. It significantly reduces the attack surface but introduces some performance overhead.
  • Kata Containers: Uses lightweight virtual machines (VMs) for each pod, providing hardware-level isolation. Higher overhead than gVisor but offers stronger isolation guarantees.

These runtimes are becoming more integrated and easier to deploy in Kubernetes 1.30+ and beyond, often managed via a RuntimeClass resource.

# Example RuntimeClass for gVisor
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc # Or 'kata' for Kata Containers

Then, reference it in your PodSpec:

# ... inside a PodSpec for an agent
spec:
  runtimeClassName: gvisor
  # ... rest of your pod definition

Trade-offs: While offering superior isolation, gVisor and Kata Containers introduce performance overhead and increase the operational complexity of your cluster. Reserve them for your most critical and high-risk agents.

Layer 6: Runtime Security and Observability within the Sandbox

Even within a sandbox, continuous monitoring and runtime security are essential. Deploy eBPF-based security tools like Falco or Tetragon, and OpenTelemetry agents within the agent-sandbox itself. These tools can monitor the behavior of your agents, detect anomalies, and enforce policies at the syscall level, providing an additional layer of defense that extends beyond static configurations.

Practical Example: Securing Prometheus Node Exporter

Let’s apply these principles to a common agent: the Prometheus Node Exporter. It requires access to the host filesystem to collect metrics.

# prometheus-node-exporter-ds.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: agent-sandbox
  labels:
    app.kubernetes.io/name: node-exporter
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: node-exporter
  template:
    metadata:
      labels:
        app.kubernetes.io/name: node-exporter
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9100"
    spec:
      serviceAccountName: my-agent-sa # Use our dedicated SA
      hostPID: true # Required for some process metrics, but carefully considered
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534 # Nobody user
        fsGroup: 65534 # Ensure filesystem access for group too
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: node-exporter
        image: quay.io/prometheus/node-exporter:v1.6.1 # Assuming this is a recent stable version in 2026
        args:
          - --path.sysfs=/host/sys
          - --path.rootfs=/host/root
          - --path.proc=/host/proc
          - --web.listen-address=:9100
        ports:
          - name: http-metrics
            containerPort: 9100
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 200m
            memory: 256Mi
        volumeMounts:
          - name: proc
            mountPath: /host/proc
            readOnly: true
          - name: sys
            mountPath: /host/sys
            readOnly: true
          - name: root
            mountPath: /host/root
            readOnly: true
      volumes:
        - name: proc
          hostPath:
            path: /proc
        - name: sys
          hostPath:
            path: /sys
        - name: root
          hostPath:
            path: /
            type: DirectoryReadOnly

This DaemonSet demonstrates a Node Exporter running in our agent-sandbox. Note the hostPID: true, hostPath volumes, and specific securityContext settings. While hostPID and hostPath are powerful, we mitigate risks by:

  • Running runAsNonRoot with a non-privileged user.
  • Using seccompProfile: RuntimeDefault to restrict syscalls.
  • Making hostPath mounts readOnly where possible.
  • Enforcing Network Policies to restrict where node-exporter can send data.

Common Pitfalls and Troubleshooting

  • Permission Denied (RBAC): Agent fails to start or interact with the Kubernetes API. Check kubectl auth can-i <verb> <resource> -n agent-sandbox --as=system:serviceaccount:agent-sandbox:my-agent-sa.
  • Permission Denied (PSS/SecurityContext): Pod fails to schedule or crashes due to securityContext violations or host access issues. Check kubectl describe pod <pod-name> -n agent-sandbox for admission controller errors indicating PSS or policy violations.
  • Network Connectivity: Agent can’t reach its data sink or external API. Use kubectl exec into the agent pod, curl or ping target endpoints, and review NetworkPolicy logs (if your CNI supports it) or kubectl describe networkpolicy.
  • Resource Starvation: Agent performs poorly or restarts frequently. Check kubectl top pod -n agent-sandbox and kubectl logs for OOMKilled messages. Adjust requests and limits in the LimitRange or directly in the agent’s PodSpec.
  • Incorrect HostPath: Agent fails to find expected files on the host. Double-check hostPath definitions and agent configurations for correct paths, especially if the agent expects specific symlinks or directory structures.

Actionable Takeaways

Running agents effectively on Kubernetes in 2026 demands a sophisticated approach. The Agent Sandbox methodology provides a robust framework:

  1. Isolate with Namespaces & RBAC: Dedicate a namespace and grant agents minimal API permissions.
  2. Harden with PSS & Policy Engines: Use baseline PSS with dynamic admission controllers (OPA/Kyverno) for tightly controlled exceptions, moving beyond the capabilities of deprecated PodSecurityPolicy resources.
  3. Contain with Network Policies: Implement strict egress/ingress rules, whitelisting only essential communication pathways.
  4. Govern Resources: Apply ResourceQuota and LimitRange to prevent resource hogging and ensure predictable performance.
  5. Consider Enhanced Runtimes: For extreme isolation, explore gVisor or Kata Containers with RuntimeClass for high-risk workloads.
  6. Monitor with Runtime Security: Deploy eBPF-based tools within the sandbox for real-time threat detection and behavioral analysis.

By meticulously implementing these layers, you transform potentially risky agents into trusted, secure, and performant components of your cloud-native infrastructure, maintaining operational excellence even as your clusters scale.

Discussion Questions

  1. What are the most challenging “agent types” you encounter in your Kubernetes environments today, and what specific security or isolation requirements do they present that push the boundaries of current sandboxing techniques?
  2. Beyond the techniques discussed, what advanced Kubernetes security features or third-party tools do you foresee becoming indispensable for agent sandboxing in the next 1-2 years, especially concerning supply chain security and AI-driven threat detection?

Comments

No comments yet. Be the first!

We use cookies to enhance your browsing experience, serve personalized content, and analyze our traffic. By clicking "OK", you consent to our use of cookies and agree to our Terms of Service & Privacy Policy.