From Security Blocked to Prod Ready: ClickHouse on Docker Hardened Images

Ah, ClickHouse. The analytical database that laughs in the face of petabytes, delivering query speeds that make traditional systems weep. It’s a DevOps dream for data engineers and a performance hero for analysts. Naturally, many of us reach for Docker to containerize this powerhouse, enjoying its portability, scalability, and ease of deployment.

But then, reality bites. Your security team, bless their diligent hearts, takes one look at your docker run clickhouse/clickhouse-server command or a basic Dockerfile and promptly slams the brakes. “Running as root? Too many open ports? Unnecessary packages? No way this is going to production!”

Sound familiar? Welcome to 2026, where security isn’t an afterthought; it’s baked into every layer of our infrastructure, especially when deploying critical data services. While ClickHouse itself is robust, throwing it into a default, unhardened Docker container is like putting a supercar in a cardboard box. It might work, but it’s not safe, secure, or sustainable.

This post isn’t about shying away from Docker for ClickHouse. It’s about embracing the power of containerization while meeting stringent security and operational standards. We’re going to transform our ClickHouse Docker images from security liabilities into production-ready assets.

The Core Problem: Default Images and Your Security Posture

The official ClickHouse Docker images (e.g., clickhouse/clickhouse-server:24.4.2.19-lts) are fantastic starting points. They offer convenience, but by design, they prioritize broad compatibility and ease of use over a minimal, hardened production footprint. Here are the common culprits that trigger security alerts:

1. Root User Execution: Processes inside containers often run as root by default. If an attacker breaches the container, they gain root privileges within it, potentially simplifying privilege escalation to the host system.
2. Bloated Base Images: Many default images are built on larger distributions, including a plethora of utilities (bash, ping, apt, curl, wget) unnecessary for a running ClickHouse server. Every added package is a potential vulnerability.
3. Lack of Resource Limits: Without explicit CPU, memory, and I/O limits, a rogue query or misconfigured ClickHouse instance can starve other services or even bring down the host.
4. No Health Checks: A container merely “running” doesn’t mean the service inside is healthy. Without proper health checks, your orchestrator (Kubernetes, Docker Swarm) might keep sending traffic to a non-responsive ClickHouse instance.
5. Sensitive Configuration in Image Layers: Baking passwords or full configuration files directly into image layers makes them discoverable and immutable, posing significant security risks.

Our mission is clear: mitigate these risks and build a ClickHouse container that our security team will love as much as our data team loves its performance.

Solution 1: Multi-Stage Builds and a Minimal Base Image

The first step to a hardened image is minimizing its attack surface. This is where multi-stage builds shine. We’ll use a builder stage with all necessary tools, and then copy only the essential runtime artifacts to a much smaller runtime stage. For ClickHouse, ubuntu:24.04-slim or debian:12-slim are excellent starting points for the runtime image, offering a good balance between size and compatibility.

Let’s refine a Dockerfile for ClickHouse 24.x LTS:

# syntax=docker/dockerfile:1.7-experimental

# --- Stage 1: Builder for ClickHouse ---
FROM ubuntu:24.04 AS builder

LABEL maintainer="Your DevOps Team"

# Install necessary packages for downloading and installing ClickHouse
RUN apt-get update && \
    apt-get install -y --no-install-recommends ca-certificates curl gnupg dirmngr && \
    rm -rf /var/lib/apt/lists/*

# Add ClickHouse GPG key and APT repository
RUN mkdir -p /etc/apt/keyrings && \
    curl -fsSL https://packages.clickhouse.com/CA/ClickHousePublicDogtag.asc | gpg --dearmor > /etc/apt/keyrings/clickhouse.gpg && \
    echo "deb [signed-by=/etc/apt/keyrings/clickhouse.gpg] https://packages.clickhouse.com/deb stable main" > /etc/apt/sources.list.d/clickhouse.list

# Install ClickHouse server
RUN apt-get update && \
    apt-get install -y --no-install-recommends clickhouse-server=24.4.2.19-lts && \
    rm -rf /var/lib/apt/lists/* && \
    apt-get clean

# --- Stage 2: Runtime Image ---
FROM ubuntu:24.04-slim

ARG CLICKHOUSE_USER="clickhouse"
ARG CLICKHOUSE_UID="999"
ARG CLICKHOUSE_GID="999"

RUN groupadd --gid ${CLICKHOUSE_GID} ${CLICKHOUSE_USER} && \
    useradd --uid ${CLICKHOUSE_UID} --gid ${CLICKHOUSE_GID} --shell /bin/false --no-create-home ${CLICKHOUSE_USER}

# Copy ClickHouse binaries and libraries
COPY --from=builder /usr/bin/clickhouse /usr/bin/clickhouse
COPY --from=builder /usr/lib/clickhouse /usr/lib/clickhouse
COPY --from=builder /usr/share/clickhouse /usr/share/clickhouse

# Create directories and set permissions for the ClickHouse user
RUN mkdir -p /var/lib/clickhouse \
           /var/log/clickhouse-server \
           /etc/clickhouse-server \
           /etc/clickhouse-client \
    && chown -R ${CLICKHOUSE_USER}:${CLICKHOUSE_USER} /var/lib/clickhouse \
                                                     /var/log/clickhouse-server \
    && chmod 750 /var/lib/clickhouse \
                 /var/log/clickhouse-server

# Copy default configurations (will be overridden by bind mounts in prod)
COPY --from=builder /etc/clickhouse-server/config.xml /etc/clickhouse-server/config.xml
COPY --from=builder /etc/clickhouse-server/users.xml /etc/clickhouse-server/users.xml

EXPOSE 8123 8443 9000 9440

USER ${CLICKHOUSE_USER}

HEALTHCHECK --interval=30s --timeout=10s --retries=5 \
    CMD clickhouse-client --query "SELECT 1" || exit 1

VOLUME /var/lib/clickhouse
VOLUME /var/log/clickhouse-server
VOLUME /etc/clickhouse-server

ENTRYPOINT ["/usr/bin/clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml"]

Key Hardening Actions in the Dockerfile:

• FROM ubuntu:24.04-slim: A much smaller base image than full Ubuntu.
• Non-Root User: We create clickhouse user with specific UID/GID and then use USER clickhouse. All runtime operations are performed under this user.
• Minimal Copies: Only copy exactly what’s needed from the builder stage.
• Dedicated Directories: Explicitly create and set permissions for data and logs.
• HEALTHCHECK: Ensures the ClickHouse server is truly responsive, not just running.
• VOLUME: Clearly defines where persistent data should be mounted, preventing accidental data loss when containers are destroyed.

To build this, ensure you have BuildKit enabled (it’s often default in modern Docker Desktop/Engine, or set DOCKER_BUILDKIT=1):

docker build -t my-hardened-clickhouse:24.4.2.19-lts .

Solution 2: Runtime Hardening with Docker Compose (or Kubernetes)

Building a secure image is half the battle. Running it securely is the other. Here’s how to integrate our hardened image into a docker-compose.yaml (easily translatable to Kubernetes deployments):

version: '3.8'

services:
  clickhouse-server:
    image: my-hardened-clickhouse:24.4.2.19-lts
    container_name: clickhouse-hardened
    ports:
      - "8123:8123" # HTTP
      - "9000:9000" # Native TCP
    volumes:
      - clickhouse_data:/var/lib/clickhouse
      - clickhouse_log:/var/log/clickhouse-server
      # Bind mount production configuration
      - ./config/config.xml:/etc/clickhouse-server/config.xml:ro
      - ./config/users.xml:/etc/clickhouse-server/users.xml:ro
    environment:
      # Use env vars for sensitive configs (e.g., initial password)
      CLICKHOUSE_DEFAULT_PASSWORD: ${CLICKHOUSE_DEFAULT_PASSWORD:-myStrongPassword123}
    ulimits:
      nofile:
        soft: 262144
        hard: 262144
    deploy: # Resource limits for orchestrators
      resources:
        limits:
          cpus: '4.0'
          memory: '16G'
        reservations:
          cpus: '2.0'
          memory: '8G'
    restart: unless-stopped
    security_opt: # Advanced security
      - no-new-privileges:true
      # Consider custom seccomp profiles for tighter control
      # - seccomp=./seccomp/clickhouse-seccomp.json
      
volumes:
  clickhouse_data:
  clickhouse_log:

Key Runtime Hardening Actions:

• Bind Mounts for Configuration: config.xml and users.xml are mounted from the host as read-only (:ro). This keeps sensitive configuration out of image layers and allows dynamic updates without rebuilding the image.
• Secrets Management: Use environment variables (CLICKHOUSE_DEFAULT_PASSWORD) for development/testing. For production, integrate with orchestrator-native secret management (Docker Secrets, Kubernetes Secrets, HashiCorp Vault).
• Resource Limits (deploy.resources / --cpus, --memory): Essential for stable operations, preventing resource hogging. ClickHouse is memory-intensive, so fine-tune these based on your workload.
• ulimits: ClickHouse can open many files (e.g., for parts files in merge tree tables). Increase nofile limit to prevent issues.
• security_opt: no-new-privileges:true: Prevents the container from gaining new privileges, even if an attacker finds an exploit that would normally allow it.
• Consider Custom Seccomp/AppArmor Profiles: For the ultimate security, create a custom seccomp profile that whitelists only the necessary system calls for ClickHouse. This is advanced but highly effective.

Troubleshooting and Common Pitfalls

1. Permission Denied Errors (Non-Root User): This is the most common issue. Ensure that the CLICKHOUSE_USER (e.g., clickhouse) has ownership and appropriate read/write permissions for /var/lib/clickhouse, /var/log/clickhouse-server, and any configuration files it needs to modify. Double-check chown and chmod commands in your Dockerfile.
2. Missing Libraries/Dependencies: When switching to a slim base image, ldd /usr/bin/clickhouse on the builder stage is your best friend. It lists all shared libraries the binary depends on. You might need to explicitly install packages like libssl3, libicu74, etc., in your runtime stage if they are not part of ubuntu:24.04-slim.
3. ClickHouse Not Starting/Exiting Immediately: Check container logs (docker logs <container_id>). It’s often a configuration error (config.xml, users.xml), permission issue, or a missing dependency.
4. Performance Degradation:
   • Insufficient Resource Limits: ClickHouse needs CPU and RAM. Ensure your limits (cpus, memory) are adequate for your workload.
   • Slow Storage: ClickHouse performs best on fast NVMe SSDs. If using network storage, ensure it can deliver the required IOPS. Docker volumes on a slow underlying disk will hinder performance.

The Future of Container Security: What to Watch For

As we move deeper into 2026, keep an eye on:

• Enhanced SBOM (Software Bill of Materials) Generation: Making it easier to generate comprehensive SBOMs for full transparency.
• Runtime Observability & Behavioral Analysis: Detecting anomalous behavior within containers at runtime.
• Widespread Adoption of Container Sandboxing: Technologies like gVisor and Kata Containers offering stronger isolation.
• Automated Policy Enforcement: Closer integration of security policies directly into CI/CD pipelines.

Actionable Takeaways

• Never run as root in production containers. Always create a dedicated non-root user.
• Embrace multi-stage builds to drastically reduce image size and attack surface.
• Bind-mount configurations as read-only volumes, especially for production. Avoid baking secrets into images.
• Implement robust health checks to ensure service availability, not just container liveness.
• Set explicit resource limits to prevent resource starvation and ensure predictable performance.
• Leverage advanced security options like no-new-privileges and explore custom seccomp profiles.

By following these principles, you’ll not only satisfy your security team but also build a more resilient, performant, and maintainable ClickHouse deployment on Docker.

What are your thoughts?

1. What’s the trickiest security hurdle you’ve faced when deploying data-intensive applications like ClickHouse in containers?
2. Beyond what we covered, what advanced security practices are you implementing for your containerized databases in 2026?

Looking forward to hearing your insights!