Self-Healing Docker Services — Health Checks, Restart Policies & Watchdog

A container that crashes and stays down is a production incident. A container that crashes and restarts itself in under 30 seconds is just a blip in the metrics. This project implements Docker-native self-healing — health checks, restart policies, and a watchdog — so your services recover automatically without waking anyone up.

The Three Layers of Self-Healing

Most engineers only implement Layer 1. This project implements all three.

Layer 1 — Docker Restart Policies

The simplest form of self-healing. If the container process exits, Docker restarts it automatically.

services:
  app:
    image: myapp:latest
    restart: unless-stopped   # restart on crash, respect manual stops
    # Options:
    # no            - never restart (default)
    # always        - restart always, including on boot
    # on-failure    - restart only on non-zero exit codes
    # unless-stopped - restart always except when manually stopped

Layer 2 — Health Checks

A running container isn't necessarily a healthy container. A web server can be running but returning 500 errors. A database can be running but refusing connections. Health checks test the actual service behaviour, not just the process state.

services:
  app:
    image: myapp:latest
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s      # check every 30 seconds
      timeout: 10s       # fail if no response in 10s
      retries: 3         # mark unhealthy after 3 consecutive failures
      start_period: 40s  # grace period on container start

Once a container is marked unhealthy, Docker logs the event, updates the container status, and other services can query it. Combined with restart policies, this handles most production failure scenarios.

Layer 3 — Watchdog Service

The watchdog catches what health checks miss — network timeouts, partial failures, third-party dependency outages. It runs as a separate container and actively polls your services, triggering recovery actions when needed.

import docker
import time
import requests

client = docker.from_env()

def check_and_heal(service_name, health_url):
    try:
        r = requests.get(health_url, timeout=5)
        if r.status_code != 200:
            raise Exception(f"Bad status: {r.status_code}")
    except Exception as e:
        print(f"[HEAL] {service_name} unhealthy: {e}")
        container = client.containers.get(service_name)
        container.restart()
        print(f"[HEAL] {service_name} restarted")

while True:
    check_and_heal("web-app", "http://web-app:8080/health")
    time.sleep(30)

Complete Docker Compose Setup

version: '3.8'
services:
  app:
    image: myapp:1.0.0
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      retries: 3
    ports: ["8080:8080"]

  watchdog:
    build: ./watchdog
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      app:
        condition: service_healthy

  prometheus:
    image: prom/prometheus:latest
    restart: unless-stopped
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports: ["9090:9090"]

Key Lessons

• Health check endpoints should test real service behaviour — not just return 200 unconditionally
• The watchdog needs access to the Docker socket — mount /var/run/docker.sock carefully and restrict its permissions
• Log every recovery event with a timestamp — when the next incident happens, you'll want the history
• Set start_period in health checks — give services time to initialise before the first health check fires

What's Next

• Slack/webhook alerts when recovery events occur
• Prometheus metrics for recovery event counts and MTTR
• Kubernetes migration — replacing this with liveness and readiness probes