Automating a Production Kubernetes Cluster with Ansible & Vagrant — Beyond minikube

Most Kubernetes tutorials point you at minikube or k3s — single-binary tools that hide everything Kubernetes actually does. This project takes a different approach: a production-style multi-node cluster provisioned from scratch using Vagrant and configured end-to-end with Ansible, exposing every step that managed services abstract away.

The result is a three-node Kubernetes cluster — one control plane, two workers — bootstrapped with kubeadm, running containerd as the container runtime and Calico for pod networking. Every step is automated. No manual kubectl commands, no hand-editing config files.

Why Not minikube or k3s?

minikube and k3s are excellent for getting something running fast. They are not excellent for learning what Kubernetes actually does. They hide:

• How the control plane components initialize and talk to each other
• How worker nodes join and register with the API server
• How a CNI plugin establishes pod-to-pod networking across nodes
• How container runtimes are configured and wired into kubelet
• The kernel-level prerequisites Kubernetes needs (swap, sysctl, modules)

Building a cluster with kubeadm forces you to understand all of it. This project automates that process so it's reproducible — but it doesn't hide it.

Cluster Architecture

Ansible Role Architecture

The automation is structured into five purpose-built roles, applied in a deliberate sequence so dependencies are always satisfied before they're needed:

common — OS preparation

Runs on all nodes before anything Kubernetes-related. Disables swap (Kubernetes requires it off), loads necessary kernel modules (br_netfilter, overlay), and sets sysctl parameters for container networking:

# Kernel modules required by Kubernetes
- name: Load kernel modules
  modprobe:
    name: "{{ item }}"
    state: present
  loop:
    - overlay
    - br_netfilter

# Required sysctl settings
- name: Set network sysctl params
  sysctl:
    name: "{{ item.key }}"
    value: "{{ item.value }}"
    sysctl_set: yes
  loop:
    - {key: "net.bridge.bridge-nf-call-iptables",  value: "1"}
    - {key: "net.bridge.bridge-nf-call-ip6tables", value: "1"}
    - {key: "net.ipv4.ip_forward",                  value: "1"}

containerd — container runtime

Installs and configures containerd with SystemdCgroup = true — the setting most tutorials skip, and the one that causes the most silent failures. Without it, kubelet and containerd disagree on cgroup management and the cluster behaves unpredictably.

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

kubernetes — tooling installation

Adds the Kubernetes apt repository, installs kubeadm, kubelet, and kubectl at a pinned version, and holds them to prevent unintended upgrades that would break the cluster.

- name: Install Kubernetes packages
  apt:
    name:
      - kubeadm={{ k8s_version }}-*
      - kubelet={{ k8s_version }}-*
      - kubectl={{ k8s_version }}-*
    state: present

- name: Hold Kubernetes packages at version
  dpkg_selections:
    name: "{{ item }}"
    selection: hold
  loop: [kubeadm, kubelet, kubectl]

master — control plane bootstrap

Runs kubeadm init with the correct pod CIDR for Calico, generates the kubeconfig, deploys the Calico CNI manifest, and extracts the join token — which gets passed to the worker role automatically via Ansible facts.

- name: Initialize control plane
  command: >
    kubeadm init
    --apiserver-advertise-address={{ ansible_host }}
    --pod-network-cidr=192.168.0.0/16
    --node-name=k8s-master

- name: Extract join command
  command: kubeadm token create --print-join-command
  register: join_command

- name: Store join command for workers
  set_fact:
    k8s_join_command: "{{ join_command.stdout }}"

worker — node enrollment

Uses the join command stored as an Ansible fact to attach each worker to the control plane. Fully automated — no manual token copy-paste.

- name: Join worker to cluster
  command: "{{ hostvars['k8s-master']['k8s_join_command'] }}"

The Hardest Bug — Node IP Misconfiguration

Every node initially registered with the same NAT IP address (10.0.2.15) instead of its private network address. The cluster appeared to form correctly, then failed silently — pods scheduled to workers couldn't communicate, and kubectl get nodes showed all nodes at the same IP.

The root cause: Vagrant creates two network interfaces per VM — a NAT interface (eth0) and a private network interface (eth1). By default, kubelet advertises eth0. The fix is to force kubelet to use the correct interface explicitly:

KUBELET_EXTRA_ARGS=--node-ip={{ ansible_host }}

This is passed via the kubelet configuration file in the Ansible role. It's one of the most common issues in multi-node Vagrant + Kubernetes setups and barely documented in official guides.

Deployment — Three Commands

# 1. Provision the VMs
make up

# 2. Configure the entire cluster
cd ansible && ansible-playbook -i inventory/hosts.ini playbooks/site.yml

# 3. Verify
kubectl get nodes
# NAME         STATUS   ROLES
# k8s-master   Ready    control-plane
# worker-1     Ready    <none>
# worker-2     Ready    <none>

Demo Application

An NGINX deployment runs across both worker nodes, exposed via a Kubernetes Service. This validates the full stack: scheduling, pod networking across nodes (via Calico), and service discovery.

kubectl apply -f kubernetes/demo-app/deployment.yaml
kubectl apply -f kubernetes/demo-app/service.yaml
kubectl get pods -o wide
# Shows pods distributed across worker-1 and worker-2

What's Already Working

• Automated three-node cluster provisioning — control plane + 2 workers
• containerd runtime with correct cgroup configuration
• Calico CNI — cross-node pod networking operational
• Automated worker join via Ansible facts (no manual token handling)
• Demo NGINX workload scheduled across both workers
• Full Makefile interface for one-command operations

Roadmap — What's Coming

• Ingress controller (NGINX Ingress) with routing rules
• Persistent storage provisioning (local-path or NFS)
• Prometheus + Grafana monitoring stack on the cluster
• GitHub Actions CI/CD pipeline deploying to the cluster
• Automated health-check playbook for cluster validation

Key Lessons

• Swap must be disabled — Kubernetes refuses to start with swap on. Automate this in common, not as a manual prerequisite
• SystemdCgroup matters — mismatch between containerd and kubelet cgroup drivers causes silent, confusing failures
• Node IP is not automatic in Vagrant — always set --node-ip explicitly on multi-interface VMs
• kubeadm hides less than you think — it still requires you to understand the components it bootstraps
• Ansible facts replace manual token passing — the join command flows automatically from master role to worker role