Kubernetes on Wael Nasreddine

💀 How I Accidentally DDoSed My Own Kubernetes Cluster

Mon, 01 Jan 0001 00:00:00 +0000

Introduction Link to heading

For the past few weeks, my production Kubernetes cluster has been fighting a ghost. Every few days (or sometimes hours), the entire Control Plane would commit suicide. Critical components like the scheduler and controller manager would crash simultaneously, claiming they had "lost leadership."

Usually, this happens when the network fails. But my network was fine. The nodes were up. The CPUs were idling.

🔒Locking Down Mastodon: Cilium Network Policies & Audit Mode

Mon, 01 Jan 0001 00:00:00 +0000

The Goal: Defense in Depth Link to heading

Running Mastodon on my bare metal cluster exposes it to the wider internet for federation. If the Mastodon container were ever compromised (RCE, supply chain attack, etc.), I need to ensure the attacker cannot use it as a jumpbox to scan my internal home lab network or access sensitive data stores (Postgres/Redis) indiscriminately.

The objective was to implement a Zero Trust model using Cilium Network Policies (CNP):

🚨Alerting on Kyverno Policy Validation Failures with Loki

Mon, 01 Jan 0001 00:00:00 +0000

Context Link to heading

We needed to create an alert for Kyverno policy violations across our clusters. The goal was to catch when a resource fails validation (e.g., policy="require-requests-limits") and surface relevant metadata like the resource name, kind, and target namespace directly in the alert.

The Data Link to heading

Kyverno logs validation failures to stderr in JSON format. Below is a trimmed example of the relevant fields we extract:

Cilium Network Policies

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
 name: mastodon-egress-federation
 namespace: mastodon
spec:
 egress:
 - toCIDRSet:
 - cidr: 0.0.0.0/0
 except:
 - 192.168.0.0/16
 - 100.64.0.0/10
 - 10.244.0.0/16
 - 10.96.0.0/12
 endpointSelector:
 matchLabels:
 app.kubernetes.io/name: mastodon
---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
 name: mastodon-infra-core
 namespace: mastodon
spec:
 egress:
 - toEndpoints:
 - matchLabels:
 io.kubernetes.pod.namespace: kube-system
 k8s-app: kube-dns
 toPorts:
 - ports:
 - port: "53"
 protocol: UDP
 rules:
 dns:
 - matchPattern: '*'
 - toEndpoints:
 - matchLabels:
 io.kubernetes.pod.namespace: pg-vchord-cluster
 toPorts:
 - ports:
 - port: "5432"
 protocol: TCP
 - toEndpoints:
 - matchLabels:
 k8s:io.kubernetes.pod.namespace: minio
 v1.min.io/tenant: production
 toPorts:
 - ports:
 - port: "9000"
 protocol: TCP
 - toFQDNs:
 - matchName: mastodon.redis.prod.nasreddine.com
 toPorts:
 - ports:
 - port: "6379"
 protocol: TCP
 endpointSelector:
 matchLabels:
 app.kubernetes.io/name: mastodon
---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
 name: mastodon-ingress-cloudflare
 namespace: mastodon
spec:
 endpointSelector:
 matchLabels:
 app.kubernetes.io/name: mastodon
 ingress:
 - fromEndpoints:
 - matchLabels:
 app.kubernetes.io/name: cloudflare-tunnel
 io.kubernetes.pod.namespace: cloudflare-tunnel
 toPorts:
 - ports:
 - port: "3000"
 protocol: TCP
 - port: "4000"
 protocol: TCP
 - port: "8080"
 protocol: TCP

Cloudflare Tunnel - Troubleshooting - Cannot determine default origin certificate path

Mon, 01 Jan 0001 00:00:00 +0000

Cloudflare Tunnel - Troubleshooting - Cannot determine default origin certificate path Link to heading

Problem Link to heading

I installed a Cloudflare Tunnel on my Kubernetes cluster using the Helm Chart and I used the token given by the Cloudflare Zero Trust tunnel creation. When the tunnel starts up, it crashes with the following error:

2025-06-09T05:39:13Z ERR Cannot determine default origin certificate path. No file cert.pem in [~/.cloudflared ~/.cloudflare-warp ~/cloudflare-warp /etc/cloudflared /usr/local/etc/cloudflared]. You need to specify the origin certificate path by specifying the origincert option in the configuration file, or set TUNNEL_ORIGIN_CERT environment variable originCertPath=

error parsing tunnel ID: Error locating origin cert: client didn't specify origincert path

Analysis Link to heading

I found online related to the above error mostly discussions about deployment on R-pi or other OSes but nothing on Kubernetes. I started to compare the tunnel configuration I was given with the one running on my cluster and although both are JSON based, the JSON keys are different

Cloudflare Tunnel for Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

Cloudflare Tunnel for Kubernetes Link to heading

Problem Link to heading

Hosting services on Kubernetes is amazing but running Kubernetes inside a home network can be quite risky as it's incredibly difficult to ensure the environment is secure, monitored for intruders and backed up --- It just lacks the backing of a company/team behind it. With that in mind, how is it possible to open websites to the internet without opening ports on the router?

Disaster Recovery: Talos etcd Corruption & Kyverno CNI Deadlock

Mon, 01 Jan 0001 00:00:00 +0000

Incident Summary: Talos Control Plane & CNI Deadlock Link to heading

Overview Link to heading

During a physical disk migration for the TrueNAS VM (running on the Proxmox bare-metal host), severe I/O latency spikes caused a cascading failure across the Kubernetes production cluster. The incident unfolded in two distinct, critical phases: a complete loss of the Talos etcd control plane, followed by a severe CNI routing deadlock caused by admission webhooks.

enable_audit.sh

Mon, 01 Jan 0001 00:00:00 +0000

#!/usr/bin/env bash
set -euo pipefail

NAMESPACE="mastodon"
SELECTOR="app.kubernetes.io/name=mastodon"

echo "🔍 Finding Mastodon pods and enabling Cilium Policy Audit Mode..."

# 1. Loop through pod names directly
for POD_NAME in $(kubectl get pods -n "$NAMESPACE" -l "$SELECTOR" -o jsonpath='{.items[*].metadata.name}'); do

 # 2. Get the NODE NAME from the Pod itself (Reliable)
 NODE_NAME=$(kubectl get pod -n "$NAMESPACE" "$POD_NAME" -o jsonpath='{.spec.nodeName}')

 # 3. Get the ENDPOINT ID from the CiliumEndpoint (Same name as pod)
 EP_ID=$(kubectl get cep -n "$NAMESPACE" "$POD_NAME" -o jsonpath='{.status.id}')

 echo "---------------------------------------------------"
 echo "target: $POD_NAME"
 echo " node: $NODE_NAME"
 echo " id: $EP_ID"

 # 4. Find the Cilium Agent Pod running on that specific node
 CILIUM_POD=$(kubectl get pod -n kube-system -l k8s-app=cilium --field-selector spec.nodeName="$NODE_NAME" -o jsonpath='{.items[0].metadata.name}')

 if [[ -z "$CILIUM_POD" ]]; then
 echo "❌ ERROR: Could not find Cilium agent on node '$NODE_NAME'. Skipping..."
 continue
 fi

 # 5. Exec into the agent and toggle Audit Mode
 echo " exec: Enabling PolicyAuditMode on $CILIUM_POD..."
 kubectl exec -n kube-system "$CILIUM_POD" -c cilium-agent -- cilium-dbg endpoint config "$EP_ID" PolicyAuditMode=Enabled

 echo " ✅ Done."
done

echo "---------------------------------------------------"
echo "🎉 Audit Mode enabled. You can now apply the NetworkPolicy without dropping traffic."

Kubernetes - List Abnormally Terminated Pods Across All Namespaces

Mon, 01 Jan 0001 00:00:00 +0000

Problem Link to heading

I want to get a list of all pods that have terminated abnormally along with their restart reason and count.

Solution Link to heading

Run the following command:

kubectl get pods --all-namespaces -o custom-columns='NAME:.metadata.name,REASON:.status.containerStatuses[0].lastState.terminated.reason,RESTARTS:.status.containerStatuses[0].restartCount' | grep -v '<none>\|Running\|Completed'

Post-Mortem: Recovering a Corrupted Talos etcd Cluster

Mon, 01 Jan 0001 00:00:00 +0000

The Incident Link to heading

During a recent physical disk migration for a TrueNAS VM running on Proxmox, the resulting I/O latency spikes and storage disruption caused a massive ungraceful state change. This led to a complete loss of consensus in the 3-node Talos Linux control plane.

The Kubernetes API became completely unreachable, and the Virtual IP (VIP) dropped. Checking the control plane nodes revealed a relentless stream of errors:

Post-Mortem: The Admission Webhook CNI Deadlock

Mon, 01 Jan 0001 00:00:00 +0000

The Incident Link to heading

Following a successful bare-metal restore of a Talos etcd cluster, the control plane was healthy, but the entire pod network remained offline. Cilium (running in strict kube-proxy replacement mode) was completely failing to route traffic.

Critical infrastructure pods like talos-ccm, coredns, and hubble-relay were constantly crashing. Inspecting their logs revealed they were being firewalled by the host network when trying to reach the Kubernetes API or CoreDNS:

🧠 Analysis: Etcd Consensus in a Hybrid Optane/SATA Cluster

Mon, 01 Jan 0001 00:00:00 +0000

Architecture:

Node 1 (CP1): Intel Optane (Ultra-Low Latency ~10µs)
Node 2 (CP2): Intel Optane (Ultra-Low Latency ~10µs)
Node 3 (CP3): SATA SSD rpool (Variable Latency 10ms - 500ms+)

1. The Physics of Consensus (Speed) Link to heading

Etcd uses the Raft Consensus Algorithm. For a cluster of size $N=3$, the Quorum (votes required to commit data) is calculated as:

$\text{Quorum} = \lfloor \frac{3}{2} \rfloor + 1 = 2$

Kubernetes on Wael Nasreddine

💀 How I Accidentally DDoSed My Own Kubernetes Cluster

Introduction Link to heading

🔒Locking Down Mastodon: Cilium Network Policies & Audit Mode

The Goal: Defense in Depth Link to heading

🚨Alerting on Kyverno Policy Validation Failures with Loki

Context Link to heading

The Data Link to heading

Cilium Network Policies

Cloudflare Tunnel - Troubleshooting - Cannot determine default origin certificate path

Cloudflare Tunnel - Troubleshooting - Cannot determine default origin certificate path Link to heading

Problem Link to heading

Analysis Link to heading

Cloudflare Tunnel for Kubernetes

Cloudflare Tunnel for Kubernetes Link to heading

Problem Link to heading

Disaster Recovery: Talos etcd Corruption & Kyverno CNI Deadlock

Incident Summary: Talos Control Plane & CNI Deadlock Link to heading

Overview Link to heading

enable_audit.sh

Kubernetes - List Abnormally Terminated Pods Across All Namespaces

Problem Link to heading

Solution Link to heading

Related Link to heading

Post-Mortem: Recovering a Corrupted Talos etcd Cluster

The Incident Link to heading

Post-Mortem: The Admission Webhook CNI Deadlock

The Incident Link to heading

🧠 Analysis: Etcd Consensus in a Hybrid Optane/SATA Cluster

1. The Physics of Consensus (Speed) Link to heading