Kubernetes Networking Deep Dive

Kubernetes networking is one of the most powerful — and frequently misunderstood — layers of the platform. At its core, Kube provides a flat, scalable network where every pod can reach every other pod without NAT, and where network interfaces are assigned independently of the underlying host. In practice, achieving that guarantee requires careful interaction between the container runtime, CNI plugins, services, and the kernel. This article walks through the architecture, common pitfalls, and production-ready patterns you need to know.

The Kubernetes Networking Model

Kubernetes defines four networking responsibilities that every cluster must satisfy:

Container-to-container networking within the same pod via localhost.
Pod-to-pod networking across the cluster without Network Address Translation (NAT).
Pod-to-Service communication using stable virtual IPs.
External-to-Service access through ingress controllers or LoadBalancer-type services.

The design assumes a flat network fabric: each pod receives a unique IP address from a pod CIDR, and kube-proxy (or an equivalent implementation) ensures routing across node boundaries.

Pod IP Allocation

Every pod is assigned exactly one IP address at creation time. This IP persists for the pod's lifetime, even if containers restart. The IP comes from a pool defined by the CNI plugin — typically a subnet delegated from the node CIDR.

apiVersion: v1
kind: Pod
metadata:
  name: frontend
spec:
  containers:
  - name: web
    image: nginx
    ports:
    - containerPort: 80

In this example, frontend might receive 10.244.1.5. Other pods address it directly at that IP.

Container Network Interface (CNI) Plugins

CNI is the de facto standard for configuring pod networking in Kubernetes. When a pod is created, the kubelet invokes the configured CNI plugin, which bridges or routes the pod's veth pair into the cluster network.

Popular CNI Implementations

Plugin	Architecture	Notable Features
Calico	BGP-based routing	Policy enforcement, eBPF dataplane
Flannel	VXLAN overlay	Simple, good for small clusters
Cilium	eBPF-based	Network policies at L7, service mesh integration
Kindnet	Custom kinsmod	Used by kind, minimal

Installing a CNI Plugin

Most providers ship their manifests via Git. For Calico on a self-managed cluster:

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/tigera-operator.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/custom-resources.yaml

Verify that pods are receiving IPs:

kubectl get pods -A -o wide | head -5

Services and kube-proxy

A Service is an abstract way to expose an application running as a set of pods. The kube-proxy component maintains network rules on each node, forwarding traffic from the Service IP to the appropriate backend pods.

Service Types

ClusterIP — Default; exposes the Service on an internal cluster IP.
NodePort — Exposes the Service on each node's IP at a static port.
LoadBalancer — Provisions an external load balancer (cloud-dependent).
ExternalName — Maps a Service to a DNS name via CNAME.

How kube-proxy Works

kube-proxy有三种主要实现模式:

# Check current mode
kubectl get configmap -n kube-system kube-proxy -o yaml | grep mode

Mode	Mechanism	Latency	Scalability
`userspace`	iptables → userspace proxy	Higher	Poor (>100 nodes)
`iptables`	Netfilter rules	Moderate	Good (~500 nodes)
`ipvs`	Netfilter + kernel balancing	Lowest	Best (>500 nodes)

Modern clusters default to ipvs when available. Switching requires updating the kube-proxy ConfigMap:

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
ipvs:
  strictARP: true

Best Practice: Always enable strictARP: true with Cilium or any eBPF-based CNI. It prevents ARP cache poisoning and ensures consistent source-routing for packet replies.

Network Policies

Network Policies allow you to restrict traffic between pods using label selectors. By default, all pods can communicate freely — policies opt you into a deny-by-default model.

Deny All Ingress Traffic

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
spec:
  podSelector: {}
  policyTypes:
  - Ingress

This policy blocks all inbound traffic to every pod in the namespace.

Allow Only Specific Traffic

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

Warning: Network Policies are namespace-scoped. A pod in namespace prod cannot be targeted by a policy defined in namespace dev.

Ingress Controllers

Ingress resources provide HTTP/HTTPS routing rules for external traffic. Unlike Services, Ingress requires a dedicated controller that watches the API and configures a reverse proxy (e.g., NGINX, Traefik, Envoy).

Basic Ingress Configuration

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-service
            port:
              number: 80

TLS Termination

spec:
  tls:
  - hosts:
    - app.example.com
    secretName: tls-secret

Best Practice: Store TLS certificates in Secrets and reference them in Ingress resources. Never hardcode certificate data in manifests.

Common Networking Issues and Debugging

Diagnosing Pod Connectivity

# Test DNS resolution
kubectl run test -it --rm --image=busybox -- wget -qO- http://kubernetes.default.svc

# Verify CNI is active on a node
sudo crictl pods | wc -l
sudo ip netns exec <pod-namespace> ip addr show

# Check kube-proxy rules
sudo iptables-save | grep -i kube

Service Not Receiving Traffic

Confirm endpoints exist:

   kubectl get endpoints <service-name> -n <namespace>

Check if the Service selector matches pod labels.
Verify kube-proxy is running and using the correct mode.

CNI Plugin Failing After Node Restart

CNI plugins configure networking at the node level. If a node restarts without restoring the CNI state, new pods will fail to receive IPs. Recovery steps:

# Restart the CNI daemonset
kubectl rollout restart daemonset calico-node -n calico-system

# Verify pod CIDR assignment
kubectl describe node <node-name> | grep -A5 CIDR

Advanced Patterns

HostNetwork Pods

Setting hostNetwork: true binds the pod to the node's network namespace. This bypasses CNI entirely and is useful for network monitoring agents or low-latency applications.

spec:
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet

Caution: HostNetwork pods share the node's IP and can cause port conflicts. They also bypass Network Policies.

Multus and Multiple Network Interfaces

Multus CNI enables pods to attach to multiple networks simultaneously — for example, a primary Kubernetes network plus a secondary SR-IOV or DPDK interface.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-conf
spec:
  config: '{
    "cniVersion": "0.3.1",
    "type": "macvlan",
    "master": "eth0",
    "mode": "bridge"
  }'

Then reference it in a pod annotation:

annotations:
  k8s.v1.cni.cncf.io/networks: macvlan-conf

Observability and Monitoring

Metrics Server

The Metrics Server exposes resource metrics (CPU, memory) and is required for HPA. Install it separately:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Network Monitoring Tools

Tool	Purpose
tcpdump	Packet capture on a pod
curl	HTTP connectivity testing
nslookup	DNS debugging
Cilium Hubble	End-to-end observability with flow logs

# Capture traffic from a specific pod
kubectl exec -n default my-pod -- tcpdump -i eth0 -c 10 port 80

Conclusion

Kubernetes networking operates at the intersection of Linux networking primitives, cloud infrastructure, and application topology. Mastering it requires understanding how CNI plugins allocate IPs, how kube-proxy routes traffic, and how Network Policies enforce security boundaries. Start with the basics — a healthy CNI, working Services, and a simple Ingress controller — then layer on advanced patterns like multus, Cilium eBPF datapath, and mTLS-only traffic as your requirements grow.

When debugging, always start at the edges: verify the pod has an IP, confirm the Service has endpoints, and trace the request through the CNI plugin. Most issues resolve to one of these three layers.

Tech Master Hub