Kubernetes Networking

Networking is a central part of Kubernetes, but it can be challenging to understand exactly how it is expected to work. There are 4 distinct networking problems to address:

1. Highly-coupled container-to-container communications
2. Pod-to-Pod communications
3. Pod-to-Service communications
4. External-to-Service communications

Kubernetes is all about sharing machines between applications. Typically, sharing machines requires ensuring that two applications do not try to use the same ports. Coordinating ports across multiple developers is very difficult to do at scale and exposes users to cluster-level issues outside of their control.

Dynamic port allocation brings a lot of complications to the system - every application has to take ports as flags, the API servers have to know how to insert dynamic port numbers into configuration blocks, services have to know how to find each other, etc. Rather than deal with this, Kubernetes takes a different approach.

Container networking and pods

Containers inside each pod share certain resources, although not all, so that they’re not fully isolated. Kubernetes achieves this by configuring Docker to have all containers of a pod share the same set of Linux namespaces instead of each container having its own set.

The following diagram from the excellent treatment on the subject of isolation layers in K8s clusters

Isolation provided by layer of Kubernetes

Container Networking from Scratch - Kristen Jacobs, Oracle, slides

Introduction to CNI, the Container Network Interface Project - Bryan Boreham & Dan Williams, slides

How to Write a CNI Plugin From Scratch - Eran Yanay, Twistlock

With respect to networking, containers within a pod behave as if they are on the same host. They can all reach each other’s ports on localhost. This offers simplicity (static ports known a priori), security (ports bound to localhost are visible within the pod but never outside it), and performance. This also reduces friction for applications moving from the world of uncontainerized apps on physical or virtual hosts. People running application stacks together on the same host have already figured out how to make ports not conflict and have arranged for clients to find them.

The approach does reduce isolation between containers within a pod — ports could conflict, and there can be no container-private ports, but these seem to be relatively minor issues with plausible future workarounds. Besides, the premise of pods is that containers within a pod share some resources (volumes, cpu, ram, etc.) and therefore expect and tolerate reduced isolation. Additionally, the user can control what containers belong to the same pod whereas, in general, they don’t control what pods land together on a host.

Inter-POD Networking

All pods in a Kubernetes cluster reside in a single flat, shared, network-address space (shown in figure below), which means every pod can access every other pod at the other pod’s IP address. No NAT (Network Address Translation) gateways exist between them. When two pods send network packets between each other, they’ll each see the actual IP address of the other as the source IP in the packet.

Flat network

Consequently, communication between pods is always simple. It doesn’t matter if two pods are scheduled onto a single or onto different worker nodes; in both cases the containers inside those pods can communicate with each other across the flat NAT-less network, much like computers on a local area network (LAN), regardless of the actual inter-node network topology. Like a computer on a LAN, each pod gets its own IP address and is accessible from all other pods through this network established specifically for pods. This is usually achieved through an additional software-defined network layered on top of the actual network. The discussion in the section container networking provided information on various backend plugins