Tales of the Kubernetes Ingress Networking: Deployment Patterns for External Load Balancers

Traefik's Logo

How to Access the Slides?

QRCode to this presentation

How to Use the Slides?

  • Browse the slides: Use the arrows

    • Change chapter: Left/Right arrows

    • Next or previous slide: Top and bottom arrows

  • Overview of the slides: keyboard’s shortcut "o"

  • Speaker mode (and notes): keyboard’s shortcut "s"

Whoami

damien

Containous

https://containo.us

  • We Believe in Open Source

  • We Deliver Traefik, Traefik Enterprise Edition and Maesh

  • Commercial Support

  • 30 people distributed, 90% tech

Containous Logo

Once Upon A Time

There was Kubernetes cluster.

This Cluster Had Nodes and Pods

kubernetes nodes pods

But Pods Had Private IPs

How to route traffic to these pods? And between pods on different nodes?

kubernetes route pods

Services Came To The Rescue

Their goal: Expose Pods to allow incoming traffic

kubernetes services

Services are Load-Balancers

  • Services have 1-N Endpoints

  • EndPoints are determined by Kubernetes API

kubernetes services lb
One exception: Services of types ExternalName

Different Kinds of Services

for different communications use cases:

  • From inside: type "ClusterIP" (default).

  • From outside: types "NodePort" and "LoadBalancer".

Services: ClusterIP

Virtual IP, private to the cluster, cluster)-wide (e.g. works from any node to any other node)

kubernetes services type cluster ip

Services: NodePort

Uses public IPs and ports of the nodes, kind of "Routing grid"

kubernetes services type nodeport

Services: LoadBalancer

Same as NodePort,excepts it requires (and uses) an external Load Balancer.

kubernetes services type lb

Services are Not Enough

  • Context: Exposes externally a bunch of applications

  • Challenge: overhead of allocation for LBs. For each application:

    • One LB resource (either a machine or a dedicated appliance)

    • At least one public IP

    • DNS nightmare (think about the CNAMEs to create…​)

    • No centralization of certificates, logs, etc.

And Then Came the Ingress

Example with Traefik as Ingress Controller:

traefik kubernetes diagram

Notes About The Ingresses

Ingress Are Standard Kubernetes Applications

  • Deployed as Pods (Deployment or as DaemonSet)

  • Exposed with a Service:

    • You still need access from the outside

    • But only one service to deal with (ideally)

Ingress Have Services Too

kubernetes ingress impl

Why Should I Care?

  • Simplified Setup:

    • Single entrypoint, less configuration, better measures

    • Less resources used

    • Separation of concerns: differents algorithms for load balancing, etc.

Why Challenges Does It Make?

  • Designed for (simple) HTTP/HTTPS cases

    • TCP/UDP can be used, but are not first-class citizens

    • "Virtual Host First" centric

  • Feels like you must carefully select your (only) Ingress Controller

So What?

  • Kubernetes gives you freedom:

    • You can use multiple Ingress Controllers!

  • Kubernetes gives you choices:

    • So much deployment patterns that you can do almost anything

External Load Balancers

Did you Just Say "External"?

  • Outside the "Borders" of Kubernetes:

    • Depends on your "platform" (as in infrastructure/cloud)

  • Still Managed by Kubernetes (Automation)

    • Requires "plugins" (operators/modules) per Load Balancer provider

    • No API or no Kubernetes support: requires switching to NodePort

Tell Me Your Kubernetes Distribution

…​and I’ll tell you which LB to use…​

Cloud Managed Kubernetes

  • Cloud providers provides their own external LBs

    • Fully Automated Management with APIs

    • Great UX due to the integration: works out of the box

    • Benefits from cloud provider HA and performances

  • But:

    • You have to pay for this :)

    • Configuration is cloud-specific (using annotations)

    • Relies on LB implementation limits

Bare-Metal Kubernetes

Aka. "Run it on your boxes"

  • Best approach: Metal LB, a Load Balancer implementation for Kubernetes, hosted inside Kubernetes

    • Uses all Kubernetes primitives (HA, deployment, etc.)

    • Allows Layer 2 routing as well as BGP

    • But…​ still not considered production ready

  • Otherwise: external static (or legacy) LB

    • Requires switching to NodePort Service

Cloud "Semi-Managed" Kubernetes

  • Depends on the compute provider: cloud or bare-metal

  • You need a tool for mananaging clusters: kubeadm, kops, etc.

    • Most of these tools already manage LB if the provider does.

Source IP on the Kingdom of Kubernetes

Business Case: Source IP

As a business manager, I need my system to know the IP of the emitters of the requests to track usage, write access logs for legals reasons and limit traffic in some cases.

NAT/DNAT/SNAT

  • NAT stands for "Network Adress Translation"

    • IPv4 world: Routers "masquerades" IPs, to allow routing from different network

  • DNAT stands for "Destination NAT"

    • Masquerade of the destination IP with the internal pod IP

  • SNAT stands for "Source NAT"

    • Masquerade of the source IP with the router’s IP

NAT/DNAT/SNAT

nat

Preserve Source IP

  • Rule: We do NOT want SNAT to happen

  • Challenge: many intermediate components can interfere and SNAT the packets in our back!

Inside Kubernetes: kube-proxy

  • kube-proxy is a Kubernetes component, running on each worker node

  • Role: manage the virtual IPs used for Services

  • Challenge with Source IP: kube-proxy might SNAT requests

  • SNAT by kube-proxy depends on the Service:

    • Let’s do a tour of Services Types!

Source IP with Service ClusterIP

  • When kube-proxy is in "iptables" mode: no SNAT ✅

    • This is the default mode

    • No intermediate component

kubernetes source ip svc cip iptables

Source IP with Service NodePort (Default)

  • SNAT is done ❌ (routing to the node where pod is):

    • First node to node routing through nodes network

    • Then node to pod routing through pod network

kubernetes source ip svc nodeport

Source IP with Service NodePort (Local Endpoint)

  • No SNAT ✅ with externalTrafficPolicy set to Local

  • Downside: Dropped request if no pod on receiving node

kubernetes source ip svc nodeport local

Source IP with Service LoadBalancer (Default)

  • Default: SNAT is done ❌, same as NodePort

    • External Load Balancer can route to any node

    • If no local endpoint: Node to node routing with SNAT

Source IP with Service LoadBalancer (Local Endpoint)

  • However, No SNAT ✅ for load balancers implementing Local externalTrafficPolicy:

    • GKE/GCE LB, Amazon NLB, etc.

    • 🛠Nodes without local endpoints are removed from the LB by failing healthchecks

    • 👍🏽Pros: no dropped request from client view, but nodes always ready

    • 👎🏼Cons: relies on healthcheck timeouts

Alternatives When SNAT Happen

  • Sometimes, SNAT is mandadatory

    • External LB

    • Network Constraint

    • Ingress Controller in the middle

  • "Network is based on layers" - let’s use another layer:

    • If using HTTP, retrieve the Source IP from headers

    • If using TCP/UDP, use the "Proxy Protocol"

    • Or use distributed logging and tracing

HTTP Protocol Headers

  • X-Forwarded-From holds a comma-separated list of all the source IPs SNAT during all network hops.

    • ✅ if you have an External LoadBalancer or an Ingress Controller supporting this header.

    • ⚠️ Not standard (header starting with X-) so not all HTTP appliance might support it.

Proxy Protocol

  • Introduced by HAProxy

  • Happens at Layer 4 (Transport) for TCP/UDP

  • Goal: "chain proxies / reverse-proxies without losing the client information"

  • Supported by a lot of appliances in 2019: AWS ELB, Traefik, Apache, Nginx, Varnish, etc.

  • Use Case: when SNAT happen AND not way to use HTTP. H

Distributing Logging and Tracing

  • 🛠 Idea:

    • Collect the source IP as soon as possible in distributed logging

    • Use distributed tracing to track the request in the system

  • 👍🏽Pros: no more complex network setups, distributed logging and tracing stacks are already on your Kubernetes cluster (or will soon be)

  • 👎🏼Cons: relies on the distributed logging/tracing stacks

Demo Time!

Demo 1

  • Amazon EKS: Capturing Source IP with the local external Load Balancer traffic policy

Demo 2

  • Bare-Metal Kubernetes: Use Traefik for capturing Source IP on HTTP headers

Sources

Thank you!

@DamienDuportal

dduportal

QRCode to this presentation