Architecture
 |
| Nginx ingress controller with Azure Kubernetes Service |
Overview
This blog is part of a series that's inspired by one of my recent client engagements. As I mentioned in my previous blog, Application gateway as ingress controller has direct access to all kubernetes pods and performs the layer 7 functions. In contrast, in this next blog I cover the "in-cluster" ingress controller with nginx and AKS which performs the layer 7 functions. There is a great blog article here that explains the differences. In-cluster ingress controller creates the kubernetes services resource and the Azure Layer 4 load balancer as shown in the architecture.. In comparison to the Azure application gateway, the traffic flows an extra hop.Test Drive
Fanout
http://akscolors.penguintrails.com (default web site)http://akscolors.penguintrails.com/cyan
http://akscolors.penguintrails.com/magenta
http://akscolors.penguintrails.com/yellow
Virtual Host
http://cyan.penguintrails.com
http://magenta.penguintrails.com
http://yellow.penguintrails.com
http://akscolors.eastus.cloudapp.azure.com (default)
Github
Pre-requisites:
Link to my github repo:
https://github.com/nehalineogi/aks-nginx-ingress
Git clone my repository and off you go...
git clone https://github.com/nehalineogi/aks-nginx-ingress.git
# create ingress controller
# Add the official stable repository
helm repo add stable https://kubernetes-charts.storage.googleapis.com/
#
# create ingress controlller
# Use Helm to deploy an NGINX ingress controller
#
helm install nginx-ingress stable/nginx-ingress \
--namespace ingress-ns \
--set controller.replicaCount=3 \
--set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux
https://github.com/nehalineogi/aks-nginx-ingress
Git clone my repository and off you go...
git clone https://github.com/nehalineogi/aks-nginx-ingress.git
# create ingress controller
#
# Create Namespace
kubectl create namespace ingress-ns# Add the official stable repository
helm repo add stable https://kubernetes-charts.storage.googleapis.com/
#
# create ingress controlller
# Use Helm to deploy an NGINX ingress controller
#
helm install nginx-ingress stable/nginx-ingress \
--namespace ingress-ns \
--set controller.replicaCount=3 \
--set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux
#
# Deployments
#
kubectl apply -f red-deployment.yaml
kubectl apply -f green-deployment.yaml
kubectl apply -f blue-deployment.yaml
kubectl apply -f white-deployment.yaml
#
# Ingress
#fanout
kubectl apply -f colors-fanout.yaml
#virtual host
#virtual host
kubectl apply -f colors-virtual-host.yaml
# Cleanup:
kubectl delete namespace ingress-ns
kubectl delete clusterrole nginx-ingress
kubectl delete clusterrolebinding nginx-ingress
Concepts - Ingress Deployments
Some ingress concepts here
Simple fanout
A fanout configuration routes traffic from a single IP address to more than one Service, based on the HTTP URI being requested. An Ingress allows you to keep the number of load balancers down to a minimum. For example, a setup like:
foo.bar.com -> 178.91.123.132 -> / foo service1:4200
/ bar service2:8080
Name based virtual hosting
Name-based virtual hosts support routing HTTP traffic to multiple host names at the same IP address.
foo.bar.com --| |-> foo.bar.com service1:80
| 178.91.123.132 |
bar.foo.com --| |-> bar.foo.com service2:80
Simple fanout
A fanout configuration routes traffic from a single IP address to more than one Service, based on the HTTP URI being requested. An Ingress allows you to keep the number of load balancers down to a minimum. For example, a setup like:
foo.bar.com -> 178.91.123.132 -> / foo service1:4200
/ bar service2:8080
Name based virtual hosting
Name-based virtual hosts support routing HTTP traffic to multiple host names at the same IP address.
foo.bar.com --| |-> foo.bar.com service1:80
| 178.91.123.132 |
bar.foo.com --| |-> bar.foo.com service2:80
AKS Side
kubectl get nodes -o wideNAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
aks-nodepool1-96863235-vmss000000 Ready agent 21d v1.15.10 172.16.240.4 <none> Ubuntu 16.04.6 LTS 4.15.0-1077-azure docker://3.0.10+azure
aks-nodepool1-96863235-vmss000001 Ready agent 21d v1.15.10 172.16.240.15 <none> Ubuntu 16.04.6 LTS 4.15.0-1071-azure docker://3.0.10+azure
aks-nodepool1-96863235-vmss000002 Ready agent 21d v1.15.10 172.16.240.26 <none> Ubuntu 16.04.6 LTS 4.15.0-1071-azure docker://3.0.10+azure
kubectl get deploy,pods,services,ingress -o wide -n ingress-ns
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.extensions/nginx-ingress-controller 3/3 3 3 4m52s nginx-ingress-controller quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0 app=nginx-ingress,release=nginx-ingress
deployment.extensions/nginx-ingress-default-backend 1/1 1 1 4m52s nginx-ingress-default-backend k8s.gcr.io/defaultbackend-amd64:1.5 app=nginx-ingress,release=nginx-ingress
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/nginx-ingress-controller-f559556f9-cbmzq 1/1 Running 0 4m52s 172.16.240.14 aks-nodepool1-96863235-vmss000000 <none> <none>
pod/nginx-ingress-controller-f559556f9-lhbfz 1/1 Running 0 4m52s 172.16.240.23 aks-nodepool1-96863235-vmss000001 <none> <none>
pod/nginx-ingress-controller-f559556f9-tqgkl 1/1 Running 0 4m52s 172.16.240.29 aks-nodepool1-96863235-vmss000002 <none> <none>
pod/nginx-ingress-default-backend-76b8f499cb-sw8pv 1/1 Running 0 4m52s 172.16.240.10 aks-nodepool1-96863235-vmss000000 <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/nginx-ingress-controller LoadBalancer 10.20.8.97 52.170.163.131 80:31982/TCP,443:30146/TCP 4m52s app.kubernetes.io/component=controller,app=nginx-ingress,release=nginx-ingress
service/nginx-ingress-default-backend ClusterIP 10.20.137.158 <none> 80/TCP 4m52s app.kubernetes.io/component=default-backend,app=nginx-ingress,release=nginx-ingress
kubectl describe service nginx-ingress-controller -n ingress-ns
Name: nginx-ingress-controller
Namespace: ingress-ns
Labels: app=nginx-ingress
chart=nginx-ingress-1.36.2
component=controller
heritage=Helm
release=nginx-ingress
Annotations: <none>
Selector: app.kubernetes.io/component=controller,app=nginx-ingress,release=nginx-ingress
Type: LoadBalancer
IP: 10.20.8.97
LoadBalancer Ingress: 52.170.163.131
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 31982/TCP
Endpoints: 172.16.240.14:80,172.16.240.23:80,172.16.240.29:80
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 30146/TCP
Endpoints: 172.16.240.14:443,172.16.240.23:443,172.16.240.29:443
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadBalancer 7m service-controller Ensuring load balancer
Normal EnsuredLoadBalancer 6m37s service-controller Ensured load balancer
kubectl describe service nginx-ingress-default-backend -n ingress-ns
Name: nginx-ingress-default-backend
Namespace: ingress-ns
Labels: app=nginx-ingress
chart=nginx-ingress-1.36.2
component=default-backend
heritage=Helm
release=nginx-ingress
Annotations: <none>
Selector: app.kubernetes.io/component=default-backend,app=nginx-ingress,release=nginx-ingress
Type: ClusterIP
IP: 10.20.137.158
Port: http 80/TCP
TargetPort: http/TCP
Endpoints: 172.16.240.10:8080
Session Affinity: None
Events: <none>
Azure Side (Load Balancer)
 |
| backend Pool with VMSS Nodes |
 |
| Load Balancer Public IPs |
 |
| Node Port for the Exposed nginx controller service |
 |
| Rule showing the HTTP heath probe on the NodePort |
 |
| LB Rules for tcp/80 |
Deployments:
kubectl get deploy,pods,services,ingress -o wide -n colors-ns
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.extensions/cyan-deployment 2/2 2 2 57s cyan-app hashicorp/http-echo app=cyan
deployment.extensions/magenta-deployment 2/2 2 2 53s magenta-app hashicorp/http-echo app=magenta
deployment.extensions/yellow-deployment 2/2 2 2 49s yellow-app hashicorp/http-echo app=yellow
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/cyan-deployment-85c5fd4b7-49n25 1/1 Running 0 56s 172.16.240.13 aks-nodepool1-96863235-vmss000000 <none> <none>
pod/cyan-deployment-85c5fd4b7-s2gx4 1/1 Running 0 56s 172.16.240.30 aks-nodepool1-96863235-vmss000002 <none> <none>
pod/magenta-deployment-64fcd58c6f-9m4n8 1/1 Running 0 52s 172.16.240.32 aks-nodepool1-96863235-vmss000002 <none> <none>
pod/magenta-deployment-64fcd58c6f-ljc2n 1/1 Running 0 52s 172.16.240.5 aks-nodepool1-96863235-vmss000000 <none> <none>
pod/yellow-deployment-5ff564c666-8s9b6 1/1 Running 0 49s 172.16.240.12 aks-nodepool1-96863235-vmss000000 <none> <none>
pod/yellow-deployment-5ff564c666-mswkf 1/1 Running 0 49s 172.16.240.6 aks-nodepool1-96863235-vmss000000 <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/cyan-service ClusterIP 10.20.140.182 <none> 8080/TCP 56s app=cyan
service/magenta-service ClusterIP 10.20.101.186 <none> 8080/TCP 52s app=magenta
service/yellow-service ClusterIP 10.20.46.169 <none> 8080/TCP 49s app=yellow
Validations:
I've created a busybox container to run all the tests. Let's validate connectivity to red pods and cluster serivce.
/ # ifconfig
eth0 Link encap:Ethernet HWaddr 16:2A:D2:E5:D0:13
inet addr:172.16.240.20 Bcast:0.0.0.0 Mask:255.255.255.0
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:2273586 errors:0 dropped:0 overruns:0 frame:0
TX packets:105 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:95504870 (91.0 MiB) TX bytes:7271 (7.1 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
/ # wget -qO- 172.16.240.13
wget: can't connect to remote host (172.16.240.13): Connection refused
/ # wget -qO- 172.16.240.13:8080
cyan
/ # wget -qO- 172.16.240.30:8080
cyan
/ # wget -qO- 10.20.140.182:8080
cyan
Ingress:
Fanout:kubectl apply -f colors-fanout.yaml
kubectl describe ingress colors-fanout-ingress -n colors-ns
Name: colors-fanout-ingress
Namespace: colors-ns
Address: 172.16.240.15,172.16.240.26,172.16.240.4
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
akscolors.penguintrails.com
/cyan cyan-service:8080 (172.16.240.13:8080,172.16.240.30:8080)
/magenta magenta-service:8080 (172.16.240.32:8080,172.16.240.5:8080)
/yellow yellow-service:8080 (172.16.240.12:8080,172.16.240.6:8080)
cyan-service:8080 (172.16.240.13:8080,172.16.240.30:8080)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"colors-fanout-ingress","namespace":"colors-ns"},"spec":{"rules":[{"host":"akscolors.penguintrails.com","http":{"paths":[{"backend":{"serviceName":"cyan-service","servicePort":8080},"path":"/cyan"},{"backend":{"serviceName":"magenta-service","servicePort":8080},"path":"/magenta"},{"backend":{"serviceName":"yellow-service","servicePort":8080},"path":"/yellow"},{"backend":{"serviceName":"cyan-service","servicePort":8080},"path":null}]}}]}}
Validations:
kubectl exec -it pingtest-b4b6f8cf-5xk7v sh
/ # ifconfig
eth0 Link encap:Ethernet HWaddr 16:2A:D2:E5:D0:13
inet addr:172.16.240.20 Bcast:0.0.0.0 Mask:255.255.255.0
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:2275879 errors:0 dropped:0 overruns:0 frame:0
TX packets:191 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:95605042 (91.1 MiB) TX bytes:13569 (13.2 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
/ # wget -qO- --header="Host:akscolors.penguintrails.com" http://10.20.8.97/cyan
cyan
/ # wget -qO- --header="Host:akscolors.penguintrails.com" http://10.20.8.97/magenta
magenta
/ # wget -qO- --header="Host:akscolors.penguintrails.com" http://10.20.8.97/yellow
yellow
/ # wget -qO- --header="Host:akscolors.penguintrails.com" http://10.20.8.97/
cyan
/ #
Virtual Host
kubectl apply -f colors-virtual-host.yaml
ingress.extensions/colors-virtual-host-ingress created
kubectl get ingress -o wide -n colors-ns
NAME HOSTS ADDRESS PORTS AGE
colors-fanout-ingress akscolors.penguintrails.com 172.16.240.15,172.16.240.26,172.16.240.4 80 19m
colors-virtual-host-ingress cyan.penguintrails.com,magenta.penguintrails.com,yellow.penguintrails.com + 1 more... 172.16.240.15,172.16.240.26,172.16.240.4 80 74s
kubectl describe ingress colors-virtual-host-ingress -n colors-ns
Name: colors-virtual-host-ingress
Namespace: colors-ns
Address: 172.16.240.15,172.16.240.26,172.16.240.4
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
cyan.penguintrails.com
cyan-service:8080 (172.16.240.13:8080,172.16.240.30:8080)
magenta.penguintrails.com
magenta-service:8080 (172.16.240.32:8080,172.16.240.5:8080)
yellow.penguintrails.com
yellow-service:8080 (172.16.240.12:8080,172.16.240.6:8080)
*
cyan-service:8080 (172.16.240.13:8080,172.16.240.30:8080)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"colors-virtual-host-ingress","namespace":"colors-ns"},"spec":{"rules":[{"host":"cyan.penguintrails.com","http":{"paths":[{"backend":{"serviceName":"cyan-service","servicePort":8080}}]}},{"host":"magenta.penguintrails.com","http":{"paths":[{"backend":{"serviceName":"magenta-service","servicePort":8080}}]}},{"host":"yellow.penguintrails.com","http":{"paths":[{"backend":{"serviceName":"yellow-service","servicePort":8080}}]}},{"http":{"paths":[{"backend":{"serviceName":"cyan-service","servicePort":8080}}]}}]}}
kubernetes.io/ingress.class: nginx
Events:
Validations:
kubectl exec -it pingtest-b4b6f8cf-5xk7v sh
/ # ifconfig
eth0 Link encap:Ethernet HWaddr 16:2A:D2:E5:D0:13
inet addr:172.16.240.20 Bcast:0.0.0.0 Mask:255.255.255.0
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:2278466 errors:0 dropped:0 overruns:0 frame:0
TX packets:233 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:95716051 (91.2 MiB) TX bytes:16907 (16.5 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
/ # wget -qO- --header="Host:cyan.penguintrails.com" http://10.20.8.97
cyan
/ # wget -qO- --header="Host:magenta.penguintrails.com" http://10.20.8.97
magenta
/ # wget -qO- --header="Host:yellow.penguintrails.com" http://10.20.8.97
yellow
/ # wget -qO- http://10.20.8.97
cyan
/ #
Outbound from the PODS:
There is an outbound rule that gets created so all egress traffic from the pods get's SNAT'd to the load balancer Public IP address
kubectl exec -it pingtest-b4b6f8cf-5xk7v sh
/ # ifconfig
eth0 Link encap:Ethernet HWaddr 16:2A:D2:E5:D0:13
inet addr:172.16.240.20 Bcast:0.0.0.0 Mask:255.255.255.0
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:2217728 errors:0 dropped:0 overruns:0 frame:0
TX packets:36 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:93146060 (88.8 MiB) TX bytes:2347 (2.2 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
/ # wget -qO- icanhazip.com
52.152.246.245
1 comment:
Hello friends, how is the whole thing, and what you would like to say regarding this piece of writing, in my view its actually awesome for me.
Post a Comment