Friday, May 29, 2020

Scalable layer-4 NVA Design in Azure

Architecture

In this blog post we will walk through a scalable active/active NVA (Network Virtual Appliance) design, take a closer look at three different use cases, the challenges and how to preserve flow symmetry for east-west and north-south traffic.
On-Prem to Azure - East-West traffic using Internal Load Balancer HA Ports

Summary

This blog post is inspired by one of my recent partner engagement. Last week I worked with a partner where we needed to design a scalable architecture with NVAs in Azure. When using NVAs in Azure, designing a scalable and resilient architecture is important because the platform level redundancy and scalability is only provided when using Azure native firewalls aka firewall PaaS service.  When using NVAs in Azure, there are certain design considerations and best practices to ensure flow symmetry.  Different firewall vendors have published their best practices. While this particular example is based on Cisco ASAv,  the focus of this blog is on key architectural components which stays the same for any NVA vendor.  There is a great blog article here that notes some of the key differences between NVAs and native firewalls.

Microsoft and NVA vendor documentation

There's great documentation from NVA vendors and Azure and I've provided the link to those here.

Microsoft Documentation on HA ports and NVA here
Palo Alto documentation link here
Cisco documentation link here


Scribble

Warning - geek alert!   I started to scribble this design and the flows got out of control. Let's untangle this and get a deeper understanding of two types of traffic flows

  1. East-West traffic : Preserve flow symmetry using  single NIC NVA and internal load balancer (ILB) with HA Ports 
  2. North-South traffic: Preserve flow symmetry with ELB-NVA-ILB aka "NVA load balancer sandwich design" using SNAT.


Joys of Software Defined Networking....Let's untangle


Cisco Side Configuration:

This is view of how the firewall interfaces are configured. Note: Outside interface is only used when using the north-south design.

nn-asav-01# sh ip add
System IP Addresses:
Interface                Name                   IP address      Subnet mask     Method
GigabitEthernet0/0       outside                172.16.250.4    255.255.255.0   DHCP
GigabitEthernet0/1       transit                172.16.3.4      255.255.255.0   DHCP
Management0/0            management             172.16.1.8      255.255.255.0   DHCP
nn-asav-01# sh run route
route outside 0.0.0.0 0.0.0.0 172.16.250.1 1
route transit 10.1.0.0 255.255.0.0 172.16.3.1 1
route transit 10.2.0.0 255.255.0.0 172.16.3.1 1
route transit 168.63.129.16 255.255.255.255 172.16.3.1 1
route transit 192.168.190.0 255.255.255.0 172.16.3.1 1

 Allow ssh from Load balancer probes
nn-asav-01# sh run ssh
ssh stricthostkeycheck
ssh timeout 60
ssh version 2
ssh key-exchange group dh-group14-sha256ssh 168.63.129.16 255.255.255.255 transit
ssh 0.0.0.0 0.0.0.0 management
ssh 168.63.129.16 255.255.255.255 outside
nn-asav-02# sh ip add
System IP Addresses:
Interface                Name                   IP address      Subnet mask     Method
GigabitEthernet0/0       outside                172.16.250.5    255.255.255.0   DHCP
GigabitEthernet0/1       transit                172.16.3.5      255.255.255.0   DHCP
Management0/0            management             172.16.1.9      255.255.255.0   DHCP

 nn-asav-02# sh run route
route outside 0.0.0.0 0.0.0.0 172.16.250.1 1
route transit 10.1.0.0 255.255.0.0 172.16.3.1 1
route transit 10.2.0.0 255.255.0.0 172.16.3.1 1
route transit 168.63.129.16 255.255.255.255 172.16.3.1 1
route transit 192.168.190.0 255.255.255.0 172.16.3.1 1

Understanding Flows and Key Design Points

This particular blog post is based on Active-Active NVA configuration. There's also Active/standby design which requires some automation effort to ensure fast failover. There is more documentation on the active/standby design here.

Common Design principles

  • Active-Active NVA Configuration
  • Fast failover and per-instance health probes
  • Scale-out to N active instances

East-west traffic:

  • This is a single NIC design with single transit interface
  • Use HA ports with Internal Load Balancer for east-west flow symmetry
  • Use UDRs with east-west traffic to steer the traffic via the NVA
  • When using UDRs from spoke to on-prem traffic make sure that  "route-propagation" is disabled on the spoke UDRs. This setting is available under UDR configuration.

North-South traffic:

  • This is a two NIC design with Outside and transit interface
  • Typical design is as an internet edge firewall
  • Use SNAT with ELB + Firewall + ILB aka  firewall load balancer sandwich
  • Associate Public IP on the egress interface of the NVA for outbound NAT
  • NSGs should be applied on transit and outside interfaces
Be sure to checkout the limitations here

East-West Traffic

This is a single NIC design with single transit interface. You can still have a management interface on the NVA. Internal load balancer is configured to Use HA ports  for east-west flow symmetry.  Use UDRs  to steer the traffic via the NVA. There are two different use cases that we will review

1. For spoke to spoke traffic the UDRs are configured on spoke subnets. 
2. For on-prem to Azure traffic the UDR is configured on the gateway subnet.

From Azure Documentation here
A high availability (HA) ports load-balancing rule is a variant of a load-balancing rule, configured on an internal Standard Load Balancer. You can simplify the use of a load balancer by providing a single rule to load-balance all TCP and UDP flows that arrive on all ports of an internal Standard Load Balancer. The load-balancing decision is made per flow. This action is based on the following five-tuple connection: source IP address, source port, destination IP address, destination port, and protocol

Spoke-to-Spoke traffic:
Green path: Spoke1 (Source: VM)-> UDR -> ILB (.250) -> NVA1->Spoke2(Dest: VM)
Return Traffic (Orange path): Spoke2 (VM) ->UDR -> ILB (.250) -> Same NVA1 -> Spoke2 (VM1)

Load balancing decision made on the basis of the 5-tuple hash which stays the same.
Example:

SRC-IP: 10.1.1.4 DST-IP:10.2.2.4 SRC-PORT:45678  DST-PORT-3389

SRC-IP: 10.2.2.4 DST-IP: 10.1.1.4 SRC-PORT:3389 DST-PORT: 45678

Spoke-to-spoke - East-West traffic using Internal Load Balancer HA Ports

Azure side Configuration:

Spoke1-route-table

Spoke2-route-table



Azure Load Balancer configuration:

 Note: LB probe is configure for tcp/22



On-prem to Azure traffic:

Blue path: On-Prem (Source: VM)->ExpressRoute/VPN -> Gateway Subnet-> UDR -> ILB (.250) -> NVA1->Spoke1(Dest: VM)
Return Traffic (Magenta path): Spoke1 (VM) ->UDR -> ILB (.250) -> Same NVA1 -> ExpressRoute or VPN -> On-Prem (Dest VM)

Load balancing decision made on the basis of the 5-tuple hash which stays the same.

Example:
SRC-IP: 192.168.190.190 DST-IP:10.1.1.4 SRC-PORT:45678  DST-PORT-22
Return Traffic SRC-IP: 10.1.1.4 DST-IP: 192.168.190.190 SRC-PORT:22 DST-PORT: 45678

For on-prem to Azure traffic the UDR is configured on the gateway subnet as well as the spoke subnets to route via ILB ( Internal load balancer). Be sure to disable the "route propagation" checkbox on the Spoke UDR to steer the traffic via NVA.

On-Prem to Azure - East-West traffic using Internal Load Balancer HA Ports
Gateway-Subnet UDR to steer on-prem traffic via ILB



North South Traffic

This is a two NIC design with Outside and transit interface. Use SNAT with ELB + Firewall + ILB aka  firewall load balancer sandwich. Associate Public IP on the egress interface of the NVA for outbound NAT
Internet to Azure - North-South traffic using firewall load Balancer sandwich design using SNAT
Azure side configuration:

Option#1: (SNAT on NVA egress interface with public IP assigned to the NVA)
Map the outside interface on the firewall to a public IP in Azure

Map Public IP to outside interface on ASAv-01

Map Public IP to outside interface on ASAv-02



Option #2:   SNAT on firewall egress interface and ELB SNAT

Azure External Loadbalancer outbound SNAT rule

External load balancer frontend configuration

ELB outbound SNAT rule

Testing and Validations


LB Probes on tcp/22

nn-asav-01# sh capture transit  | i 168.63
  49: 14:29:54.171667       168.63.129.16.65329 > 172.16.3.4.22: R 2388247501:2388247501(0) ack 1086528740 win 0
  50: 14:29:54.172125       168.63.129.16.65350 > 172.16.3.4.22: SWE 2553081639:2553081639(0) win 8192 <mss 1440,nop,wscale 8,nop,nop,sackOK>
  51: 14:29:54.172171       172.16.3.4.22 > 168.63.129.16.65350: S 2684413220:2684413220(0) ack 2553081640 win 8192 <mss 1380>
  52: 14:29:54.172476       168.63.129.16.65350 > 172.16.3.4.22: . ack 2684413221 win 64860
  53: 14:29:54.173040       172.16.3.4.22 > 168.63.129.16.65350: P 2684413221:2684413240(19) ack 2553081640 win 32768
  56: 14:29:54.234072       168.63.129.16.65350 > 172.16.3.4.22: . ack 2684413240 win 64841
 101: 14:29:57.233981       172.16.3.4.33344 > 168.63.129.16.80: S 1483242750:1483242750(0) win 32768 <mss 1460,nop,nop,timestamp 600641695 0>
 129: 14:30:00.171744       168.63.129.16.65350 > 172.16.3.4.22: R 2553081640:2553081640(0) ack 2684413240 win 0
 130: 14:30:00.172430       168.63.129.16.65403 > 172.16.3.4.22: SWE 723143324:723143324(0) win 8192 <mss 1440,nop,wscale 8,nop,nop,sackOK>

East West Traffic


Spoke-to-spoke
Notice two packet for request and reply, entering and leaving the same transit interface

nn-asav-01# sh capture transit | i icmp
  21: 13:45:12.085215       10.1.1.4 > 10.2.2.4 icmp: echo request
  22: 13:45:12.085231       10.1.1.4 > 10.2.2.4 icmp: echo request
  23: 13:45:12.087031       10.2.2.4 > 10.1.1.4 icmp: echo reply
  24: 13:45:12.087046       10.2.2.4 > 10.1.1.4 icmp: echo reply
  62: 13:45:13.086909       10.1.1.4 > 10.2.2.4 icmp: echo request
  63: 13:45:13.086924       10.1.1.4 > 10.2.2.4 icmp: echo request
  64: 13:45:13.088893       10.2.2.4 > 10.1.1.4 icmp: echo reply
  65: 13:45:13.088908       10.2.2.4 > 10.1.1.4 icmp: echo reply
 110: 13:45:14.088999       10.1.1.4 > 10.2.2.4 icmp: echo request
 111: 13:45:14.089106       10.1.1.4 > 10.2.2.4 icmp: echo request
 112: 13:45:14.091242       10.2.2.4 > 10.1.1.4 icmp: echo reply
 113: 13:45:14.091258       10.2.2.4 > 10.1.1.4 icmp: echo reply
 154: 13:45:15.090357       10.1.1.4 > 10.2.2.4 icmp: echo request
 155: 13:45:15.090418       10.1.1.4 > 10.2.2.4 icmp: echo request
 156: 13:45:15.096583       10.2.2.4 > 10.1.1.4 icmp: echo reply
 157: 13:45:15.096598       10.2.2.4 > 10.1.1.4 icmp: echo reply


On-Prem to Spoke

Notice two packet for request and reply, entering and leaving the same transit interface

nn-asav-01# sh capture transit | i icmp
  37: 13:42:23.172903       192.168.190.190 > 10.1.1.4 icmp: echo request
  38: 13:42:23.173010       192.168.190.190 > 10.1.1.4 icmp: echo request
  39: 13:42:23.174780       10.1.1.4 > 192.168.190.190 icmp: echo reply
  40: 13:42:23.174795       10.1.1.4 > 192.168.190.190 icmp: echo reply
  75: 13:42:24.170569       192.168.190.190 > 10.1.1.4 icmp: echo request
  76: 13:42:24.170675       192.168.190.190 > 10.1.1.4 icmp: echo request
  77: 13:42:24.172674       10.1.1.4 > 192.168.190.190 icmp: echo reply
  78: 13:42:24.172690       10.1.1.4 > 192.168.190.190 icmp: echo reply
 116: 13:42:25.171118       192.168.190.190 > 10.1.1.4 icmp: echo request
 117: 13:42:25.171240       192.168.190.190 > 10.1.1.4 icmp: echo request
 118: 13:42:25.173102       10.1.1.4 > 192.168.190.190 icmp: echo reply
 119: 13:42:25.173117       10.1.1.4 > 192.168.190.190 icmp: echo reply


North South Traffic

Transit interface
nn-asav-02# sh capture transit | i 116.202
 638: 13:53:09.315947       10.2.2.4.49396 > 116.202.55.106.80: SWE 426120704:426120704(0) win 8192 <mss 1418,nop,wscale 8,nop,nop,sackOK>
 646: 13:53:09.407327       116.202.55.106.80 > 10.2.2.4.49396: S 1483057760:1483057760(0) ack 426120705 win 62780 <mss 1380,nop,nop,sackOK,nop,wscale 0>
 647: 13:53:09.408777       10.2.2.4.49396 > 116.202.55.106.80: . ack 1483057761 win 512
 648: 13:53:09.409082       10.2.2.4.49396 > 116.202.55.106.80: P 426120705:426120827(122) ack 1483057761 win 512
 649: 13:53:09.500340       116.202.55.106.80 > 10.2.2.4.49396: . ack 426120827 win 62658
 650: 13:53:09.500340       116.202.55.106.80 > 10.2.2.4.49396: P 1483057761:1483058174(413) ack 426120827 win 62658
 651: 13:53:09.500355       116.202.55.106.80 > 10.2.2.4.49396: F 1483058174:1483058174(0) ack 426120827 win 62658
 652: 13:53:09.501728       10.2.2.4.49396 > 116.202.55.106.80: . ack 1483058175 win 510
 653: 13:53:09.514820       10.2.2.4.49396 > 116.202.55.106.80: F 426120827:426120827(0) ack 1483058175 win 510
 654: 13:53:09.606200       116.202.55.106.80 > 10.2.2.4.49396: . ack 426120828 win 62658

Outside interface
nn-asav-02# sh capture outside | i 116.202
 689: 13:53:09.316038       172.16.250.5.49396 > 116.202.55.106.80: SWE 1497025038:1497025038(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK>
 703: 13:53:09.407312       116.202.55.106.80 > 172.16.250.5.49396: S 186944561:186944561(0) ack 1497025039 win 62780 <mss 1460,nop,nop,sackOK,nop,wscale 0>
 704: 13:53:09.408792       172.16.250.5.49396 > 116.202.55.106.80: . ack 186944562 win 512
 705: 13:53:09.409082       172.16.250.5.49396 > 116.202.55.106.80: P 1497025039:1497025161(122) ack 186944562 win 512
 706: 13:53:09.500325       116.202.55.106.80 > 172.16.250.5.49396: . ack 1497025161 win 62658
 707: 13:53:09.500340       116.202.55.106.80 > 172.16.250.5.49396: P 186944562:186944975(413) ack 1497025161 win 62658
 708: 13:53:09.500355       116.202.55.106.80 > 172.16.250.5.49396: F 186944975:186944975(0) ack 1497025161 win 62658
 709: 13:53:09.501744       172.16.250.5.49396 > 116.202.55.106.80: . ack 186944976 win 510
 710: 13:53:09.514835       172.16.250.5.49396 > 116.202.55.106.80: F 1497025161:1497025161(0) ack 186944976 win 510
 711: 13:53:09.606185       116.202.55.106.80 > 172.16.250.5.49396: . ack 1497025162 win 62658
 712: 13:53:09.699075       172.16.250.5.49396 > 116.202.55.106.80: R 1497025162:1497025162(0) ack 186944976 win 510

nn-asav-02# sh run nat
!
object network spoke-subnet
 nat (transit,outside) dynamic interface

nn-asav-02# sh xlate
4 in use, 156 most used
Flags: D - DNS, e - extended, I - identity, i - dynamic, r - portmap,
       s - static, T - twice, N - net-to-net

TCP PAT from transit:10.2.2.4/49382 to outside:172.16.250.5/49382 flags ri idle 0:05:05 timeout 0:00:30
TCP PAT from transit:10.2.2.4/49374 to outside:172.16.250.5/49374 flags ri idle 0:05:07 timeout 0:00:30
TCP PAT from transit:10.2.2.4/49358 to outside:172.16.250.5/49358 flags ri idle 0:10:03 timeout 0:00:30
TCP PAT from transit:10.2.2.4/49258 to outside:172.16.250.5/49258 flags ri idle 0:15:07 timeout 0:00:30


Spoke 2 VM:

PS C:\Users\nehali> ipconfig

Windows IP Configuration
Ethernet adapter Ethernet:

   Connection-specific DNS Suffix  . : 5ttmzc5dk2gupcntn2oz1xw1zh.bx.internal.cloudapp.net
   Link-local IPv6 Address . . . . . : fe80::f9ad:fe46:ec88:47d5%12
   IPv4 Address. . . . . . . . . . . : 10.2.2.4
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 10.2.2.1



PS C:\Users\nehali> Invoke-WebRequest -Uri http://icanhazip.com | Select-Object Content

Content
-------
52.149.196.22...




at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)