Scalable layer-4 NVA Design in Azure

Architecture

In this blog post we will walk through a scalable active/active NVA (Network Virtual Appliance) design, take a closer look at three different use cases, the challenges and how to preserve flow symmetry for east-west and north-south traffic.

On-Prem to Azure - East-West traffic using Internal Load Balancer HA Ports

Summary

This blog post is inspired by one of my recent partner engagement. Last week I worked with a partner where we needed to design a scalable architecture with NVAs in Azure. When using NVAs in Azure, designing a scalable and resilient architecture is important because the platform level redundancy and scalability is only provided when using Azure native firewalls aka firewall PaaS service.  When using NVAs in Azure, there are certain design considerations and best practices to ensure flow symmetry.  Different firewall vendors have published their best practices. While this particular example is based on Cisco ASAv,  the focus of this blog is on key architectural components which stays the same for any NVA vendor.  There is a great blog article here that notes some of the key differences between NVAs and native firewalls.

Microsoft and NVA vendor documentation

There's great documentation from NVA vendors and Azure and I've provided the link to those here.

Microsoft Documentation on HA ports and NVA here
Palo Alto documentation link here
Cisco documentation link here

Scribble

Warning - geek alert!   I started to scribble this design and the flows got out of control. Let's untangle this and get a deeper understanding of two types of traffic flows

1. East-West traffic : Preserve flow symmetry using  single NIC NVA and internal load balancer (ILB) with HA Ports 
2. North-South traffic: Preserve flow symmetry with ELB-NVA-ILB aka "NVA load balancer sandwich design" using SNAT.

Joys of Software Defined Networking....Let's untangle

Cisco Side Configuration:

This is view of how the firewall interfaces are configured. Note: Outside interface is only used when using the north-south design.

nn-asav-01# sh ip add
System IP Addresses:
Interface                Name                   IP address      Subnet mask     Method
GigabitEthernet0/0       outside                172.16.250.4    255.255.255.0   DHCP
GigabitEthernet0/1       transit                172.16.3.4      255.255.255.0   DHCP
Management0/0            management             172.16.1.8      255.255.255.0   DHCP


nn-asav-01# sh run route
route outside 0.0.0.0 0.0.0.0 172.16.250.1 1
route transit 10.1.0.0 255.255.0.0 172.16.3.1 1
route transit 10.2.0.0 255.255.0.0 172.16.3.1 1
route transit 168.63.129.16 255.255.255.255 172.16.3.1 1
route transit 192.168.190.0 255.255.255.0 172.16.3.1 1

Allow ssh from Load balancer probes
nn-asav-01# sh run ssh
ssh stricthostkeycheck
ssh timeout 60
ssh version 2
ssh key-exchange group dh-group14-sha256ssh 168.63.129.16 255.255.255.255 transit
ssh 0.0.0.0 0.0.0.0 management
ssh 168.63.129.16 255.255.255.255 outside

nn-asav-02# sh ip add
System IP Addresses:
Interface                Name                   IP address      Subnet mask     Method
GigabitEthernet0/0       outside                172.16.250.5    255.255.255.0   DHCP
GigabitEthernet0/1       transit                172.16.3.5      255.255.255.0   DHCP
Management0/0            management             172.16.1.9      255.255.255.0   DHCP

nn-asav-02# sh run route
route outside 0.0.0.0 0.0.0.0 172.16.250.1 1
route transit 10.1.0.0 255.255.0.0 172.16.3.1 1
route transit 10.2.0.0 255.255.0.0 172.16.3.1 1
route transit 168.63.129.16 255.255.255.255 172.16.3.1 1
route transit 192.168.190.0 255.255.255.0 172.16.3.1 1

Understanding Flows and Key Design Points

This particular blog post is based on Active-Active NVA configuration. There's also Active/standby design which requires some automation effort to ensure fast failover. There is more documentation on the active/standby design here.

Common Design principles

• Active-Active NVA Configuration
• Fast failover and per-instance health probes
• Scale-out to N active instances

East-west traffic:

• This is a single NIC design with single transit interface
• Use HA ports with Internal Load Balancer for east-west flow symmetry
• Use UDRs with east-west traffic to steer the traffic via the NVA
• When using UDRs from spoke to on-prem traffic make sure that  "route-propagation" is disabled on the spoke UDRs. This setting is available under UDR configuration.

North-South traffic:

• This is a two NIC design with Outside and transit interface
• Typical design is as an internet edge firewall
• Use SNAT with ELB + Firewall + ILB aka  firewall load balancer sandwich
• Associate Public IP on the egress interface of the NVA for outbound NAT
• NSGs should be applied on transit and outside interfaces

Be sure to checkout the limitations here

East-West Traffic

This is a single NIC design with single transit interface. You can still have a management interface on the NVA. Internal load balancer is configured to Use HA ports  for east-west flow symmetry.  Use UDRs  to steer the traffic via the NVA. There are two different use cases that we will review

1. For spoke to spoke traffic the UDRs are configured on spoke subnets. 

2. For on-prem to Azure traffic the UDR is configured on the gateway subnet.

From Azure Documentation here

A high availability (HA) ports load-balancing rule is a variant of a load-balancing rule, configured on an internal Standard Load Balancer. You can simplify the use of a load balancer by providing a single rule to load-balance all TCP and UDP flows that arrive on all ports of an internal Standard Load Balancer. The load-balancing decision is made per flow. This action is based on the following five-tuple connection: source IP address, source port, destination IP address, destination port, and protocol

Spoke-to-Spoke traffic:

Green path: Spoke1 (Source: VM)-> UDR -> ILB (.250) -> NVA1->Spoke2(Dest: VM)

Return Traffic (Orange path): Spoke2 (VM) ->UDR -> ILB (.250) -> Same NVA1 -> Spoke2 (VM1)

Load balancing decision made on the basis of the 5-tuple hash which stays the same.

Example:

SRC-IP: 10.1.1.4 DST-IP:10.2.2.4 SRC-PORT:45678  DST-PORT-3389

SRC-IP: 10.2.2.4 DST-IP: 10.1.1.4 SRC-PORT:3389 DST-PORT: 45678

Spoke-to-spoke - East-West traffic using Internal Load Balancer HA Ports

Azure side Configuration:

Spoke1-route-table

Spoke2-route-table

Azure Load Balancer configuration:

Note: LB probe is configure for tcp/22

On-prem to Azure traffic:

Blue path: On-Prem (Source: VM)->ExpressRoute/VPN -> Gateway Subnet-> UDR -> ILB (.250) -> NVA1->Spoke1(Dest: VM)

Return Traffic (Magenta path): Spoke1 (VM) ->UDR -> ILB (.250) -> Same NVA1 -> ExpressRoute or VPN -> On-Prem (Dest VM)

Load balancing decision made on the basis of the 5-tuple hash which stays the same.

Example:

SRC-IP: 192.168.190.190 DST-IP:10.1.1.4 SRC-PORT:45678  DST-PORT-22

Return Traffic SRC-IP: 10.1.1.4 DST-IP: 192.168.190.190 SRC-PORT:22 DST-PORT: 45678

For on-prem to Azure traffic the UDR is configured on the gateway subnet as well as the spoke subnets to route via ILB ( Internal load balancer). Be sure to disable the "route propagation" checkbox on the Spoke UDR to steer the traffic via NVA.

On-Prem to Azure - East-West traffic using Internal Load Balancer HA Ports

Gateway-Subnet UDR to steer on-prem traffic via ILB

North South Traffic

This is a two NIC design with Outside and transit interface. Use SNAT with ELB + Firewall + ILB aka  firewall load balancer sandwich. Associate Public IP on the egress interface of the NVA for outbound NAT

Internet to Azure - North-South traffic using firewall load Balancer sandwich design using SNAT

Azure side configuration:

Option#1: (SNAT on NVA egress interface with public IP assigned to the NVA)
Map the outside interface on the firewall to a public IP in Azure

Map Public IP to outside interface on ASAv-01

Map Public IP to outside interface on ASAv-02

Option #2:   SNAT on firewall egress interface and ELB SNAT

Azure External Loadbalancer outbound SNAT rule

External load balancer frontend configuration

ELB outbound SNAT rule

Testing and Validations

LB Probes on tcp/22

nn-asav-01# sh capture transit  | i 168.63

  49: 14:29:54.171667       168.63.129.16.65329 > 172.16.3.4.22: R 2388247501:2388247501(0) ack 1086528740 win 0

  50: 14:29:54.172125       168.63.129.16.65350 > 172.16.3.4.22: SWE 2553081639:2553081639(0) win 8192 <mss 1440,nop,wscale 8,nop,nop,sackOK>

  51: 14:29:54.172171       172.16.3.4.22 > 168.63.129.16.65350: S 2684413220:2684413220(0) ack 2553081640 win 8192 <mss 1380>

  52: 14:29:54.172476       168.63.129.16.65350 > 172.16.3.4.22: . ack 2684413221 win 64860

  53: 14:29:54.173040       172.16.3.4.22 > 168.63.129.16.65350: P 2684413221:2684413240(19) ack 2553081640 win 32768

  56: 14:29:54.234072       168.63.129.16.65350 > 172.16.3.4.22: . ack 2684413240 win 64841

 101: 14:29:57.233981       172.16.3.4.33344 > 168.63.129.16.80: S 1483242750:1483242750(0) win 32768 <mss 1460,nop,nop,timestamp 600641695 0>

 129: 14:30:00.171744       168.63.129.16.65350 > 172.16.3.4.22: R 2553081640:2553081640(0) ack 2684413240 win 0

 130: 14:30:00.172430       168.63.129.16.65403 > 172.16.3.4.22: SWE 723143324:723143324(0) win 8192 <mss 1440,nop,wscale 8,nop,nop,sackOK>

East West Traffic

Spoke-to-spoke

Notice two packet for request and reply, entering and leaving the same transit interface

nn-asav-01# sh capture transit | i icmp

  21: 13:45:12.085215       10.1.1.4 > 10.2.2.4 icmp: echo request

  22: 13:45:12.085231       10.1.1.4 > 10.2.2.4 icmp: echo request

  23: 13:45:12.087031       10.2.2.4 > 10.1.1.4 icmp: echo reply

  24: 13:45:12.087046       10.2.2.4 > 10.1.1.4 icmp: echo reply

  62: 13:45:13.086909       10.1.1.4 > 10.2.2.4 icmp: echo request

  63: 13:45:13.086924       10.1.1.4 > 10.2.2.4 icmp: echo request

  64: 13:45:13.088893       10.2.2.4 > 10.1.1.4 icmp: echo reply

  65: 13:45:13.088908       10.2.2.4 > 10.1.1.4 icmp: echo reply

 110: 13:45:14.088999       10.1.1.4 > 10.2.2.4 icmp: echo request

 111: 13:45:14.089106       10.1.1.4 > 10.2.2.4 icmp: echo request

 112: 13:45:14.091242       10.2.2.4 > 10.1.1.4 icmp: echo reply

 113: 13:45:14.091258       10.2.2.4 > 10.1.1.4 icmp: echo reply

 154: 13:45:15.090357       10.1.1.4 > 10.2.2.4 icmp: echo request

 155: 13:45:15.090418       10.1.1.4 > 10.2.2.4 icmp: echo request

 156: 13:45:15.096583       10.2.2.4 > 10.1.1.4 icmp: echo reply

 157: 13:45:15.096598       10.2.2.4 > 10.1.1.4 icmp: echo reply

On-Prem to Spoke

Notice two packet for request and reply, entering and leaving the same transit interface

nn-asav-01# sh capture transit | i icmp

  37: 13:42:23.172903       192.168.190.190 > 10.1.1.4 icmp: echo request

  38: 13:42:23.173010       192.168.190.190 > 10.1.1.4 icmp: echo request

  39: 13:42:23.174780       10.1.1.4 > 192.168.190.190 icmp: echo reply

  40: 13:42:23.174795       10.1.1.4 > 192.168.190.190 icmp: echo reply

  75: 13:42:24.170569       192.168.190.190 > 10.1.1.4 icmp: echo request

  76: 13:42:24.170675       192.168.190.190 > 10.1.1.4 icmp: echo request

  77: 13:42:24.172674       10.1.1.4 > 192.168.190.190 icmp: echo reply

  78: 13:42:24.172690       10.1.1.4 > 192.168.190.190 icmp: echo reply

 116: 13:42:25.171118       192.168.190.190 > 10.1.1.4 icmp: echo request

 117: 13:42:25.171240       192.168.190.190 > 10.1.1.4 icmp: echo request

 118: 13:42:25.173102       10.1.1.4 > 192.168.190.190 icmp: echo reply

 119: 13:42:25.173117       10.1.1.4 > 192.168.190.190 icmp: echo reply

North South Traffic

Transit interface

nn-asav-02# sh capture transit | i 116.202

 638: 13:53:09.315947       10.2.2.4.49396 > 116.202.55.106.80: SWE 426120704:426120704(0) win 8192 <mss 1418,nop,wscale 8,nop,nop,sackOK>

 646: 13:53:09.407327       116.202.55.106.80 > 10.2.2.4.49396: S 1483057760:1483057760(0) ack 426120705 win 62780 <mss 1380,nop,nop,sackOK,nop,wscale 0>

 647: 13:53:09.408777       10.2.2.4.49396 > 116.202.55.106.80: . ack 1483057761 win 512

 648: 13:53:09.409082       10.2.2.4.49396 > 116.202.55.106.80: P 426120705:426120827(122) ack 1483057761 win 512

 649: 13:53:09.500340       116.202.55.106.80 > 10.2.2.4.49396: . ack 426120827 win 62658

 650: 13:53:09.500340       116.202.55.106.80 > 10.2.2.4.49396: P 1483057761:1483058174(413) ack 426120827 win 62658

 651: 13:53:09.500355       116.202.55.106.80 > 10.2.2.4.49396: F 1483058174:1483058174(0) ack 426120827 win 62658

 652: 13:53:09.501728       10.2.2.4.49396 > 116.202.55.106.80: . ack 1483058175 win 510

 653: 13:53:09.514820       10.2.2.4.49396 > 116.202.55.106.80: F 426120827:426120827(0) ack 1483058175 win 510

 654: 13:53:09.606200       116.202.55.106.80 > 10.2.2.4.49396: . ack 426120828 win 62658

Outside interface

nn-asav-02# sh capture outside | i 116.202

 689: 13:53:09.316038       172.16.250.5.49396 > 116.202.55.106.80: SWE 1497025038:1497025038(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK>

 703: 13:53:09.407312       116.202.55.106.80 > 172.16.250.5.49396: S 186944561:186944561(0) ack 1497025039 win 62780 <mss 1460,nop,nop,sackOK,nop,wscale 0>

 704: 13:53:09.408792       172.16.250.5.49396 > 116.202.55.106.80: . ack 186944562 win 512

 705: 13:53:09.409082       172.16.250.5.49396 > 116.202.55.106.80: P 1497025039:1497025161(122) ack 186944562 win 512

 706: 13:53:09.500325       116.202.55.106.80 > 172.16.250.5.49396: . ack 1497025161 win 62658

 707: 13:53:09.500340       116.202.55.106.80 > 172.16.250.5.49396: P 186944562:186944975(413) ack 1497025161 win 62658

 708: 13:53:09.500355       116.202.55.106.80 > 172.16.250.5.49396: F 186944975:186944975(0) ack 1497025161 win 62658

 709: 13:53:09.501744       172.16.250.5.49396 > 116.202.55.106.80: . ack 186944976 win 510

 710: 13:53:09.514835       172.16.250.5.49396 > 116.202.55.106.80: F 1497025161:1497025161(0) ack 186944976 win 510

 711: 13:53:09.606185       116.202.55.106.80 > 172.16.250.5.49396: . ack 1497025162 win 62658

 712: 13:53:09.699075       172.16.250.5.49396 > 116.202.55.106.80: R 1497025162:1497025162(0) ack 186944976 win 510

nn-asav-02# sh run nat

!

object network spoke-subnet

 nat (transit,outside) dynamic interface

nn-asav-02# sh xlate

4 in use, 156 most used

Flags: D - DNS, e - extended, I - identity, i - dynamic, r - portmap,

       s - static, T - twice, N - net-to-net

TCP PAT from transit:10.2.2.4/49382 to outside:172.16.250.5/49382 flags ri idle 0:05:05 timeout 0:00:30

TCP PAT from transit:10.2.2.4/49374 to outside:172.16.250.5/49374 flags ri idle 0:05:07 timeout 0:00:30

TCP PAT from transit:10.2.2.4/49358 to outside:172.16.250.5/49358 flags ri idle 0:10:03 timeout 0:00:30

TCP PAT from transit:10.2.2.4/49258 to outside:172.16.250.5/49258 flags ri idle 0:15:07 timeout 0:00:30

Spoke 2 VM:

PS C:\Users\nehali> ipconfig

Windows IP Configuration

Ethernet adapter Ethernet:

   Connection-specific DNS Suffix  . : 5ttmzc5dk2gupcntn2oz1xw1zh.bx.internal.cloudapp.net

   Link-local IPv6 Address . . . . . : fe80::f9ad:fe46:ec88:47d5%12

   IPv4 Address. . . . . . . . . . . : 10.2.2.4

   Subnet Mask . . . . . . . . . . . : 255.255.255.0

   Default Gateway . . . . . . . . . : 10.2.2.1

PS C:\Users\nehali> Invoke-WebRequest -Uri http://icanhazip.com | Select-Object Content

Content

-------

52.149.196.22...