I was asked yesterday if I’d ever created a site to site VPN between a VNet inside an Azure Stack Development Kit (ASDK) and my on-premises network. The thought had occurred to me in the past, but given the ASDK’s funky networking, I’d given it a wide berth. Instead, in the past I’ve blogged about how to enable inbound NAT to any public IP address in an ASDK, which until now has met my needs.
As I work on more and more customer scenarios though, the need to assign a public VIP to every VM in an environment in order to access it from the corporate network becomes more troublesome, and doesn’t accurately reflect how production works. This prevents many dev/test scenarios, and many scenarios where customers want to test integration of their existing on-premises systems with Azure Stack Hub before committing to a full Azure Stack Hub deployment.
With gauntlet laid down and curiosity piqued, I set about figuring out how I’d get a site to site VPN set up between my corporate (in my case home) network, and a VNet in my ASDK. This blog will go through (some of) the methodology I used and lessons learned in doing this, rather than just a step by step of the conclusions. Hopefully this will help you understand why some things are set up how they are, so if they don’t work in your network, you can figure out why.
Network Topology
My network is fairly straightforward, and all Ubiquiti Unifi hardware. Cable company modem connects to Unifi Security Gateway 3P, which connects to a Unifi Switch 8, which connects to three other Unifi Switch 8s in different rooms. It also has TV, Xbox, PS4 etc wired into it. Two of the switches in other rooms have wireless APs connected to their PoE ports and serve Wi-Fi for the house, while the third switch is in my lab and has a few servers connected to it. For the purposes of simplicity, I’m just showing the high level items here, including the three physical servers I use for lab work.
Currently when I need to run interoperability scenarios between VMs running on the Hyper-V or VMware hosts and VMs or PaaS components deployed in the ASDK, I use the inbound NAT tweak that I’ve previously documented here . Within the Unifi Router, the static routes set up to enable this look like the below. The AzS-Router1 VM in this deployment has IP 192.168.1.111, which I only note because I used a different IP in the documentation linked just previously.
This is all well and good, but if I deploy a VM without a Public VIP, I have no way to access it from outside the ASDK. My first step in rectifying this was to deploy a VM in the ASDK without an external/public IP address. I called this VPN-VM1. Its Public IP is nonexistent, and Private IP is 192.168.50.4, which is only accessible to VMs deployed in the ASDK and in the same VNet.
Step one in setting up a Site to Site VPN in Azure Stack Hub is to create some pre-requisite items within the ASDK. These are:
Virtual Network
This virtual network will be where the VMs that you want to access from outside the ASDK will reside, as well as the Virtual Network Gateway (below). The Address Space of the Virtual Network has to be sufficiently large to contain at least one subnet for the VMs to run in, and another for the Virtual Network Gateway to run in.In this instance, I’ve named the Virtual Network s2s-vpn-vnet1, and given it an address space of 192.168.50.0/23. This allows two /24 subnets, 192.168.50.0/24 and 192.168.51.0/24. There’s nothing stopping you from creating (for example) a VNet with a /16 address space, with a large number of subnets of varying sizes within it. By default, everything inside a VNet can communicate to everything else in that VNet, regardless of subnet. NSG rules can be used to restrict east-west traffic between subnets. The subnet for the Virtual Network Gateway has to be called GatewaySubnet - this is an Azure convention that must be maintained.
Virtual Network Gateway
The Virtual Network Gateway is the VPN endpoint inside the Azure Stack Hub environment. In an ASDK, this is a shared service, which runs on a single Azure Stack Infrastructure VM that we’ve previously spoken about, in this case AzS-GWY01. This being the case, the external IP address for every Virtual Network Gateway in the ASDK here is always the same, and is the IP address of AzS-GWY01. It’s interesting to note that this prevents us from ever doing a Site to Site VPN between VNets inside an Azure Stack Hub, as both ends of the VPN would share the same IP address.
Name the Virtual Network Gateway, and choose a SKU for it. SKU considerations are here , really the main thing we need to consider for this scenario is bandwidth, so we choose the High Performance SKU.
Note that when creating a Public IP Address for this, it’ll assign a Dynamic IP address. You cannot change this - the IP will be assigned after creation and will be the same as the Public IP of the AzS-GWY01 Infrastructure VM, even though it says it’s Dynamic.
Local Network Gateway
The Local Network Gateway is a representation of the other end of the Site to Site VPN. This could be a VPN gateway in Public Azure, an RRAS server running in or out of the Azure Stack Hub, or any one of a huge number of physical or virtual networking devices which can act as a VPN endpoint. In this case, the Unifi Router in my network will be the Local Network Gateway.
Name the Local Network Gateway, set its IP Address to the WAN IP address of the other VPN endpoint - in this case the WAN IP of my Ubiquiti Unifi Gateway. The Address Space should be filled in with any addresses which are on your local network that you want to access over the VPN from inside the ASDK. In my case, this is just my LAN IP range of 192.168.1.0/24.
Connection
The Connection is the logical representation of the connection between the Virtual Network Gateway and the Local Network Gateway, and is where the pre-shared key for authentication is stored.
Create the Connection, choosing the Virtual Network Gateway and Local Network Gateway previously created. Choose a pre-shared key/password that you want to use for the VPN endpoints to use to handshake successfully.
<
The Connection is created, and… shows as Not Connected, which isn’t surprising, as we haven’t created any configuration at the other end yet.
How you create the next part will depend on what you’re setting up. You can use any VPN endpoint that you want that supports L2TP/IPSec/IKEv2. The Unifi Security Gateway has a configuration page to create a S2S VPN, as shown below.
The Remote Subnets in this case are any subnets in the ASDK VNet created earlier that contain VMs that you want to access over the VPN. The Peer IP is the IP Address of the other end of the VPN, in this case the Virtual Network Gateway IP, 192.168.102.1. The Local WAN IP is the WAN IP of the Unifi Router, or your VPN device’s IP address. The Pre-Shared Key is the key you set when creating the Connection in the ASDK. In the Unifi setup I can select ‘Azure Dynamic Routing’ as an IPsec profile, which works great for Azure but as we’ll see doesn’t quite cut it for an ASDK in this setup.
Normally at this stage, you can initiate traffic from one side or the other of the VPN, it’ll connect, and everything will be happy!
In this case though… no cigar.
Unfortunately, although we can access the Virtual Network Gateway IP (192.168.102.1) from the LAN, it’s only because of the steps previously gone through in this blog , which involves the creation of an inbound NAT. The VPN will not correctly pass traffic over this NAT, so no connection can take place.
To overcome this, there are a few steps that need to be taken. First, logon to the ASDK physical host, open up a PowerShell Administrator window, and run the following commands:
|
|
This will add some new static NAT port mappings between the ASDK host and AzS-GWY01 to the existing BGPNAT NAT rules, which will allow the VPN traffic to be successfully passed to the Virtual Network Gateway.
In order to take advantage of this, the settings in the Unifi Router need to be updated to point to the ASDK Host, from where traffic will be statically NATted through to the Virtual Network Gateway, rather than directly to the IP of the Gateway. My ASDK is 192.168.1.110 on my LAN, yours will be different.
After doing this… it still doesn’t connect. There were a few tweaks I still had to make to my setup to get this working successfully, how you achieve this will depend on your network device.
On the Unifi Router, I edited /etc/ipsec.conf to the following:
|
|
The key changes here are:
###The addition of rightid=192.168.102.1
Because the Virtual Network Gateway is behind a NAT, it isn’t identified with the correct IP automatically, so we need to tell the Unifi Gateway that a return from 192.168.102.1 from something sent to 192.168.1.110 is all good.
###ike=aes256-sha2_384-prfsha384-ecp384!
This matches the IKE settings the ASDK sent in handshake traffic, your mileage may vary
###forceencaps=yes
Forces UDP encapsulation for ESP packets, which allows traffic to survive through the NAT.
###esp=aes256gcm16!
This matches the ESP settings the ASDK sent in handshake traffic, your mileage may vary.
If you make any changes to the config in the Unifi web GUI, it’ll overwrite this config and the VPN will break.
After making these changes, you can bring the VPN up - there are a load of ways to do this, the easiest is just to ping something in the ASDK VNet from the LAN. In order to watch what’s happening, I SSH’d into the Unifi router, and ran sudo ipsec up peer-192.168.1.110-tunnel-vti, which is the name of the VPN. As you can see, everything reports as successful.
Success! The Connection shows as Connected in the Azure Stack portal :D
On the Unifi Gateway, I ran sudo tcpdump -npi vti64, to monitor live traffic across the VPN tunnel. Then I ran a ping from my laptop (192.168.1.31) to a VM in the ASDK VNet (192.168.50.4). Success! Everything works! :)
Now I can create Site to Site VPNs from my home network to any VNet in the ASDK, which is pretty cool. It should also be possible to set up VPNs into third party network appliances running in ASDK VNets as well, including RRAS on a Windows VM if you so desired - it should also be easier to do this. I really wanted to get this working using the built-in network gateway, and I’m happy to say that it’s definitely achievable.