Today, I’m taking you along on a bit of an adventure I had recently in my home lab. Like many of us, I enjoy tinkering, testing, and sometimes… well, making life a bit tricky for myself just to learn something new. My latest challenge: setting up a slightly peculiar IPSecsite-to-site VPN tunnel between my UniFi gateway and a remote Ubuntu server, all in route-based mode, with the added fun of my Ubuntu box being tucked away behind NAT.
Why on Earth a Site-to-Site VPN? My Lab Setup Story
It all kicked off with a fairly straightforward need: some of my virtual machines and containers on my main network (Site A), managed by a trusty UniFi gateway (running UniFi OS), needed a second public IP address to get out onto the internet. My Site A is its own little world with several subnets (from 10.0.0.0/16 up to 10.3.0.0/16).
I’d looked into a few options:
Physical WAN2? No dice. Only one fibre connection coming into the house. I did have a bit of a Heath Robinson setup involving a virtualised MikroTik CHR injecting an L2TP connection, but honestly, it wasn’t exactly elegant and I didn’t fancy the idea of my packets taking unnecessary detours.
The built-in UniFi VPN client?WireGuard or OpenVPN, fair enough. I gave it a whirl with a WireGuard server at my remote site (Site B). It worked for getting my Site A machines out via Site B’s IP. But then, curiosity got the better of me: what if I wanted my LXC containers on the Ubuntu server at Site B to be able to hit services back at Site A? The UniFi interface wouldn’t let me configure the routes quite how I wanted for that two-way chat. Bit frustrating, that!
So, that left the last resort before thinking about more drastic measures: the good old site-to-site VPN. UniFi OS offers IPSec or OpenVPN. Having a slight personal aversion to OpenVPN (often resource-hungry and not the speediest), my choice naturally gravitated towards IPSec.
The goal was clear:
Let specific machines at Site A browse the web using Site B’s public IP.
Open the door for machines at Site B (on Ubuntu) to chat with those at Site A.
And as a bonus, be able to forward ports from Site B’s public IP to services hosted at Site A.
The little twist: at Site B, I don’t control the internet gateway. My VPN endpoint had to be my Ubuntu server (10.100.0.2), which itself sits behind the ISP router’s NAT. Looked like it would be fun!
UniFi and StrongSwan: What’s Under the Bonnet
Before diving headfirst into the config, a small detail that matters quite a bit: UniFi OS (at least v4.x, based on Debian 11) uses a well-known open-source implementation for IPSec behind the scenes: StrongSwan (around version 5.9.x as I write this). The graphical interface, pretty as it is, just generates standard config files for StrongSwan. Knowing that helps understand things a bit better and figure out how to set up the other end of the tunnel.
The Dilemma: Policy-Based or Route-Based?
IPSec can decide which traffic to encrypt in two main ways:
Policy-Based: You set strict rules: this source network can talk to that destination network via the tunnel. Simple on the surface, but quickly becomes limiting if you want finer-grained routing.
Route-Based: You create a virtual network interface dedicated to the tunnel (a VTI, Virtual Tunnel Interface, or using XFRM under Linux). Then, the system’s standard routing decides what traffic gets sent into this interface. Anything that goes in gets encrypted. Much more flexible!
I’ll admit, I first tried the easy route with policy-based mode on the Ubuntu side. Big disappointment: couldn’t get the internet traffic routing (0.0.0.0/0) from Site A to Site B working properly. Things went pear-shaped. Even though UniFi was set to route-based, it seemed to expect a compatible routing setup on the other side for everything to flow nicely, especially the return traffic.
No two ways about it, had to roll up my sleeves and go full route-based on both ends. That meant configuring StrongSwan with a VTI on Ubuntu, all while behind NAT… Challenge accepted!
The Prerequisites
Before getting our hands dirty, let’s check we’ve got everything we need:
A UniFi gateway running UniFi OS v4.1 or newer.
A Linux server, in my case Ubuntu 24.04.2 LTS, with StrongSwan and its extra plugins installed (sudo apt update && sudo apt install strongswan strongswan-pki libstrongswan-extra-plugins).
Admin access on both bits of kit, obviously.
Static public IP addresses (or properly configured dynamic DNS names) for both sites.
Firewall Rules: Dead important, this! You need to allow the right ports and protocols inbound on each site’s public IP (directly on the UniFi gateway at Site A, and on Site B’s gateway with port forwarding to the Ubuntu server’s local IP 10.100.0.2):
UDP/500: For the initial key exchange (ISAKMP / IKE).
UDP/4500: For NAT-Traversal (NAT-T), which wraps IPSec in UDP when one end is behind NAT.
IP Protocol 50 (ESP): This carries the actual encrypted data. Note: When NAT-T is active, ESP travels inside UDP/4500, so explicitly allowing protocol 50 isn’t always strictly needed, but when in doubt, best to allow it.
The Target Network Architecture
To get a clearer picture, here’s what our setup looks like:
Site A
Public IP : 80.67.160.2/19
4 subnets :
10.0.0.0/16
10.1.0.0/16
10.2.0.0/16
10.3.0.0/16
tunnel IP : 192.168.255.1/30
Site B
Public IP : 45.13.104.2/22
2 subnets :
10.100.0.0/16
10.101.0.0/16
server local IP : 10.100.0.2/16
tunnel IP : 192.168.255.2/30
(Of course, these IPs are just examples, inspired by real events!)
The Configuration: Into the Tunnels We Go
1. Site A Side: The UniFi Gateway
Off to the UniFi Network Application > Settings > VPN > Site-to-Site VPN web interface.
Hit save. UniFi OS will then translate all this into StrongSwan config files deep within the system. The key takeaway here is that in route-based mode, UniFi uses mark values to identify tunnel packets and route them correctly, without StrongSwan needing to install routes itself (install_routes = no).
A Look at the Config Files Generated by the Web UI
StrongSwan Daemon Configuration:
/etc/strongswan.conf
# Generated automatically by ubios-udapi-servercharon { install_routes = no routing_table = 0 delete_rekeyed = yes make_before_break = no}
/etc/ipsec.conf
# Generated automatically by ubios-udapi-serverconfig setup charondebug="ike 0, knl 0, cfg 0, net 0"include /etc/ipsec.d/tunnels/*.config
/etc/ipsec.secrets
# Generated automatically by ubios-udapi-serverinclude /etc/ipsec.d/tunnels/*.secret
Here, we whip out the text editor and configure everything by hand.
a) General StrongSwan Config
First file, let’s make sure StrongSwan loads all its modern gubbins.
/etc/strongswan.conf
charon { # Let the system handle routes, not StrongSwan directly install_routes = no routing_table = 0 # Standard options delete_rekeyed = yes make_before_break = no # Very important: load the plugins! load_modular = yes plugins { include strongswan.d/charon/*.conf } # For debugging, you can enable logging here if needed # filelog { ... }}
The load_modular = yes is crucial for plugins like kernel-netlink (which talks to the kernel for VTI/XFRM) and updown (which runs our magic script) to be active. Without it, you risk falling back on outdated methods.
b) Main IPSec Config File
This one’s dead simple, it just points to where the tunnel configurations live.
/etc/ipsec.conf
config setup # Debug logging for charon (set to 0 in production) charondebug="ike 0, knl 0, cfg 0, net 0"# Include all connection configs from the tunnels directoryinclude /etc/ipsec.d/tunnels/*.config
c) The Shared Secret
First, the global file that includes the specific secrets:
/etc/ipsec.secrets
include /etc/ipsec.d/tunnels/*.secret
Then, the file holding our actual secret key for this specific tunnel:
/etc/ipsec.d/tunnels/unifi.secret
# Format: <Local ID> <Remote ID> : PSK "YourSuperStrongSecretKey"# We use the public IPs as identifiers for the IKE negotiation45.13.104.2 80.67.160.2 : PSK "bmcCJdIBH1yi6JLOckjJhDHdYACBfvwd"
Careful, the identifiers here (45.13.104.2 and 80.67.160.2) must match the leftid and rightid defined in the connection config just below.
d) The Connection Config
This is the engine room on the Ubuntu side.
/etc/ipsec.d/tunnels/unifi.config
conn unifi-to-ubuntu # A name for our connection ## The Basics ## auto=start # Start the tunnel as soon as StrongSwan launches authby=secret # We're using our secret key (PSK) type=tunnel # It's an IPSec tunnel ## Dead Peer Detection (DPD) ## dpdaction=restart # If UniFi stops responding, restart the connection dpddelay=30s # Check every 30s dpdtimeout=120s # Wait 120s before giving up ## The Addresses ## # 'left' is the local source IP for the VTI tunnel (our private IP) left=10.100.0.2 # 'right' is the destination IP (UniFi's public IP) right=80.67.160.2 # 'mark': The magic mark! Must be identical to the 'key' in the VTI script. # This links the IPSec policy to the virtual interface in the kernel. mark=6 ## Traffic to Encrypt (Traffic Selectors) ## # 'leftsubnet': What traffic originating from Site B can go through the tunnel? # 0.0.0.0/0 = all locally initiated traffic could potentially use it. # Could restrict to 10.100.0.0/16,10.101.0.0/16 if needed. leftsubnet=0.0.0.0/0 # 'rightsubnet': Which remote networks (Site A) are reachable via this tunnel? # !! Important !! Unlike UniFi, putting 0.0.0.0/0 here caused me some grief. # Specifying the exact networks works like a charm. rightsubnet=192.168.255.0/30,10.0.0.0/16,10.1.0.0/16,10.2.0.0/16,10.3.0.0/16 fragmentation=yes # Allow fragmentation, it can help compress=no # IPSec compression is rarely useful ## Phase 1 (IKEv2): Secure Negotiation ## # 'leftid': How our Ubuntu server identifies itself (its public IP, even though 'left' is private due to NAT) leftid=45.13.104.2 # 'rightid': How UniFi identifies itself (its public IP) rightid=80.67.160.2 keyexchange=ikev2 # Use IKEv2 aggressive=no # Definitely no aggressive mode (old and less secure) # The IKE crypto recipe (must be IDENTICAL to UniFi's) ike=aes256-sha256-modp2048! # Encryption-Integrity-DHGroup reauth=yes # Allow re-authentication ikelifetime=28800s # IKE key lifetime (8h) ## Phase 2 (ESP): Data Encryption ## # The ESP crypto recipe (must also be IDENTICAL to UniFi's and IKE here) esp=aes256-sha256-modp2048! # Encryption-Integrity-DHGroup(PFS) rekey=yes # Rekey ESP regularly keylife=3600s # ESP key lifetime (1h) keyingtries=%forever # Retry indefinitely if it fails # forceencaps=no # Force NAT-T? Pleasant surprise, 'no' was enough, auto-detection worked. # Set to 'yes' if you're having NAT trouble. ## The Magic Script ## # This script will run when the tunnel comes up or goes down leftupdown=/etc/ipsec.d/vti-up-down
A few key points here:
The difference between left (local private IP) and leftid (public identification IP) is typical when dealing with NAT.
The mark=6 is the vital link between this IPSec config and the vti0 interface we’re about to create.
The specific rightsubnet (not 0.0.0.0/0) was the key to getting StrongSwan to create the correct XFRM policies and making routing work on Ubuntu. The mysteries of different implementations…
The leftupdown points to our script that will orchestrate all the networking bits.
e) The Orchestrator Script
This Bash script is the one that actually configures the network when the IPSec tunnel comes up or goes down. Don’t forget to make it executable: sudo chmod +x /etc/ipsec.d/vti-up-down.
/etc/ipsec.d/vti-up-down
#!/bin/bash# Script to manage the VTI interface and routing for StrongSwan# Function for debug messages (useful in /var/log/syslog or strongswan.log)debug_echo() { echo "[VTI SCRIPT DEBUG] $1" >&2; }# --- Our variables (Change WAN_IF!) ---VTI_IF="vti0" # Name for our virtual tunnel interfaceWAN_IF="enp0s6" # !! Change this to YOUR WAN interface name on Ubuntu !!LOCAL_VTI_TUNNEL_ENDPOINT="10.100.0.2" # Private IP of our Ubuntu serverREMOTE_TUNNEL_ENDPOINT="80.67.160.2" # Public IP of the UniFi gatewayMARK="6" # The same mark as in unifi.config!MTU="1419" # MTU for VTI (often slightly less than 1500 with IPSec)TUNNEL_NETWORK="192.168.255.0/30" # The tunnel's networkLOCAL_TUNNEL_IP="192.168.255.2/30" # IP for vti0 on UbuntuREMOTE_TUNNEL_IP="192.168.255.1" # IP for vti0 on the UniFi side (our future gateway)REMOTE_NETWORKS="10.0.0.0/16 10.1.0.0/16 10.2.0.0/16 10.3.0.0/16" # Site A networksdebug_echo "Script started. Action: ${PLUTO_VERB}" # PLUTO_VERB holds things like up-client, down-client etc.case "${PLUTO_VERB}" in up-client|up-host) # The tunnel's coming up! debug_echo "--- UP Phase ---" # 1. Create the VTI interface: the magic happens here! # 'ip tunnel add ... mode vti key ${MARK}' links the interface to the IPSec mark. debug_echo "Creating VTI: ip tunnel add ${VTI_IF} local ${LOCAL_VTI_TUNNEL_ENDPOINT} remote ${REMOTE_TUNNEL_ENDPOINT} mode vti key ${MARK}" ip tunnel add ${VTI_IF} local ${LOCAL_VTI_TUNNEL_ENDPOINT} remote ${REMOTE_TUNNEL_ENDPOINT} mode vti key ${MARK} # 2. Recommended sysctl settings for VTI debug_echo "Configuring sysctl for ${VTI_IF}" sysctl -w net.ipv4.conf.${VTI_IF}.disable_policy=1 # Important for marking # 3. Give our VTI an IP address debug_echo "Assigning IP: ip addr add ${LOCAL_TUNNEL_IP} dev ${VTI_IF}" ip addr add ${LOCAL_TUNNEL_IP} dev ${VTI_IF} # 4. Bring up the VTI interface debug_echo "Bringing interface up: ip link set ${VTI_IF} up mtu ${MTU}" ip link set ${VTI_IF} up mtu ${MTU} # 5. Add routes to Site A via the VTI # All traffic to these networks will go via vti0 -> marking -> IPSec encryption for net in ${REMOTE_NETWORKS}; do debug_echo "Adding route: ip route add ${net} dev ${VTI_IF}" ip route add ${net} dev ${VTI_IF} done # 6. Enable IP forwarding on the Ubuntu server (if not already done) debug_echo "Enabling IP Forwarding" sysctl -w net.ipv4.ip_forward=1 # 7. Set up NAT (Masquerade) for traffic coming from Site A via VTI and going out to the Internet # This is what lets machines at Site A use Site B's public IP! debug_echo "Setting up NAT for ${REMOTE_NETWORKS} going out via ${WAN_IF}" for net in ${REMOTE_NETWORKS}; do # We only add the rule if it doesn't already exist iptables -t nat -C POSTROUTING -s ${net} -o ${WAN_IF} -j MASQUERADE &>/dev/null || \ (debug_echo "Adding NAT rule for ${net}"; iptables -t nat -A POSTROUTING -s ${net} -o ${WAN_IF} -j MASQUERADE) done # Do the same for the tunnel network itself, just in case iptables -t nat -C POSTROUTING -s ${TUNNEL_NETWORK} -o ${WAN_IF} -j MASQUERADE &>/dev/null || \ (debug_echo "Adding NAT rule for ${TUNNEL_NETWORK}"; iptables -t nat -A POSTROUTING -s ${TUNNEL_NETWORK} -o ${WAN_IF} -j MASQUERADE) # 8. Allow traffic through the firewall (FORWARD chain) # Allow traffic to come IN via vti0 and go OUT via vti0 iptables -C FORWARD -i ${VTI_IF} -j ACCEPT &>/dev/null || \ (debug_echo "Adding inbound FORWARD rule from ${VTI_IF}"; iptables -A FORWARD -i ${VTI_IF} -j ACCEPT) iptables -C FORWARD -o ${VTI_IF} -j ACCEPT &>/dev/null || \ (debug_echo "Adding outbound FORWARD rule to ${VTI_IF}"; iptables -A FORWARD -o ${VTI_IF} -j ACCEPT) debug_echo "--- UP Phase finished ---" ;; down-client|down-host) # The tunnel's going down! Time to clean up. debug_echo "--- DOWN Phase ---" # Do the reverse operations, ignoring errors if things no longer exist # 1. Remove FORWARD rules iptables -D FORWARD -i ${VTI_IF} -j ACCEPT &>/dev/null || true iptables -D FORWARD -o ${VTI_IF} -j ACCEPT &>/dev/null || true # 2. Remove NAT rules for net in ${REMOTE_NETWORKS}; do iptables -t nat -D POSTROUTING -s ${net} -o ${WAN_IF} -j MASQUERADE &>/dev/null || true done iptables -t nat -D POSTROUTING -s ${TUNNEL_NETWORK} -o ${WAN_IF} -j MASQUERADE &>/dev/null || true # 3. Remove routes for net in ${REMOTE_NETWORKS}; do ip route del ${net} dev ${VTI_IF} &>/dev/null || true done # 4. Bring down and delete the VTI interface ip link set ${VTI_IF} down || true ip tunnel del ${VTI_IF} || true debug_echo "--- DOWN Phase finished ---" ;;esacdebug_echo "Script finished."exit 0
This script really is the heart of the automation on the Ubuntu side. It creates the vti0 interface, linking it to {properties} mark=6, sets the IP, brings the interface up, adds the routes to Site A, and sets up the necessary NAT (Masquerade) and firewall rules so traffic can flow and Site A machines can get out to the internet via Site B.
f) Apply and Start!
After creating/editing all these files:
# Tell StrongSwan to re-read its configsudo ipsec reload# Restart the service to be sure and to kick off the auto=start connectionssudo systemctl restart strongswan-starter # or just strongswan depending on your system# Check the service is running okaysudo systemctl status strongswan-starter
Analysis: The Little ‘Aha!’ Moments and Mysteries
So, what did we learn on this little adventure?
UniFi’s 0.0.0.0/0 vs Ubuntu’s Specifics: Why UniFi uses leftsubnet=0.0.0.0/0 and rightsubnet=0.0.0.0/0 in its StrongSwan config, but copying that directly to Ubuntu with a manual VTI didn’t work, remains a bit of a mystery. UniFi OS likely has its own internal routing logic that relies on the mark to only encrypt relevant traffic. On Ubuntu, specifying the exact rightsubnets was the key so the XFRM policies were created correctly with mark=6 and the kernel knew what to do.
VTI and XFRM: The VTI interface is a handy abstraction on top of the Linux kernel’s XFRM framework. It gives us a standard network interface (vti0) we can route over. The ip tunnel add ... key ${MARK} command makes the link between this interface and the IPSec mark, which itself is linked to the encryption policies defined in StrongSwan. It’s a winning trio!
LibreSwan, the Red Herring: I also had a go with LibreSwan, an alternative to StrongSwan, which offers slightly more integrated VTI support in its config. But couldn’t get the crypto proposals to match the UniFi end (NO_PROPOSAL_CHOSEN). It just goes to show you need a perfect match for the IKE and ESP algorithms between both ends.
load_modular = yes: Don’t forget this line in strongswan.conf! Without it, StrongSwan might not load the modern plugins and fall back on less flexible methods.
NAT-Traversal: Nice surprise, NAT-T worked automatically (forceencaps=no) despite my Ubuntu server being behind NAT. StrongSwan and UniFi correctly detected the situation and switched to UDP/4500 encapsulation.
The mark: I’ll say it again, the consistency of the mark between StrongSwan’s conn config and the ip tunnel add ... key command is absolutely vital. It’s the glue holding everything together.
Checks: Is It Working?
The moment of truth! How do we know if our tunnel is up and running properly?
StrongSwan Status (on Ubuntu):
sudo ipsec status# Look for the "ESTABLISHED" line for your unifi-to-ubuntu connection.sudo ipsec statusall# For the nitty-gritty details: negotiated algorithms, SPIs, bytes transferred...# Check the networks listed after "===" match your rightsubnet.
VTI Interface and Routing (on Ubuntu):
ip link show vti0# Should show UP, with the correct MTU.ip addr show vti0# Should show the 192.168.255.2/30 IP address.ip route show dev vti0# Should list the routes towards Site A's 10.x.x.x/16 networks.
XFRM Policies (on Ubuntu):
sudo ip xfrm policy# Check the 'dir out' policies towards Site A networks: they must have the 'mark 0x6'.# Also check the 'dir in' and 'dir fwd' policies coming from those networks.sudo ip xfrm state# Should show the active ESP Security Associations (SAs).
VPN Status (on UniFi):
The UniFi GUI should show a nice green circle next to your site-to-site VPN.
If you SSH into the UniFi gateway, commands like ipsec status, ip link, ip route can also give some info.
The Final Tests (the most important bit!):
From Ubuntu (Site B), try pinging a machine at Site A (e.g., ping 10.0.1.10).
From a machine at Site A, try pinging Ubuntu’s tunnel IP (ping 192.168.255.2) or its local IP (ping 10.100.0.2).
The ultimate test: On a machine at Site A, temporarily set its default gateway to Ubuntu’s tunnel IP (192.168.255.2). Then, check your outgoing public IP (e.g., using curl ipinfo.io). You should see Site B’s public IP (45.13.104.2)! If so, routing and NAT are working! (Don’t forget to change the gateway back afterwards, or better yet, set up policy-based routing on your UniFi gateway to only route specific machines via the tunnel).
Conclusion: Bob’s Your Uncle!
And there you have it! It wasn’t exactly a walk in the park, especially with NAT and the subtleties of route-based mode between UniFi OS and a manual StrongSwan/VTI setup on Ubuntu. But what a satisfying feeling when you see packets flowing just the way you want!
The key really was understanding the interplay between StrongSwan, packet marking (mark/key), the VTI interface, and the kernel’s routing, not forgetting the crucial importance of matching the crypto settings and tweaking the rightsubnet config on the Ubuntu side.
Now, I’ve got a robust and flexible tunnel that not only lets me link my sites but also lets me finely choose how my traffic goes out to the internet. I hope sharing this experience might help some of you who might be tackling a similar setup. Feel free to shout if you have any questions or feedback!