VPN Adventures: Tunneling

Disclaimer: I know relatively little about networking under Linux, so don’t be surprised if some of what’s here is flat out wrong. Then again, that’s why Google is your friend…

Edited 6/25/2019 to include instructions on configuring pitiable rules to allow Remote Desktop access…

If you read about my adventures getting my AT&T fiber service to operate continuously at near gigabit speed you’ll know that I tried hard to keep my Cisco RV-325 in the mix to handle VPN connections from iOS devices. It was quite a struggle! The RV-325 documentation presumes you know what you’re doing, which definitely doesn’t describe me when it comes to networking, let alone VPNs.

Actually, that observation could be applied to much of the networking and VPN documentation I’ve been reading online. It tends to presume you know what you’re doing, perhaps on the theory that no one would think to try setting up a VPN or diving into the bowels of networking unless they knew what they were doing. But fools rush in where angels fear to tread…

In fairness, a lot of VPN details are inherently pretty involved. Where else do you worry about what encryption protocol you’re using — it must be the same on both client and server — how much entropy you want to include in your keys, and just what is a Diffie Hellman file anyway and why should I want one?

In any event, here’s how I eventually got my VPN setup working so that I can access it from iOS devices.

Table of Contents

My Environment

  • the server runs on a Raspberry Pi 3B+
  • the underlying OS is the Raspberry Pi-tweak of Debian Stretch, with all the latest updates installed as of May, 2019
  • I don’t use the WiFi interface on the Pi; it’s hardwired into my LAN
  • My VPN must use the tunneling approach because I need to support iOS. Apple, in its infinite wisdom, has declined to support the other mode (tap).
  • I use my Pi as a DNS server for my LAN, via dnsmasq. Which is a really slick little package…except for one !#$!#@$I& “feature” that nearly caused me to have a stroke, not once, but twice.

return to table of contents

Tunnel vs Tap

VPNs can swing in (at least) two ways, tunnel and tap. A tunnel defines a separate IP subnet, with its own addresses separate from the LAN that the VPN server sits on, and adds remote clients into that new subnet.

For example, my LAN uses the 192.168.1.x/24 subnet, and my VPN offers addresses in the 192.168.5.x/24 space. There are a few tweaks to let the two subnets interact with each other, but while not obvious, they’re fairly straightforward (I’ll describe them below).

A tap hands out IP addresses within the LAN’s subnet to the remote VPN clients, so those remote clients are automatically part of the LAN and can interact with all parts of it. A tap VPN has to have a reserved block of addresses to manage, which your DHCP setup should avoid using.

A tunnel VPN works by creating a virtual network adapter (typically called tunX, in my case tun0) that acts as a portal between the server and its remote clients. Think of it as being like a virtual ethernet switch into which all the remote clients are plugged.

return to table of contents

Step 1: Configure a Dynamic DNS Referrer

This may be optional for you, if your broadband connection comes with an IP address that never changes. My understanding is that cable connections do change addresses occasionally. AT&T fiber connections apparently change very rarely. But they can change, and if yours does, remote clients won’t be able to find your VPN server.

There are a number of services that enable you to store a DNS record that maps your broadband IP to a human-readable name, and keep the reference updated if and when you broadband provider changes the IP address on you. That’s done via a daemon that runs on the Pi and checks your external IP address periodically, updating what’s on file for you as necessary.

I’ve used DynDns.com for several years with no problem. I’m currently trying out NoIP.com because the DynDNS client that runs on the Pi, ddclient, does not encrypt the DynDNS account password in its config file. Not a big deal unless someone breaks into your Pi, but not a great practice. The NoIP.com client encrypts its config file on the Pi. NoIP is also about half the cost of DynDNS (NoIP also offers a free service…but you have to put up with ads being injected into your redirect stream, which might not be a big deal for this kind of VPN setup, but still disturbed me).

return to table of contents

Step 2: Install OpenVPN via PiVPN

I’m sure there are other open source VPN packages out there, but OpenVPN has been around for quite a while, is well supported, and offers its own iOS client-side package for interacting with the server. The logging abilities of which were really important in getting everything to work. There’s nothing like a message describing what step of the initialization process went wrong on the client side to help speed up problem solving!

You can install OpenVPN in a lot of ways, but doing it via PiVPN makes key/certificate generation and initial configuration much easier, IMHO. Follow the instructions, fill in the blanks and, while you’ll still have some work to do, you’ll never have to figure out just what a Diffie Hellman file is.

Remember to include the “referrer name” you set up in step 1 instead of your external broadband IP address, unless you’re just willing to hard-wire that IP address in (and adjust it yourself if your broadband provider changes the IP address on you).

return to table of contents

Step 3: Tweak dnsmasq Configuration

Only one change is necessary to your dnsmasq configuration, but it’s important:

# only listen on this interface
interface=eth0
interface=tun0

You have to tell dnsmasq to listen on that new, virtual tun0 interface in addition to whatever “normal” interface you have it monitor (in my case, eth0). If you don’t, your VPN clients won’t be able to resolve names and your LAN will be invisible to you. Unless you’re really good at guessing DHCP-assigned IP addresses or you stick to only static IP addresses. Figuring this out cost me an embarrassing number of hours.

BTW, dnsmasq has a bizarre (to me) design “feature” that can be coronary-inducing: like many linux daemons, you can configure it either through a single file (typically /etc/dnsmasq.conf) or through multiple files in a directory (typically /etc/dnsmasq.d). That part’s perfectly normal.

What I keep forgetting is that dnsmasq, by default, attempts to parse almost every single file in the configuration directory (there are default exceptions, but they’re pretty odd, IMHO). So, if you’re used to “only the files ending in .conf” (or whatever) getting parsed, you’re in for a rude awakening if you leave other files in that directory.

Which I did at one point, and which made the internet invisible to all the devices on my LAN (if your DNS service is down there ain’t no resolution happening and, yes, shame on me for not checking to see if dnsmasq was down — but when you’re focused on configuring network stuff it’s easy to get brain-channeled into not remembering to check the obvious).

Interestingly, dnsmasq does not recursively scan its configuration directory. So you can stick alternate and backup configuration files in a subdirectory of the configuration directory with no problem. Which is what I did after my head stopped exploding from trying to figure out why “reverting everything to the way it was” left me with a broken LAN (I have a habit of saving different versions of config files with different suffixes in config directories, like localnet.conf.tap; that’s a no-no with dnsmasq).

return to table of contents

Step 4: Tweak OpenVPN’s server.conf file

The server.conf file is typically located in the /etc/openvpn directory. After tweaking it, here’s what mine looked like:

dev tun
proto udp 
port ***redacted*** 
ca /etc/openvpn/easy-rsa/pki/ca.crt 
cert /etc/openvpn/easy-rsa/pki/issued/server_X4O6Por7FYITsLun.crt 
key /etc/openvpn/easy-rsa/pki/private/server_X4O6Por7FYITsLun.key 
dh none 
topology subnet 
server 192.168.5.0 255.255.255.0 
# Set your primary domain name server address for clients 
push "route 192.168.1.0 255.255.255.0 192.168.1.254" 
push "dhcp-option DNS 192.168.1.5" 
push "dhcp-option DOMAIN localnet" 
push "dhcp-option SEARCH localnet" 
# Prevent DNS leaks on Windows 
push "block-outside-dns" 
# Override the Client default gateway by using 0.0.0.0/1 and 
# 128.0.0.0/1 rather than 0.0.0.0/0. This has the benefit of 
# overriding but not wiping out the original default gateway. 
push "redirect-gateway def1" 
client-to-client 
keepalive 1800 3600 
remote-cert-tls client 
tls-version-min 1.2 
tls-crypt /etc/openvpn/easy-rsa/pki/ta.key 
cipher AES-256-CBC 
auth SHA256 
user nobody 
group nogroup 
persist-key 
persist-tun 
crl-verify /etc/openvpn/crl.pem 
status /var/log/openvpn-status.log 20 
status-version 3 
syslog 
verb 3 
#DuplicateCNs allow access control on a less-granular, per user basis. 
#Remove # if you will manage access by user instead of device. 
#duplicate-cn # Generated for use by PiVPN.io

The lines in this configuration file I ended up tweaking were the following:

  • Line 1: this is what tells OpenVPN to create a tunnel rather than a tap
  • Line 2: you could use TCP here, but it’s more common apparently to use UDP (you may have to ensure your firewall — that’s on the AT&T-supplied BGW210 in my case — forwards UDP connections to the Pi)
  • Line 3: by default OpenVPN works through port 1194 (I think). You should probably change that to provide a little bit more protection from hackers.
  • Line 9: this defines the subnet that the remote clients will be in
  • Line 11: this establishes a route between the VPN subnet (192.168.5.x in my case) and the LAN (192.168.1.x in my case). Without this the VPN clients won’t see the devices on the LAN.
  • Line 12: defines the DNS server that the VPN clients will be using (this points at the Pi, in my case, because it is home to dnsmasq)
  • Lines 13 – 14 and 16: not sure what these do, but I read something the recommended including them if you have Windows VPN clients, which I do.

The rest of the file is what came out of the piVPN setup and configuration process. I suspect there are steps you can take to harden your VPN beyond what the PiVPN setup does, but I haven’t had time to experiment with tweaking the OpenVPN settings to find out.

return to table of contents

Step 5: Configure Your Firewall to Forward Packets to the Pi

This is an easy one to forget, since it generally doesn’t involve the Pi (if you’re using a Pi as a LAN firewall I’m impressed). You need to configure the firewall protecting your LAN to forward the proper packets to the Pi. If you don’t, your remote clients won’t be able to connect with your OpenVPN server.

The proper packets are defined by the nature of the protocol they use (TCP or UDP) and the port number they’re trying to reach. You have to make sure you define the firewall forwarding rule for the protocol and port your OpenVPN configuration expects (in my case that was UDP and…well, the port number is a secret).

return to table of contents

Step 6: Configure iptables

Remember what I said about not understanding Linux networking? Well, after I had everything working and I rebooted my Pi to install some updates…the VPN connection stopped working.

Or rather I could remotely connect to the Pi itself, but not to any of the PCs on my LAN. Weirdly, I could ping them from the Pi, but I wasn’t able to interact with them (e.g., via Remote Desktop protocol).

Turns out I forgot that I’d tweaked iptables — the Linux firewall — but forgot to make the tweaks persistent.

Here’s the tweak:

sudo iptables -t nat -A POSTROUTING -s 192.168.5.0/24 -o eth0 -j SNAT --to-source 192.168.1.5

This ties the tunnel subnet, the one administered by OpenVPN which your remote clients live on (192.168.5.x, in my case) to the Pi (192.168.1.5, in my case) so that they can access the LAN.

Making this tweak persistent across reboots can be challenging because the approach to use will depend upon how your system currently persists firewall settings.

In my case I was able to use the netfilter-persistent package in Raspbian. Provided you have netfilter-persistent installed and running as a service all you need to do is:

> sudo netfilter-persistent save

If you want to see what’s being persisted you can look in /etc/iptables.

return to table of contents

Step 7: Install OpenVPN Client and Import Configuration File

Configuring stuff on iOS can be a pain — it’s not really intended to support complex configuration processes — which would, I suspect, be a real problem in configuring an iOS VPN client because VPNs are inherently relatively complicated.

The OpenVPN iOS client makes the whole process relatively simple, because all you do is import the client configuration file piVPN created for you on the Raspberry Pi into the ios OpenVPN client.

The hardest part of that was getting the client configuration file someplace where iOS could find it. iOS is heaviliy sandboxed so you can’t just upload a file to it. I solved that problem by uploading the client configuration file to my Microsoft OneDrive cloud account and then navigating to it within iOS via the iOS OneDrive app. Could’ve been more intuitive, but still not all that involved.

return to table of contents

Step 8: If Things Go Wrong

If everything has worked out correctly you should now be able to VPN into your LAN from a remote client. But don’t be surprised if you run into problems — VPNs are inherently pretty complex, so it’s hardly uncommon to run into configuration problems.

There are two places where you can start debugging what’s going wrong, one on the server and one on the client.

OpenVPN logs what it’s doing at /var/log/openvpn.log (at least in the default setup). The verbosity is controlled by “verb x”, line 36 in the OpenVPN server.conf file. When things are working you’ll see something like this:

May 28 15:02:24 mycroft ovpn-server[352]: OpenVPN 2.4.0 arm-unknown-linux-gnueabihf [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [M$
May 28 15:02:24 mycroft ovpn-server[352]: library versions: OpenSSL 1.0.2r  26 Feb 2019, LZO 2.08
May 28 15:02:25 mycroft ovpn-server[404]: Outgoing Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
May 28 15:02:25 mycroft ovpn-server[404]: Outgoing Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authent$
May 28 15:02:25 mycroft ovpn-server[404]: Incoming Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
May 28 15:02:25 mycroft ovpn-server[404]: Incoming Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authent$
May 28 15:02:25 mycroft ovpn-server[404]: TUN/TAP device tun0 opened
May 28 15:02:25 mycroft ovpn-server[404]: TUN/TAP TX queue length set to 100
May 28 15:02:25 mycroft ovpn-server[404]: do_ifconfig, tt->did_ifconfig_ipv6_setup=0
May 28 15:02:25 mycroft ovpn-server[404]: /sbin/ip link set dev tun0 up mtu 1500
May 28 15:02:25 mycroft ovpn-server[404]: /sbin/ip addr add dev tun0 192.168.5.1/24 broadcast 192.168.5.255
May 28 15:02:25 mycroft ovpn-server[404]: Could not determine IPv4/IPv6 protocol. Using AF_INET
May 28 15:02:25 mycroft ovpn-server[404]: Socket Buffers: R=[163840->163840] S=[163840->163840]
May 28 15:02:25 mycroft ovpn-server[404]: UDPv4 link local (bound): [AF_INET][undef]:37639
May 28 15:02:25 mycroft ovpn-server[404]: UDPv4 link remote: [AF_UNSPEC]
May 28 15:02:25 mycroft ovpn-server[404]: GID set to nogroup
May 28 15:02:25 mycroft ovpn-server[404]: UID set to nobody
May 28 15:02:25 mycroft ovpn-server[404]: MULTI: multi_init called, r=256 v=256
May 28 15:02:25 mycroft ovpn-server[404]: IFCONFIG POOL: base=192.168.5.2 size=252, ipv6=0
May 28 15:02:25 mycroft ovpn-server[404]: Initialization Sequence Completed
May 28 15:18:49 mycroft ovpn-server[404]: 166.170.43.18:23810 TLS: Initial packet from [AF_INET]166.170.43.18:23810, sid=caf298d2 2$
May 28 15:18:49 mycroft ovpn-server[404]: 166.170.43.18:23810 VERIFY OK: depth=1, CN=ChangeMe
May 28 15:18:49 mycroft ovpn-server[404]: 166.170.43.18:23810 Validating certificate key usage
May 28 15:18:49 mycroft ovpn-server[404]: 166.170.43.18:23810 ++ Certificate has key usage  0080, expects 0080
May 28 15:18:49 mycroft ovpn-server[404]: 166.170.43.18:23810 VERIFY KU OK
May 28 15:18:49 mycroft ovpn-server[404]: 166.170.43.18:23810 Validating certificate extended key usage
May 28 15:18:49 mycroft ovpn-server[404]: 166.170.43.18:23810 ++ Certificate has EKU (str) TLS Web Client Authentication, expects T$
May 28 15:18:49 mycroft ovpn-server[404]: 166.170.43.18:23810 VERIFY EKU OK
May 28 15:18:49 mycroft ovpn-server[404]: 166.170.43.18:23810 VERIFY OK: depth=0, CN=DiegoVPN
May 28 15:18:50 mycroft ovpn-server[404]: 166.170.43.18:23810 peer info: IV_GUI_VER=net.openvpn.connect.ios_3.0.2-894
May 28 15:18:50 mycroft ovpn-server[404]: 166.170.43.18:23810 peer info: IV_VER=3.2
May 28 15:18:50 mycroft ovpn-server[404]: 166.170.43.18:23810 peer info: IV_PLAT=ios
May 28 15:18:50 mycroft ovpn-server[404]: 166.170.43.18:23810 peer info: IV_NCP=2
May 28 15:18:50 mycroft ovpn-server[404]: 166.170.43.18:23810 peer info: IV_TCPNL=1
May 28 15:18:50 mycroft ovpn-server[404]: 166.170.43.18:23810 peer info: IV_PROTO=2

If there are problems you should find error messages. I was pretty impressed by the OpenVPN error messages; they were reasonably understandable by an ignorant layman (i.e., me). For example, at one point I had specified different encryption algorithms in the server and client configuration files. There was an error message to that effect in the log.

On the client side, at least for the iOS client, OpenVPN also provides a pretty comprehensive log of its activities:

2019-05-28 15:18:49 ----- OpenVPN Start ----- OpenVPN core 3.2 ios arm64 64-bit PT_PROXY built on Oct  3 2018 06:35:04
2019-05-28 15:18:49 Frame=512/2048/512 mssfix-ctrl=1250
2019-05-28 15:18:49 UNUSED OPTIONS
4 [resolv-retry] [infinite]
5 [nobind]
6 [persist-key]
7 [persist-tun]
10 [verify-x509-name] [server_X4O6Por7FYITsLun] [name]
13 [auth-nocache]
14 [verb] [3] 
2019-05-28 15:18:49 EVENT: RESOLVE
2019-05-28 15:18:49 Contacting [99.152.116.175]:37639/UDP via UDP
2019-05-28 15:18:49 EVENT: WAIT
2019-05-28 15:18:49 Connecting to [moose.zapto.org]:37639 (99.152.116.175) via UDPv4
2019-05-28 15:18:49 EVENT: CONNECTING
2019-05-28 15:18:49 Tunnel Options:V4,dev-type tun,link-mtu 1569,tun-mtu 1500,proto UDPv4,cipher AES-256-CBC,auth SHA256,keysize 256,key-method 2,tls-client
2019-05-28 15:18:49 Creds: UsernameEmpty/PasswordEmpty
2019-05-28 15:18:49 Peer Info:
IV_GUI_VER=net.openvpn.connect.ios 3.0.2-894
IV_VER=3.2
IV_PLAT=ios
IV_NCP=2
IV_TCPNL=1
IV_PROTO=2
IV_AUTO_SESS=1
2019-05-28 15:18:49 VERIFY OK : depth=1
cert. version    : 3
serial number    : E6:39:B2:CA:85:93:88:6C
issuer name      : CN=ChangeMe
subject name      : CN=ChangeMe
issued  on        : 2019-04-18 21:59:22
expires on        : 2029-04-15 21:59:22
signed using      : ECDSA with SHA256
EC key size      : 256 bits
basic constraints : CA=true
key usage        : Key Cert Sign, CRL Sign
2019-05-28 15:18:49 VERIFY OK : depth=0
cert. version    : 3
serial number    : E1:90:6D:75:18:89:58:99:1F:2A:82:4D:E0:D8:95:31
issuer name      : CN=ChangeMe
subject name      : CN=server_X4O6Por7FYITsLun
issued  on        : 2019-04-18 21:59:22
expires on        : 2022-04-02 21:59:22
signed using      : ECDSA with SHA256
EC key size      : 256 bits
basic constraints : CA=false
subject alt name  : server_X4O6Por7FYITsLun
key usage        : Digital Signature, Key Encipherment
ext key usage    : TLS Web Server Authentication
2019-05-28 15:18:50 SSL Handshake: TLSv1.2/TLS-ECDHE-ECDSA-WITH-AES-256-GCM-SHA384
2019-05-28 15:18:50 Session is ACTIVE
2019-05-28 15:18:50 EVENT: GET_CONFIG
2019-05-28 15:18:50 Sending PUSH_REQUEST to server...
2019-05-28 15:18:50 OPTIONS:
0 [route] [192.168.1.0] [255.255.255.0] [192.168.1.254]
1 [dhcp-option] [DNS] [192.168.1.5]
2 [dhcp-option] [DOMAIN] [localnet]
3 [dhcp-option] [SEARCH] [localnet]
4 [block-outside-dns]
5 [redirect-gateway] [def1]
6 [route-gateway] [192.168.5.1]
7 [topology] [subnet]
8 [ping] [1800]
9 [ping-restart] [3600]
10 [ifconfig] [192.168.5.2] [255.255.255.0]
11 [peer-id] [0]
12 [cipher] [AES-256-GCM] 
2019-05-28 15:18:50 PROTOCOL OPTIONS:
  cipher: AES-256-GCM
  digest: SHA256
  compress: NONE
  peer ID: 0
2019-05-28 15:18:50 EVENT: ASSIGN_IP
2019-05-28 15:18:50 NIP: preparing TUN network settings
2019-05-28 15:18:50 NIP: init TUN network settings with endpoint: 99.152.116.175
2019-05-28 15:18:50 NIP: adding IPv4 address to network settings 192.168.5.2/255.255.255.0
2019-05-28 15:18:50 NIP: adding (included) IPv4 route 192.168.5.0/24
2019-05-28 15:18:50 NIP: redirecting all IPv4 traffic to TUN interface
2019-05-28 15:18:50 NIP: adding DNS 192.168.1.5
2019-05-28 15:18:50 NIP: adding match domain localnet
2019-05-28 15:18:50 Connected via NetworkExtensionTUN
2019-05-28 15:18:50 EVENT: CONNECTED moose.zapto.org:37639 (99.152.116.175) via /UDPv4 on NetworkExtensionTUN/192.168.5.2/ gw=[/]
2019-05-28 15:20:33 EVENT: DISCONNECTED
2019-05-28 15:20:33 Raw stats on disconnect:
  BYTES_IN : 4942901
  BYTES_OUT : 285019
  PACKETS_IN : 3898
  PACKETS_OUT : 3360
  TUN_BYTES_IN : 202696
  TUN_BYTES_OUT : 4858811
  TUN_PACKETS_IN : 3351
  TUN_PACKETS_OUT : 3890
2019-05-28 15:20:33 Performance stats on disconnect:
  CPU usage (microseconds): 1541386
  Tunnel compression ratio (uplink): 1.40614
  Tunnel compression ratio (downlink): 1.01731
  Network bytes per CPU second: 3391700
  Tunnel bytes per CPU second: 3283737

Again, when there are problems, the logs in my experience provide a lot of hints about what’s going wrong.

return to table of contents

Enjoy VPNing!!!

Dear Apple,

My wife doesn’t use her iPad very often, so today we learned about an interesting aspect of your product’s user experience.

Apparently, if you allow the battery to discharge too much — which can happen when you don’t use the device very often and forget to plug it in — it can slip into a state where it needs to be either updated or restored via iTunes (the two options are similar, except the latter erases whatever you might’ve installed onto the device).

Even more interestingly, apparently there are situations where even restore will fail. Despite the device being recharged, and being accessed by iTunes via a USB connection, from an Apple Macintosh computer, no less. 100% Apple software and hardware, all the way (even including the USB cable).

I guess at this point my only options are to take it to an Apple Store to see if they can repair it, turn it into an admittedly attractive coaster, or consign it to the landfill.

Very Truly Yours,

A Less Than Ecstatic Apple Customer

p.s. Being somewhat technically inclined, I would love to know how one designs a computer, in 2018, using solid state drives, so that it can do what my wife’s iPad did. It must’ve taken quite a bit of effort to figure out how to arrange things so it could fail this way!

p.p.s I see you’ve been studying Microsoft’s error message protocol, the one where you tell me something has gone wrong — in case I hadn’t noticed that — but don’t tell me anything about what might’ve caused it. Always nice to see major companies sharing skill sets!