VPN Adventures: Tunneling

Disclaimer: I know relatively little about networking under Linux, so don’t be surprised if some of what’s here is flat out wrong. Then again, that’s why Google is your friend…

If you read about my adventures getting my AT&T fiber service to operate continuously at near gigabit speed you’ll know that I tried hard to keep my Cisco RV-325 in the mix to handle VPN connections from iOS devices. It was quite a struggle! The RV-325 documentation presumes you know what you’re doing, which definitely doesn’t describe me when it comes to networking, let alone VPNs.

Actually, that observation could be applied to much of the networking and VPN documentation I’ve been reading online. It tends to presume you know what you’re doing, perhaps on the theory that no one would think to try setting up a VPN or diving into the bowels of networking unless they knew what they were doing. But fools rush in where angels fear to tread…

In fairness, a lot of VPN details are inherently pretty involved. Where else do you worry about what encryption protocol you’re using — it must be the same on both client and server — how much entropy you want to include in your keys, and just what is a Diffie Hellman file anyway and why should I want one?

In any event, here’s how I eventually got my VPN setup working so that I can access it from iOS devices.

Table of Contents

My Environment

  • the server runs on a Raspberry Pi 3B+
  • the underlying OS is the Raspberry Pi-tweak of Debian Stretch, with all the latest updates installed as of May, 2019
  • I don’t use the WiFi interface on the Pi; it’s hardwired into my LAN
  • My VPN must use the tunneling approach because I need to support iOS. Apple, in its infinite wisdom, has declined to support the other mode (tap).
  • I use my Pi as a DNS server for my LAN, via dnsmasq. Which is a really slick little package…except for one !#$!#@$I& “feature” that nearly caused me to have a stroke, not once, but twice.

return to table of contents

Tunnel vs Tap

VPNs can swing in (at least) two ways, tunnel and tap. A tunnel defines a separate IP subnet, with its own addresses separate from the LAN that the VPN server sits on, and adds remote clients into that new subnet.

For example, my LAN uses the 192.168.1.x/24 subnet, and my VPN offers addresses in the 192.168.5.x/24 space. There are a few tweaks to let the two subnets interact with each other, but while not obvious, they’re fairly straightforward (I’ll describe them below).

A tap hands out IP addresses within the LAN’s subnet to the remote VPN clients, so those remote clients are automatically part of the LAN and can interact with all parts of it. A tap VPN has to have a reserved block of addresses to manage, which your DHCP setup should avoid using.

A tunnel VPN works by creating a virtual network adapter (typically called tunX, in my case tun0) that acts as a portal between the server and its remote clients. Think of it as being like a virtual ethernet switch into which all the remote clients are plugged.

return to table of contents

Step 1: Configure a Dynamic DNS Referrer

This may be optional for you, if your broadband connection comes with an IP address that never changes. My understanding is that cable connections do change addresses occasionally. AT&T fiber connections apparently change very rarely. But they can change, and if yours does, remote clients won’t be able to find your VPN server.

There are a number of services that enable you to store a DNS record that maps your broadband IP to a human-readable name, and keep the reference updated if and when you broadband provider changes the IP address on you. That’s done via a daemon that runs on the Pi and checks your external IP address periodically, updating what’s on file for you as necessary.

I’ve used DynDns.com for several years with no problem. I’m currently trying out NoIP.com because the DynDNS client that runs on the Pi, ddclient, does not encrypt the DynDNS account password in its config file. Not a big deal unless someone breaks into your Pi, but not a great practice. The NoIP.com client encrypts its config file on the Pi. NoIP is also about half the cost of DynDNS (NoIP also offers a free service…but you have to put up with ads being injected into your redirect stream, which might not be a big deal for this kind of VPN setup, but still disturbed me).

return to table of contents

Step 2: Install OpenVPN via PiVPN

I’m sure there are other open source VPN packages out there, but OpenVPN has been around for quite a while, is well supported, and offers its own iOS client-side package for interacting with the server. The logging abilities of which were really important in getting everything to work. There’s nothing like a message describing what step of the initialization process went wrong on the client side to help speed up problem solving!

You can install OpenVPN in a lot of ways, but doing it via PiVPN makes key/certificate generation and initial configuration much easier, IMHO. Follow the instructions, fill in the blanks and, while you’ll still have some work to do, you’ll never have to figure out just what a Diffie Hellman file is.

Remember to include the “referrer name” you set up in step 1 instead of your external broadband IP address, unless you’re just willing to hard-wire that IP address in (and adjust it yourself if your broadband provider changes the IP address on you).

return to table of contents

Step 3: Tweak dnsmasq Configuration

Only one change is necessary to your dnsmasq configuration, but it’s important:

# only listen on this interface

You have to tell dnsmasq to listen on that new, virtual tun0 interface in addition to whatever “normal” interface you have it monitor (in my case, eth0). If you don’t, your VPN clients won’t be able to resolve names and your LAN will be invisible to you. Unless you’re really good at guessing DHCP-assigned IP addresses or you stick to only static IP addresses. Figuring this out cost me an embarrassing number of hours.

BTW, dnsmasq has a bizarre (to me) design “feature” that can be coronary-inducing: like many linux daemons, you can configure it either through a single file (typically /etc/dnsmasq.conf) or through multiple files in a directory (typically /etc/dnsmasq.d). That part’s perfectly normal.

What I keep forgetting is that dnsmasq, by default, attempts to parse almost every single file in the configuration directory (there are default exceptions, but they’re pretty odd, IMHO). So, if you’re used to “only the files ending in .conf” (or whatever) getting parsed, you’re in for a rude awakening if you leave other files in that directory.

Which I did at one point, and which made the internet invisible to all the devices on my LAN (if your DNS service is down there ain’t no resolution happening and, yes, shame on me for not checking to see if dnsmasq was down — but when you’re focused on configuring network stuff it’s easy to get brain-channeled into not remembering to check the obvious).

Interestingly, dnsmasq does not recursively scan its configuration directory. So you can stick alternate and backup configuration files in a subdirectory of the configuration directory with no problem. Which is what I did after my head stopped exploding from trying to figure out why “reverting everything to the way it was” left me with a broken LAN (I have a habit of saving different versions of config files with different suffixes in config directories, like localnet.conf.tap; that’s a no-no with dnsmasq).

return to table of contents

Step 4: Tweak OpenVPN’s server.conf file

The server.conf file is typically located in the /etc/openvpn directory. After tweaking it, here’s what mine looked like:

dev tun
proto udp 
port ***redacted*** 
ca /etc/openvpn/easy-rsa/pki/ca.crt 
cert /etc/openvpn/easy-rsa/pki/issued/server_X4O6Por7FYITsLun.crt 
key /etc/openvpn/easy-rsa/pki/private/server_X4O6Por7FYITsLun.key 
dh none 
topology subnet 
# Set your primary domain name server address for clients 
push "route" 
push "dhcp-option DNS" 
push "dhcp-option DOMAIN localnet" 
push "dhcp-option SEARCH localnet" 
# Prevent DNS leaks on Windows 
push "block-outside-dns" 
# Override the Client default gateway by using and 
# rather than This has the benefit of 
# overriding but not wiping out the original default gateway. 
push "redirect-gateway def1" 
keepalive 1800 3600 
remote-cert-tls client 
tls-version-min 1.2 
tls-crypt /etc/openvpn/easy-rsa/pki/ta.key 
cipher AES-256-CBC 
auth SHA256 
user nobody 
group nogroup 
crl-verify /etc/openvpn/crl.pem 
status /var/log/openvpn-status.log 20 
status-version 3 
verb 3 
#DuplicateCNs allow access control on a less-granular, per user basis. 
#Remove # if you will manage access by user instead of device. 
#duplicate-cn # Generated for use by PiVPN.io

The lines in this configuration file I ended up tweaking were the following:

  • Line 1: this is what tells OpenVPN to create a tunnel rather than a tap
  • Line 2: you could use TCP here, but it’s more common apparently to use UDP (you may have to ensure your firewall — that’s on the AT&T-supplied BGW210 in my case — forwards UDP connections to the Pi)
  • Line 3: by default OpenVPN works through port 1194 (I think). You should probably change that to provide a little bit more protection from hackers.
  • Line 9: this defines the subnet that the remote clients will be in
  • Line 11: this establishes a route between the VPN subnet (192.168.5.x in my case) and the LAN (192.168.1.x in my case). Without this the VPN clients won’t see the devices on the LAN.
  • Line 12: defines the DNS server that the VPN clients will be using (this points at the Pi, in my case, because it is home to dnsmasq)
  • Lines 13 – 14 and 16: not sure what these do, but I read something the recommended including them if you have Windows VPN clients, which I do.

The rest of the file is what came out of the piVPN setup and configuration process. I suspect there are steps you can take to harden your VPN beyond what the PiVPN setup does, but I haven’t had time to experiment with tweaking the OpenVPN settings to find out.

return to table of contents

Step 5: Configure Your Firewall to Forward Packets to the Pi

This is an easy one to forget, since it generally doesn’t involve the Pi (if you’re using a Pi as a LAN firewall I’m impressed). You need to configure the firewall protecting your LAN to forward the proper packets to the Pi. If you don’t, your remote clients won’t be able to connect with your OpenVPN server.

The proper packets are defined by the nature of the protocol they use (TCP or UDP) and the port number they’re trying to reach. You have to make sure you define the firewall forwarding rule for the protocol and port your OpenVPN configuration expects (in my case that was UDP and…well, the port number is a secret).

return to table of contents

Step 6: Install OpenVPN Client and Import Configuration File

Configuring stuff on iOS can be a pain — it’s not really intended to support complex configuration processes — which would, I suspect, be a real problem in configuring an iOS VPN client because VPNs are inherently relatively complicated.

The OpenVPN iOS client makes the whole process relatively simple, because all you do is import the client configuration file piVPN created for you on the Raspberry Pi into the ios OpenVPN client.

The hardest part of that was getting the client configuration file someplace where iOS could find it. iOS is heaviliy sandboxed so you can’t just upload a file to it. I solved that problem by uploading the client configuration file to my Microsoft OneDrive cloud account and then navigating to it within iOS via the iOS OneDrive app. Could’ve been more intuitive, but still not all that involved.

return to table of contents

Step 7: If Things Go Wrong

If everything has worked out correctly you should now be able to VPN into your LAN from a remote client. But don’t be surprised if you run into problems — VPNs are inherently pretty complex, so it’s hardly uncommon to run into configuration problems.

There are two places where you can start debugging what’s going wrong, one on the server and one on the client.

OpenVPN logs what it’s doing at /var/log/openvpn.log (at least in the default setup). The verbosity is controlled by “verb x”, line 36 in the OpenVPN server.conf file. When things are working you’ll see something like this:

May 28 15:02:24 mycroft ovpn-server[352]: OpenVPN 2.4.0 arm-unknown-linux-gnueabihf [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [M$
May 28 15:02:24 mycroft ovpn-server[352]: library versions: OpenSSL 1.0.2r  26 Feb 2019, LZO 2.08
May 28 15:02:25 mycroft ovpn-server[404]: Outgoing Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
May 28 15:02:25 mycroft ovpn-server[404]: Outgoing Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authent$
May 28 15:02:25 mycroft ovpn-server[404]: Incoming Control Channel Encryption: Cipher 'AES-256-CTR' initialized with 256 bit key
May 28 15:02:25 mycroft ovpn-server[404]: Incoming Control Channel Encryption: Using 256 bit message hash 'SHA256' for HMAC authent$
May 28 15:02:25 mycroft ovpn-server[404]: TUN/TAP device tun0 opened
May 28 15:02:25 mycroft ovpn-server[404]: TUN/TAP TX queue length set to 100
May 28 15:02:25 mycroft ovpn-server[404]: do_ifconfig, tt->did_ifconfig_ipv6_setup=0
May 28 15:02:25 mycroft ovpn-server[404]: /sbin/ip link set dev tun0 up mtu 1500
May 28 15:02:25 mycroft ovpn-server[404]: /sbin/ip addr add dev tun0 broadcast
May 28 15:02:25 mycroft ovpn-server[404]: Could not determine IPv4/IPv6 protocol. Using AF_INET
May 28 15:02:25 mycroft ovpn-server[404]: Socket Buffers: R=[163840->163840] S=[163840->163840]
May 28 15:02:25 mycroft ovpn-server[404]: UDPv4 link local (bound): [AF_INET][undef]:37639
May 28 15:02:25 mycroft ovpn-server[404]: UDPv4 link remote: [AF_UNSPEC]
May 28 15:02:25 mycroft ovpn-server[404]: GID set to nogroup
May 28 15:02:25 mycroft ovpn-server[404]: UID set to nobody
May 28 15:02:25 mycroft ovpn-server[404]: MULTI: multi_init called, r=256 v=256
May 28 15:02:25 mycroft ovpn-server[404]: IFCONFIG POOL: base= size=252, ipv6=0
May 28 15:02:25 mycroft ovpn-server[404]: Initialization Sequence Completed
May 28 15:18:49 mycroft ovpn-server[404]: TLS: Initial packet from [AF_INET], sid=caf298d2 2$
May 28 15:18:49 mycroft ovpn-server[404]: VERIFY OK: depth=1, CN=ChangeMe
May 28 15:18:49 mycroft ovpn-server[404]: Validating certificate key usage
May 28 15:18:49 mycroft ovpn-server[404]: ++ Certificate has key usage  0080, expects 0080
May 28 15:18:49 mycroft ovpn-server[404]: VERIFY KU OK
May 28 15:18:49 mycroft ovpn-server[404]: Validating certificate extended key usage
May 28 15:18:49 mycroft ovpn-server[404]: ++ Certificate has EKU (str) TLS Web Client Authentication, expects T$
May 28 15:18:49 mycroft ovpn-server[404]: VERIFY EKU OK
May 28 15:18:49 mycroft ovpn-server[404]: VERIFY OK: depth=0, CN=DiegoVPN
May 28 15:18:50 mycroft ovpn-server[404]: peer info: IV_GUI_VER=net.openvpn.connect.ios_3.0.2-894
May 28 15:18:50 mycroft ovpn-server[404]: peer info: IV_VER=3.2
May 28 15:18:50 mycroft ovpn-server[404]: peer info: IV_PLAT=ios
May 28 15:18:50 mycroft ovpn-server[404]: peer info: IV_NCP=2
May 28 15:18:50 mycroft ovpn-server[404]: peer info: IV_TCPNL=1
May 28 15:18:50 mycroft ovpn-server[404]: peer info: IV_PROTO=2

If there are problems you should find error messages. I was pretty impressed by the OpenVPN error messages; they were reasonably understandable by an ignorant layman (i.e., me). For example, at one point I had specified different encryption algorithms in the server and client configuration files. There was an error message to that effect in the log.

On the client side, at least for the iOS client, OpenVPN also provides a pretty comprehensive log of its activities:

2019-05-28 15:18:49 ----- OpenVPN Start ----- OpenVPN core 3.2 ios arm64 64-bit PT_PROXY built on Oct  3 2018 06:35:04
2019-05-28 15:18:49 Frame=512/2048/512 mssfix-ctrl=1250
2019-05-28 15:18:49 UNUSED OPTIONS
4 [resolv-retry] [infinite]
5 [nobind]
6 [persist-key]
7 [persist-tun]
10 [verify-x509-name] [server_X4O6Por7FYITsLun] [name]
13 [auth-nocache]
14 [verb] [3] 
2019-05-28 15:18:49 EVENT: RESOLVE
2019-05-28 15:18:49 Contacting []:37639/UDP via UDP
2019-05-28 15:18:49 EVENT: WAIT
2019-05-28 15:18:49 Connecting to [moose.zapto.org]:37639 ( via UDPv4
2019-05-28 15:18:49 EVENT: CONNECTING
2019-05-28 15:18:49 Tunnel Options:V4,dev-type tun,link-mtu 1569,tun-mtu 1500,proto UDPv4,cipher AES-256-CBC,auth SHA256,keysize 256,key-method 2,tls-client
2019-05-28 15:18:49 Creds: UsernameEmpty/PasswordEmpty
2019-05-28 15:18:49 Peer Info:
IV_GUI_VER=net.openvpn.connect.ios 3.0.2-894
2019-05-28 15:18:49 VERIFY OK : depth=1
cert. version    : 3
serial number    : E6:39:B2:CA:85:93:88:6C
issuer name      : CN=ChangeMe
subject name      : CN=ChangeMe
issued  on        : 2019-04-18 21:59:22
expires on        : 2029-04-15 21:59:22
signed using      : ECDSA with SHA256
EC key size      : 256 bits
basic constraints : CA=true
key usage        : Key Cert Sign, CRL Sign
2019-05-28 15:18:49 VERIFY OK : depth=0
cert. version    : 3
serial number    : E1:90:6D:75:18:89:58:99:1F:2A:82:4D:E0:D8:95:31
issuer name      : CN=ChangeMe
subject name      : CN=server_X4O6Por7FYITsLun
issued  on        : 2019-04-18 21:59:22
expires on        : 2022-04-02 21:59:22
signed using      : ECDSA with SHA256
EC key size      : 256 bits
basic constraints : CA=false
subject alt name  : server_X4O6Por7FYITsLun
key usage        : Digital Signature, Key Encipherment
ext key usage    : TLS Web Server Authentication
2019-05-28 15:18:50 SSL Handshake: TLSv1.2/TLS-ECDHE-ECDSA-WITH-AES-256-GCM-SHA384
2019-05-28 15:18:50 Session is ACTIVE
2019-05-28 15:18:50 EVENT: GET_CONFIG
2019-05-28 15:18:50 Sending PUSH_REQUEST to server...
2019-05-28 15:18:50 OPTIONS:
0 [route] [] [] []
1 [dhcp-option] [DNS] []
2 [dhcp-option] [DOMAIN] [localnet]
3 [dhcp-option] [SEARCH] [localnet]
4 [block-outside-dns]
5 [redirect-gateway] [def1]
6 [route-gateway] []
7 [topology] [subnet]
8 [ping] [1800]
9 [ping-restart] [3600]
10 [ifconfig] [] []
11 [peer-id] [0]
12 [cipher] [AES-256-GCM] 
2019-05-28 15:18:50 PROTOCOL OPTIONS:
  cipher: AES-256-GCM
  digest: SHA256
  compress: NONE
  peer ID: 0
2019-05-28 15:18:50 EVENT: ASSIGN_IP
2019-05-28 15:18:50 NIP: preparing TUN network settings
2019-05-28 15:18:50 NIP: init TUN network settings with endpoint:
2019-05-28 15:18:50 NIP: adding IPv4 address to network settings
2019-05-28 15:18:50 NIP: adding (included) IPv4 route
2019-05-28 15:18:50 NIP: redirecting all IPv4 traffic to TUN interface
2019-05-28 15:18:50 NIP: adding DNS
2019-05-28 15:18:50 NIP: adding match domain localnet
2019-05-28 15:18:50 Connected via NetworkExtensionTUN
2019-05-28 15:18:50 EVENT: CONNECTED moose.zapto.org:37639 ( via /UDPv4 on NetworkExtensionTUN/ gw=[/]
2019-05-28 15:20:33 EVENT: DISCONNECTED
2019-05-28 15:20:33 Raw stats on disconnect:
  BYTES_IN : 4942901
  BYTES_OUT : 285019
  PACKETS_IN : 3898
  PACKETS_OUT : 3360
  TUN_BYTES_IN : 202696
  TUN_BYTES_OUT : 4858811
2019-05-28 15:20:33 Performance stats on disconnect:
  CPU usage (microseconds): 1541386
  Tunnel compression ratio (uplink): 1.40614
  Tunnel compression ratio (downlink): 1.01731
  Network bytes per CPU second: 3391700
  Tunnel bytes per CPU second: 3283737

Again, when there are problems, the logs in my experience provide a lot of hints about what’s going wrong.

return to table of contents

Enjoy VPNing!!!

Getting AT&T Fiber to Play Nicely


  • Don’t run router/firewall devices in series
  • Don’t let the AT&T Fiber installer fob you off with an Arris 5268ac or equivalent
  • Do insist on getting a BGW-210. AT&T has them, but they’re more newer and more expensive and, for whatever reason, the installers tend net to offer them.
  • If you can’t get a BGW-210 from AT&T, buy your own online. They’re pretty simple to install and configure. Plus, they’re much, much more configurable than the other Arris stuff — they even have extensive help documentation built in! — and are also much, much faster.

Some months ago I switched from Comcast’s cable-based data system to AT&T fiber. At least where I live (San Carlos, CA) it was a bit cheaper and much faster. It was wonderful getting downstream — and upstream! — data rates that were consistently above 900 megabit.

Only there was one problem. Those speeds tended not to last long, degrading, without warning to somewhere around 50 megabit. Which is below what I had been getting with Comcast.

For a while I could work around the problem by rebooting the AT&T-supplied modem/router. But the intervals at which I’d have to do that kept getting shorter and shorter and, besides, it wasn’t solving the problem, it was merely resetting things until the problem appeared again.

Actually, AT&T provides two different devices when you buy the fiber package: something that probably is the actual modem, in that it converts between the optical fiber and gigabit ethernet, and a device that I suspect is a fairly typical firewall/router with the capability of handling the output from the actual modem and splitting off VOIP for your phones. It was the router/firewall/whatever I kept rebooting, not the modem itself (which doesn’t appear to even have a reboot button).

It finally dawned on me that my problem might have been due to the fact that the AT&T router fed into a Cisco RV-325 router/firewall/VPN server. The Cisco device played an important rule during my Comcast era because the Comcast modem really was more of a modem, and didn’t contain a firewall. Putting two router/firewall devices in series like that is silly, and prone to causing problems. Even when I put the AT&T router/firewall/whatever device into something akin to “pass thru” mode (i.e., a mode which supposedly makes it act more like just a modem).

I didn’t want to abandon the Cisco device because I had finally figured out how to get its VPN capability to work with my iOS devices. That was both cool and useful, as it enabled me to fire up and remotely run my home computers when we were traveling. The AT&T hardware, an Arris 5268ac, doesn’t offer many configuration options — it appears to be intended to be used by customers with simple needs — and is pretty slow to boot.

I struggled to get the 5268ac working but finally gave up on the advice of some helpful fellow users on the AT&T Fiber Equipment forum.

I tried to get help from AT&T tech support, but it was generally pretty useless. Most of the techs denied there was any way to set up the VPN I wanted, and the one tech who said he thought it was possible arranged for me to get a BGW-210 replacement for the 5268ac…but the AT&T fulfillment department sent me a different Arris device instead and it was no better than the 5268ac.

So I bought my own BGW-210 from Amazon and installed it. The only two parts I was nervous about turned out not to be problems at all: connecting it to the modem was as simple as plugging in the special ONT cable that came out of the modem to the BGW-210, and interfacing my home phones to the VOIP outlet, even though it was designed as a “two line” outlet — the 5268ac has dual “single line” outlets — was just a matter of plugging in my “single line” plug into the “two line” outlet.

One minor PITA that may bite some people: none of the AT&T equipment I used allowed you to set up LANs in the 10.x.x.x space (there’s a reason for the limitation, which I’m not currently remembering, but you can research it online). That forced me to reconfigure my LAN into 192.168.1.x, causing some degree of hair-pulling, but it all worked out in the end.

BTW, if you are setting up a LAN that you want to VPN into, you should think about using something other than 192.168.0.x or 192.168.1.x. Those are very commonly used, and apparently problems can occur if you VPN out of a 192.168.0.x network into a 192.168.0.x network. That’s not a risk I face, but I thought I’d pass it along.

As for my adventures setting up a VPN…that’s another story.

6 Trillion Bytes in the Palm of Your Hand… Almost

I had reason to dispose of a couple of hard drives, but I didn’t want to throw them out with the data they contained. Unfortunately, they had died in such a way that my PC couldn’t recognize them on boot. Which meant I couldn’t use disk wiping software on them. I thought I was going to have to either take the risk of leaving the data on them, or pay a service to dispose of them catastrophically.

But then I had an idea.

Since I didn’t care what condition they ended up in, why not just disassemble them? And wreak havoc on the platters they contain?

So, behold, six trillion bytes of data neatly arranged on my dining room table:

By the way, taking them apart was quick and easy, provided you have the right tools (Torx, I think; the tips look like a Star of David). Only you have to find and undo the One Hidden Screw. Which I didn’t, on my first attempt…but I quickly realized it was there when I bent the metal cover:

Here’s where the One Hidden Screw is, at least on Seagate drives:

Seagate drives are a great investment if you like to disassemble hard disks. They fail relatively quickly, and in my experience generally fairly catastrophically.

After pulling the platters out I ran a magnet over both sides, and then, just because, I heated them up with a blowtorch. Not sure if I managed to reach the material’s Curie temperature, but at least I got to use a blowtorch for the first time in years.

Buddy Can You Spare a Few Bytes?

Another day in the Maelstrom of Badly Designed Software, this time involving not just an operating system function but also boot firmware. Geez, if you can’t trust your boot firmware, what can you trust?

This all began when the server which runs our DVR software (Sage; a great open source product — check it out at the community forum) began acting rather oddly. Truth to tell, this had been going on for months, but since we’ve switched about 90% of our TV watching to streaming services it was low on the priority list of things to fix. But since I had the great idea of saving my back and knees by relocating the server out of the hall closet to a little nook upstairs, I thought I might as well figure out what was wrong.

That ended up being a huge time sink and hair-puller. As my dad always said, it’s the five minute jobs that take two hours. Unfortunately, with computers the ratio is often 100 or more to 1, not 24.

I thought the problem was that one of the hard drives comprising a Windows Storage Space array was failing. No problem; I’d had to replace a failing drive once before (I’m looking at you, Seagate — can you please up your quality control??) and while it’s time consuming, it’s pretty straightforward.

The basic concept behind Storage Spaces is that you assign a bunch of individual drives to a pool and Windows virtualizes them into One Big Honking Drive. With built-in redundancy and error-checking. Expandable at will. Replaceable at will. Or so I thought.

Turns out things don’t go so well when (a) more than one drive fails at the same time (really, thanx again, Seagate!!) and (b) you’ve used up all the SATA ports on your motherboard. Kinda hard to “just add another drive” when there’s nothing to hook it to, and if you can’t add another drive, Storage Spaces won’t let you gracefully degrade the pool (e.g., shift whatever’s on the failing drives to the good drives). So even though my pool had enough space available to hold all the data on its good drives, I was stuck. Gotta be able to add in order to remove. Bizarre.

The resulting confusion and hair-pulling ultimately lead to me copying what files I could out of the pool onto a new (Western Digital) drive hooked up to an add-in SATA card I installed. The net result was the total loss of my desktop system’s file history (the server also plays that role), various backups of other systems, and about 75% of our recorded TV shows. Fortunately that was Really Bad rather than Unbelievably Disastrous since, as I mentioned, we don’t use our DVR much anymore.

Because no Really Bad Computer Day is complete with just one set of problems, I also had to fight with the Gigabyte P55 USB3 motherboard powering the server. It turns out that if the boot process can “see” a hard drive, but can’t identify it, it just stalls. Without any message or beep code or alert of any kind. And either one of the built-in SATA ports is flaky or they have to be “consumed” in a particular order (e.g., master before slave on a given channel), so… It’s disturbing to plug drives in and have them work, only to plug the same drives in to different ports and have the system freeze. With no hint as to what’s wrong.

Now, space is admittedly at a premium for firmware, so it’s not like it can contain a robust error reporting system. OTOH, modern firmware does contain a lot of stuff, including a number of messages. Would it really have been so hard to include “Uh, drive seen but not recognized on SATA port X”? Besides being really helpful, not having such messages violates what I consider to be one of the most important rules of well-designed software: don’t leave the user hanging. Log something, somewhere — screen, log file, carrier pigeon, the location doesn’t matter (so long as it’s known).

There’s nothing worse than trying to figure out a problem with no information as to what it is. It forces you to go into trial and error mode, also known as Keep Moving Everything Around Until It Mysteriously Starts Working Again. Not a pleasant experience, and not one that anyone should have to experience…so long as the software is well-designed.

The morals of the story? A few:

  • If you use Windows Storage Spaces, always leave some unused hard drive ports available in your system.
  • Better yet, think really hard about using Windows Storage Spaces without a full-time IT staff (I’ve abandoned it based on this experience).
  • If your motherboard appears to freeze during the early stages of the boot process, consider that it may be having problems recognizing hard drives but is too ashamed to let you know that.

Dear Apple,

My wife doesn’t use her iPad very often, so today we learned about an interesting aspect of your product’s user experience.

Apparently, if you allow the battery to discharge too much — which can happen when you don’t use the device very often and forget to plug it in — it can slip into a state where it needs to be either updated or restored via iTunes (the two options are similar, except the latter erases whatever you might’ve installed onto the device).

Even more interestingly, apparently there are situations where even restore will fail. Despite the device being recharged, and being accessed by iTunes via a USB connection, from an Apple Macintosh computer, no less. 100% Apple software and hardware, all the way (even including the USB cable).

I guess at this point my only options are to take it to an Apple Store to see if they can repair it, turn it into an admittedly attractive coaster, or consign it to the landfill.

Very Truly Yours,

A Less Than Ecstatic Apple Customer

p.s. Being somewhat technically inclined, I would love to know how one designs a computer, in 2018, using solid state drives, so that it can do what my wife’s iPad did. It must’ve taken quite a bit of effort to figure out how to arrange things so it could fail this way!

p.p.s I see you’ve been studying Microsoft’s error message protocol, the one where you tell me something has gone wrong — in case I hadn’t noticed that — but don’t tell me anything about what might’ve caused it. Always nice to see major companies sharing skill sets!

Azure WebJobs, Hidden UTF-8 Parameters and Azure

So as to keep others from beating their heads bloody working through “problems which shouldn’t be problems”, here’s an important, but little known, fact about batch files used to launch Azure WebJobs…

They can’t contain the UTF-8 Byte Order Mark. Affectionately known as “UTF-8 BOM”. Which really ought to known as “UTF-8 BOOM”, considering what it does to Azure when it shows up in a command file.

Which it will, by default, because Visual Studio inserts them, invisibly to you, whenever you save a file you’ve created. Sigh.

A tipoff that something’s wrong is an Azure log message which complains about an unreadable command file line, echoing some bizarre looking characters as a prefix to what >>you<< think the line is. Those characters are Azure’s attempt to display the BOM. Sadly, Azure isn’t smart enough to identify and work around this problem itself (are you listening, Microsoft? It would be really, really easy to scan for the BOM, and simply ignore it; heck, even I could figure out how to do that).

The only way to avoid this is to “Save As” the file, and then work your way through the dialog box options — by selecting Save with Encoding from the Save button — and picking the an encoding which doesn’t have the BOOM, uh, BOM. I typically use “UTF-8 without byte order mark”…which is inconveniently located near the end of a very long list of possible choices. Who knew there were so many encodings in the world?

Fortunately, you only have to do this once, because Visual Studio will remember the choice for as long as the file’s around. Unfortunately, you have to do it for every file you want treated that way; there’s no global option, at least not in Visual Studio 2015.

Hold Down That Shift Key!

I’m posting this here mostly as a reminder to me how to work around a poorly-documented aspect of migrating iTunes to a new computer.

In the current instance, I’ve recently built a new Windows PC. But the hard drive where all my iTunes media is stored is a holdover from the previous computer — where it worked just fine in iTunes. Same drive letter, same path to the media library, same file ownership credentials, etc.

Yet when I fired up iTunes and pointed it at the media folder — by editing the preferences — it refused to recognize any of the songs, videos, podcasts, what have. It helpfully offered to download all the stuff I’d purchased from the iTunes Store over the years…but who the heck wants to download gigabytes of material when the exact same files are already present on the hard drive??? Certainly not me!

It turns out the solution is simple…but completely un-intuitive. You have to launch iTunes with the shift key held down (Windows; for the Mac OS I believe it’s the option key). Which will then bring up a dialog box asking you to locate the iTunes library file (or create a new one).

Now, so far as I know, the iTunes library file, by default, is always in the root of the media folder. At least, I’ve never seen it anywhere else.

So you’d think iTunes would be smart enough, when it senses that it’s not accessing an iTunes library file, to go looking in the root of the media folder. But no, it’s too stupid to do that. Instead, it just creates a new, empty iTunes library file, points itself at it, and declares “hey, you’ve got a lot of music up in the cloud; too bad none of it’s down here!”.

With all due respect to Apple — and I’ve told people for years that they’d have to pry my iPhone out of my cold, dead fingers to get me to relinquish it — this is nuts. Plus, it’s lousy software design. Defaults are a good thing, but they have to be smart defaults. It would hardly take any time at all for iTunes, on launch, to check the root of the iTunes media folder, see an iTunes library file, realize it wasn’t pointing at it, and ask “hey, is that the library file you want me to use?” It could even parse the iTunes library file, see it was pointing to a whole bunch of media files in the drive system…and realize that this iTunes library file was the real McCoy.

Are you listening, Apple?


Thanx, Greenbow! But Not You, Cisco and Microsoft…

I’ve owned a Cisco Small Business router, model RV-325, for several years, and it’s worked very well as a firewall/router. So well, in fact, that after setting it up I think I’ve only had to log into it’s user interface once or twice to check things, or update the firmware.

But it supports VPNs, and I recently had cause to figure out how to set it up to do so. And on that front, it fell flat on its face.

Why is it that people who write hardware manuals assume you already know how to do whatever it is you’re checking the manual to do? It’s really an odd presumption… and an all too common one.

VPNs by their nature — and I am not at all an expert on them, although I know a lot more today than I did five days ago — are complicated, with many options. But that just highlights another problem, this time with hardware user interfaces: if the goal is simple — “I want to be able to access my LAN remotely” — but the steps involved are potentially complex, you need to abstract the interface to the point where the configuration process itself is simple. Or at least provide the option to do so.

When your fire up Word for the first time, you get what looks like a blank sheet of paper and a cursor. And if you start typing, lo and behold, words start appearing on the screen! Even though you didn’t configure anything. You can get started without having to be a tech guru, even when you try to print what your typed (although in that case it helps if your IT staff have named the printers in such a way that you can figure out which one is near you).

I was very much helped in my quest by a company called Greenbow, which makes a Windows VPN client. Whose user interface is admittedly a little less straightforward than perhaps it could be. But which more than makes up for that by actually generating error messages which one can figure out, at least with three days worth of knowledge of VPNs. The fact that I had to pay for it is irrelevant; it’s worth the price, just for that increased capability.

As for the Windows 10 VPN client: it’s so abstracted that I never was able to figure out where to enter certain critical data needed to make a connection. Granted, the user interface is beautifully simple. But it doesn’t support the task.


Keeping It Alive

I posted this on nVidia’s support forum, but felt it worth perpetuating somewhere else.


I was greatly relieved to see how nVidia is doing such a fine job of keeping alive the beautiful experience of having display drivers crash in the middle of work. Frankly, before I bought my GeForce 210 — running under Windows 10 — it’d been more than a decade since I’d enjoyed the fun of losing work by having a video driver crash and take down my entire system. Now I get to enjoy the ride every other day!

I also really appreciate how the nVidia Control Panel, and the nVidia Experience app, always display error messages when they open up. My particular favorite is “nVidia not available, please try again later”. I view that as a wonderful commentary on the demonstrated quality of nVidia’s software.

By dint of great effort, and working through repeated error messages, I believe my drivers are all up to snuff (I’m currently at version 341.95). I know that Windows 10 is completely up to date, since that happens automatically.

Thanx, nVidia, for perpetuating a key part of the computing experience that I feared had been lost forever.

Into the Wayback Machine

My buddy Connell runs an election/campaign advisory firm. He has a database that he uses to track voter profiles, create outreach lists, and the like. To minimize annoying voters — never a good idea when you’re running a campaign — he needs to stay on top of whoever has already voted, so he can take them off the outreach lists (who has voted is a matter of public record; how you vote is not).

Unfortunately, his database/computer guy isn’t available, and Connell wasn’t sure how to import the voting data. Worse yet, he couldn’t find the manual for the software. Between the two of us we came up with a number of different ways to try and work around these problems…but none of them succeeded. So, as always, when subtlety doesn’t work, it was time to turn to brute force :).

[Read more…]