Buddy Can You Spare a Few Bytes?

Another day in the Maelstrom of Badly Designed Software, this time involving not just an operating system function but also boot firmware. Geez, if you can’t trust your boot firmware, what can you trust?

This all began when the server which runs our DVR software (Sage; a great open source product — check it out at the community forum) began acting rather oddly. Truth to tell, this had been going on for months, but since we’ve switched about 90% of our TV watching to streaming services it was low on the priority list of things to fix. But since I had the great idea of saving my back and knees by relocating the server out of the hall closet to a little nook upstairs, I thought I might as well figure out what was wrong.

That ended up being a huge time sink and hair-puller. As my dad always said, it’s the five minute jobs that take two hours. Unfortunately, with computers the ratio is often 100 or more to 1, not 24.

I thought the problem was that one of the hard drives comprising a Windows Storage Space array was failing. No problem; I’d had to replace a failing drive once before (I’m looking at you, Seagate — can you please up your quality control??) and while it’s time consuming, it’s pretty straightforward.

The basic concept behind Storage Spaces is that you assign a bunch of individual drives to a pool and Windows virtualizes them into One Big Honking Drive. With built-in redundancy and error-checking. Expandable at will. Replaceable at will. Or so I thought.

Turns out things don’t go so well when (a) more than one drive fails at the same time (really, thanx again, Seagate!!) and (b) you’ve used up all the SATA ports on your motherboard. Kinda hard to “just add another drive” when there’s nothing to hook it to, and if you can’t add another drive, Storage Spaces won’t let you gracefully degrade the pool (e.g., shift whatever’s on the failing drives to the good drives). So even though my pool had enough space available to hold all the data on its good drives, I was stuck. Gotta be able to add in order to remove. Bizarre.

The resulting confusion and hair-pulling ultimately lead to me copying what files I could out of the pool onto a new (Western Digital) drive hooked up to an add-in SATA card I installed. The net result was the total loss of my desktop system’s file history (the server also plays that role), various backups of other systems, and about 75% of our recorded TV shows. Fortunately that was Really Bad rather than Unbelievably Disastrous since, as I mentioned, we don’t use our DVR much anymore.

Because no Really Bad Computer Day is complete with just one set of problems, I also had to fight with the Gigabyte P55 USB3 motherboard powering the server. It turns out that if the boot process can “see” a hard drive, but can’t identify it, it just stalls. Without any message or beep code or alert of any kind. And either one of the built-in SATA ports is flaky or they have to be “consumed” in a particular order (e.g., master before slave on a given channel), so… It’s disturbing to plug drives in and have them work, only to plug the same drives in to different ports and have the system freeze. With no hint as to what’s wrong.

Now, space is admittedly at a premium for firmware, so it’s not like it can contain a robust error reporting system. OTOH, modern firmware does contain a lot of stuff, including a number of messages. Would it really have been so hard to include “Uh, drive seen but not recognized on SATA port X”? Besides being really helpful, not having such messages violates what I consider to be one of the most important rules of well-designed software: don’t leave the user hanging. Log something, somewhere — screen, log file, carrier pigeon, the location doesn’t matter (so long as it’s known).

There’s nothing worse than trying to figure out a problem with no information as to what it is. It forces you to go into trial and error mode, also known as Keep Moving Everything Around Until It Mysteriously Starts Working Again. Not a pleasant experience, and not one that anyone should have to experience…so long as the software is well-designed.

The morals of the story? A few:

  • If you use Windows Storage Spaces, always leave some unused hard drive ports available in your system.
  • Better yet, think really hard about using Windows Storage Spaces without a full-time IT staff (I’ve abandoned it based on this experience).
  • If your motherboard appears to freeze during the early stages of the boot process, consider that it may be having problems recognizing hard drives but is too ashamed to let you know that.

Azure WebJobs, Hidden UTF-8 Parameters and Azure

So as to keep others from beating their heads bloody working through “problems which shouldn’t be problems”, here’s an important, but little known, fact about batch files used to launch Azure WebJobs…

They can’t contain the UTF-8 Byte Order Mark. Affectionately known as “UTF-8 BOM”. Which really ought to known as “UTF-8 BOOM”, considering what it does to Azure when it shows up in a command file.

Which it will, by default, because Visual Studio inserts them, invisibly to you, whenever you save a file you’ve created. Sigh.

A tipoff that something’s wrong is an Azure log message which complains about an unreadable command file line, echoing some bizarre looking characters as a prefix to what >>you<< think the line is. Those characters are Azure’s attempt to display the BOM. Sadly, Azure isn’t smart enough to identify and work around this problem itself (are you listening, Microsoft? It would be really, really easy to scan for the BOM, and simply ignore it; heck, even I could figure out how to do that).

The only way to avoid this is to “Save As” the file, and then work your way through the dialog box options — by selecting Save with Encoding from the Save button — and picking the an encoding which doesn’t have the BOOM, uh, BOM. I typically use “UTF-8 without byte order mark”…which is inconveniently located near the end of a very long list of possible choices. Who knew there were so many encodings in the world?

Fortunately, you only have to do this once, because Visual Studio will remember the choice for as long as the file’s around. Unfortunately, you have to do it for every file you want treated that way; there’s no global option, at least not in Visual Studio 2015.

Thanx, Greenbow! But Not You, Cisco and Microsoft…

I’ve owned a Cisco Small Business router, model RV-325, for several years, and it’s worked very well as a firewall/router. So well, in fact, that after setting it up I think I’ve only had to log into it’s user interface once or twice to check things, or update the firmware.

But it supports VPNs, and I recently had cause to figure out how to set it up to do so. And on that front, it fell flat on its face.

Why is it that people who write hardware manuals assume you already know how to do whatever it is you’re checking the manual to do? It’s really an odd presumption… and an all too common one.

VPNs by their nature — and I am not at all an expert on them, although I know a lot more today than I did five days ago — are complicated, with many options. But that just highlights another problem, this time with hardware user interfaces: if the goal is simple — “I want to be able to access my LAN remotely” — but the steps involved are potentially complex, you need to abstract the interface to the point where the configuration process itself is simple. Or at least provide the option to do so.

When your fire up Word for the first time, you get what looks like a blank sheet of paper and a cursor. And if you start typing, lo and behold, words start appearing on the screen! Even though you didn’t configure anything. You can get started without having to be a tech guru, even when you try to print what your typed (although in that case it helps if your IT staff have named the printers in such a way that you can figure out which one is near you).

I was very much helped in my quest by a company called Greenbow, which makes a Windows VPN client. Whose user interface is admittedly a little less straightforward than perhaps it could be. But which more than makes up for that by actually generating error messages which one can figure out, at least with three days worth of knowledge of VPNs. The fact that I had to pay for it is irrelevant; it’s worth the price, just for that increased capability.

As for the Windows 10 VPN client: it’s so abstracted that I never was able to figure out where to enter certain critical data needed to make a connection. Granted, the user interface is beautifully simple. But it doesn’t support the task.


They Have World Class Talent and This Support Site?!?

I bought Barbara an Intel NUC — a tiny (5 inches square, 1.5 inch tall) computer — a couple of years ago. It’s fast, quiet, small and dependable…up until now.

The other day the wired ethernet connection died. It isn’t a cabling problem — I tested the connection with a Raspberry Pi — and downloading/re-installing the drivers didn’t fix it. So I went to the Intel support site to start a repair ticket.

That’s when the fun started.

[Read more…]

“Please Contact Windows”

I’m a long-time user of the Adobe Creative Suite. So I’m more familiar with Adobe software than I’d like to be…because it is generally insanely great, from a creative point of view, and all too often not very well written, from a nuts-and-bolts point of view.

[Read more…]

End of an Era: Part 1

Sometime in the late 90s I decided to learn enough about Linux to wire my house to the internet and run my own email server. It was a frustrating process at first because linux is radically different from the various Microsoft and Amiga OSs I’d been using for many years. I can still recall the feeling of my head exploding as I tried to wrap it around all the moving parts — linux itself, DNS, IP routing, sendmail, firewalls, NAT, etc. — that were necessary to “just run my own email server”.

[Read more…]

There’s Always a First Time

I can’t tell you how much time I’ve spent twiddling my thumbs over the years as various versions of Windows let me know that they’re creating a system restore point. I understand the concept — take a snapshot of the system drive before installing anything significant so you can restore it if something goes wrong — but, at least for me, when things go wrong with Windows they really, really go wrong. As in, what’s the point of having system restore snapshots on a drive that’s hosed anyway? Since restore points apparently have to be kept on the system drive and it’s the one that takes the brunt of any problems. So I’ve always considered system restore points a waste of time and disk space.

Until today. [Read more…]

IISExpress 8, SSL and Ports 80/443

IISExpress 8 is a nice little component of Visual Studio 2012. It allows you to test and debug websites without having to learn how to administer IIS. You just hit F5 and go.

But it does have some limitations. For example, by default it doesn’t run on ports 80 and 443, the latter being for SSL-enabled http interactions. That’s apparently to maintain system security. Unfortunately, it makes testing in a real world environment — virtually everything on the web runs over 80 and 443! — more challenging than it should be. I hope Microsoft changes the installation routine for Visual Studio so that it defaults instead to a lower security, but more realistic, environment which has IISExpress run over 80 and 443.

You can get there today, but it takes a little command line work and config file modification. Because many of the explanations I found online were both confusing and somewhat misleading I thought I’d detail the steps I went through in this posting. I’ve included some explanatory commentary, but be advised I’m no IIS or SSL expert, so that material could be wrong.

All the instructions here refer to IISExpress8, Visual Studio 2012, and were performed under Windows 8 64 bit.

The game plan for enabling 80/443 support in IISExpress8 is to create a security certificate to identify your development machine as an authorized machine, make IISExpress available over ports 80/443, and then configure IISExpress to serve up those ports for your development site.

Some online instructions call for you to create a single certificate to do this (e.g., Scott Hanselman’s blog entry). But that involves moving the certificate you create into your trusted root certificate store, and that relocation process can cause problems, depending upon how you do it (see the comments at the end of Scott’s post). In addition, configuring Windows http processing to use an SSL certificate seems to require the certificate be located in your personal certificate store, not trusted root. This caused me no end of trouble trying to figure out obscure error messages about specified logon sessions not existing. The problem was solved in yet another comment to Scott’s post, located part way down the page.

In what I outline here you avoid the issue entirely because you create two certificates, one that you put into the trusted root store and one which you put into your personal certificate store. The first certificate is a trusted root certification authority — a certificate used to sign other certificates. It’s used to sign a second certificate you create to identify your IISExpress server, which is in turn placed, and left, in your personal certificate store.

I found instructions on how to do this at Raj Rao’s blog. His post is intended to configure IIS, not IISExpress, so I had to adapt it a bit. It also references a more in-depth discussion which you may find helpful to review.

Here are the commands I used (you’ll need to be doing all of this at an administrative command prompt; I ran ‘Developer Command Prompt for VS2012’, created when Visual Studio 2012 is installed). So you’ll have the certificate and key files available for later use you should do this in a directory you’ve set up to store them.

makecert -r -pe -n "CN=Kensei Root CA" -ss Root -sr localMachine -a sha1 -sky signature -sv KenseiCA.pvk KenseiCA.cer

“Kensei Root CA” is the common name for the certificate authority I’m creating. I called mine ‘Kensei Root CA’, without the quotes, because my development system is named Kensei. I also adjusted the file names for the key and certificate files accordingly (i.e., the .pvk and .cer files at the end of the command line), calling them ‘KenseiCA’ to signify that they’re a certificate authority for Kensei.

You’ll have to assign some passwords when this command executes. Don’t forget them! To keep things simple I used the same password for all the password requests.

This installs a newly-created certificate in your trusted root store. As Raj points out, you can check this by firing up MMC.exe, installing the Certificate snap-in and looking in the trusted root store. Remember to choose ‘Computer Account’ and ‘Local Computer’ when it prompts you for which set of certificate stores to access, assuming you’re working directly on your development machine. The certificate’s uses should be “all”:mmc-cert

Next we create the server identity certificate. You do this by creating certificate and key files, signing them with the trusted root certificate authority you just created, combining them in a single PFX file, and then importing the resulting PFX file into your personal certificate store.

makecert -pe -n "CN=Kensei" -a sha1 -sky exchange -eku -ic KenseiCA.cer -iv KenseiCA.pvk -sp "Microsoft RSA SChannel Cryptographic Provider" -sy 12 -sv KenseiServer.pvk c:KenseiServer.cer

pvk2pfx -pvk KenseiServer.pvk -spc KenseiServer.cer -pfx KenseiServer.pfx

The first command creates a pair of files, KenseiServer.pvk and KenseiServer.cer, which containing a private key and the server identity certificate. Notice the references to the certificate authority you just created (KenseiCA.cer and KenseiCA.pvk in my case; use whatever names you specified in creating your certificate authority). You’ll have to supply the certificate authority’s password when the makecert command runs. The second command packages the key and certificate files into a PFX file. It doesn’t assign a password to the resulting file.

I’m pretty sure the common name (“CN=…”) used is important. It should match your development computer’s name. In my case that’s kensei.

Import the PFX file into your personal certificate store using MMC, by right clicking on Certificates -> Personal -> Certificates -> All Tasks -> Import..:


When using the import wizard, you want to import into Local Machine (although actually I don’t think you’ll be given any other choice). Also, when you select the file to import, remember to pick the PFX file you created, not any of the CER files which may first show up in the dialog; use the file filter dropdown to show the PFX file. Finally, when the ‘supply password’ screen shows up, remember the PFX file has no password, so leave the password field blank.

When you’re done you should have a newly-minted security certificate in your personal certificate store. You’ll need its thumbprint — it’s “ID” — which you can get by double-clicking the certificate name, selecting the Details tab and scrolling down to Thumbprint:


Copy and paste the thumbprint information from the textbox below the field list into Notepad, and delete all the spaces from the thumbprint. It’s this compressed version of the thumbprint which acts as the certificate’s ID.

We’re done with certificates and MMC, so you can close it down. The next commands finish off the Windows http configuration:

netsh http add sslcert ipport= appid={214124cd-d05b-4309-9af9-9caa44b2b74a} certhash=107c0a5f8baf14ebe204ac2b7247b8ad7069e6b8

netsh http add urlacl url=http://kensei:80/ user=Everyone

netsh http add urlacl url=https://kensei:443/ user=Everyone

netsh firewall add portopening TCP 80 IISExpressWeb enable ALL

netsh firewall add portopening TCP 443 IISExpressWeb enable ALL

The first command tells Windows to use the server identity certificate as an SSL certificate on port 443. The appid is a GUID, which you can generate using uuidgen.exe or just use the one here (the odds are vanishingly small that this will duplicate a GUID on your system, or so I’ve been led to believe).

The certhash parameter is the id you created in Notepad from your server identity certificate’s thumbprint. If you copy and paste it from within Notepad, be aware that an invisible UTF character is often present at the very beginning of the id string. This will show up as a square box in the command prompt window when you paste the id, and will cause the command to complain bitterly. Just recall the command, cursor back to the square box, delete it and hit enter again.

The next two commands tell Windows to listen to ports 80 and 443 when processing http requests. By default, these are turned off (i.e., the ports are ignored). Note that these particular commands allow access by every user. If you want to restrict access to only particular users adjust the commands accordingly. Note that the server name specified here should match the common name (CN) you used to create the server identification certificate. In my case I kept things simple by using my computer’s name, kensei, throughout the commands.

The last two commands tell your firewall to let other IP addresses access ports 80 and 443 on your machine. This is useful if you want to see how your site looks and works from, say, a smartphone tied in to your LAN. These particular commands enable access from any other IP address, so they’re pretty insecure. That’s okay in my case because my LAN is behind a firewall, but your mileage may differ.

By the way, the firewall editing commands (i.e., the last two) are obsolete/deprecated. They still work on Windows 8, which is why I include them here, but technically I believe you’re supposed to use the following commands in their place:

netsh advfirewall firewall add rule name="IISExpressWeb" dir=in protocol=tcp localport=80 profile=private remoteip=localsubnet action=allow

netsh advfirewall firewall add rule name="IISExpressWeb" dir=in protocol=tcp localport=443 profile=private remoteip=localsubnet action=allow

I haven’t tried these myself. Also, note the profile parameter, set to private. I believe that refers to the firewall zone you’re modifying, of which there are three: private, public and domain (assuming your system runs inside a Windows domain). You may well have to tweak the profile parameter to get things to work, and may have to issue the commands multiple times, for different zones, as well.

Now, modify IISExpress’ configuration file. It’s located at C:\Users\[your user name]\Documents\IISExpress\config\applicationhost.config. Open it (Notepad will do fine, or use Visual Studio) and locate the entry for the website you’re debugging (note: you’ll have to have run the website at least once from inside Visual Studio for the entry to exist):



You may have any number of websites listed, each with its own <site> section, so be careful to modify the correct one. You want to change the <binding protocol…> lines so that the bindingInformation parameters refer to ports 80 and 443, each on a separate <binding…> line (there may only be one entry present if you haven’t debugged the site under SSL). The format, as you can see from the sample above, is:

bindingInformation=”*:[port number]:[server name]”

In this case the quotes are important. You want to use the same server name as you specified in the “netsh http add urlacl” command you issued a few moments ago. In my case, that’s ‘kensei’, without the quotes.

Now it’s time for the final step, tweaking the Visual Studio project file for your website so it knows to run SSL over port 443. You do this by opening the project file in Notepad. Don’t just double-click the file; if you do, it’ll launch Visual Studio. Instead, right click the file in a directory window and select Open With -> Notepad.

Search for and modify the <IISExpressSSLPort> line so that it refers to port 443 (it’ll have some high port number, like 44300, by default):



If you don’t do this then when you enable SSL on the site’s property page within Visual Studio it’ll use whatever default port value was in the project file. That won’t be compatible with the port 443 approach you just set up.

That’s it! Happy debugging!

Sometimes a Mind Wipe Helps

The other day the system drive — the one containing Windows and all my programs —  died unexpectedly. As in, I didn’t have a backup for it.

Lesson #23,781 learned: never run any solid state drive without a robust backup process. Actually, that’s a revision to lesson #23,103 (“never run a cheap solid state drive without a robust backup process”). Apparently, all solid state drives are both wickedly fast and notoriously unreliable. Compared to dinosaur-like spinning platter drives, at least.

So I got to experience the joys of re-installing everything, including Windows 8, from the ground up.

Actually, it wasn’t all that bad: Win8 installs much faster than previous versions, and my mainstay apps (Microsoft Office and Adobe Master Suite) are sufficiently out-of-date that I didn’t run into any “you’ve already installed our software on another computer!” nonsense. I guess software companies don’t own real computers that, you know, catastrophically fail at unexpected times. Or maybe they all do regular backups.

I also noticed some benefits from doing a clean install: the nifty Win8 power management features now work properly. I can shutdown my system in seconds, and restore it almost as quickly. In fact, I bet if I had a recent “instant on” motherboard the restore would probably be as fast as the shutdown. It’s also nice that it gets restored to exactly where I left off, with the same apps and documents open, although that feature’s been around for awhile.


I managed to dodge a bullet today.

One of the hard drives in my main desktop system has been ticking for several months now. The problem first appeared after I installed Windows 8, so I naturally assumed it had something to do with the new OS. My research into the matter seemed to confirm that, when I found a number of reports of drive ticking caused by overly aggressive head parking by Windows under some circumstances.

But none of the fixes that others used to solve their ticking problems worked in my case. So I did some more digging, and learned that the far more common reason for a drive to be ticking is that it’s about to die.

It would be particularly painful for this specific drive to die because it has all my documents on it, including multiple gigabytes of family photos and videos. And I **blush** don’t do backups as often or as thoroughly as I should.

Replacing the drive and cloning the data from the old one to the new one solved the problem. No more distracting ticking! More importantly, much less risk of losing precious data!

Repeat after me, ten times: “Post hoc ergo propter hoc“. Which is Latin for “after this, therefore because of this”. And is a very, very famous logical fallacy.

Which I often quote to others, and should have remembered myself in this instance.