Azure WebJobs, Hidden UTF-8 Parameters and Azure

So as to keep others from beating their heads bloody working through “problems which shouldn’t be problems”, here’s an important, but little known, fact about batch files used to launch Azure WebJobs…

They can’t contain the UTF-8 Byte Order Mark. Affectionately known as “UTF-8 BOM”. Which really ought to known as “UTF-8 BOOM”, considering what it does to Azure when it shows up in a command file.

Which it will, by default, because Visual Studio inserts them, invisibly to you, whenever you save a file you’ve created. Sigh.

A tipoff that something’s wrong is an Azure log message which complains about an unreadable command file line, echoing some bizarre looking characters as a prefix to what >>you<< think the line is. Those characters are Azure’s attempt to display the BOM. Sadly, Azure isn’t smart enough to identify and work around this problem itself (are you listening, Microsoft? It would be really, really easy to scan for the BOM, and simply ignore it; heck, even I could figure out how to do that).

The only way to avoid this is to “Save As” the file, and then work your way through the dialog box options — by selecting Save with Encoding from the Save button — and picking the an encoding which doesn’t have the BOOM, uh, BOM. I typically use “UTF-8 without byte order mark”…which is inconveniently located near the end of a very long list of possible choices. Who knew there were so many encodings in the world?

Fortunately, you only have to do this once, because Visual Studio will remember the choice for as long as the file’s around. Unfortunately, you have to do it for every file you want treated that way; there’s no global option, at least not in Visual Studio 2015.

Hold Down That Shift Key!

I’m posting this here mostly as a reminder to me how to work around a poorly-documented aspect of migrating iTunes to a new computer.

In the current instance, I’ve recently built a new Windows PC. But the hard drive where all my iTunes media is stored is a holdover from the previous computer — where it worked just fine in iTunes. Same drive letter, same path to the media library, same file ownership credentials, etc.

Yet when I fired up iTunes and pointed it at the media folder — by editing the preferences — it refused to recognize any of the songs, videos, podcasts, what have. It helpfully offered to download all the stuff I’d purchased from the iTunes Store over the years…but who the heck wants to download gigabytes of material when the exact same files are already present on the hard drive??? Certainly not me!

It turns out the solution is simple…but completely un-intuitive. You have to launch iTunes with the shift key held down (Windows; for the Mac OS I believe it’s the option key). Which will then bring up a dialog box asking you to locate the iTunes library file (or create a new one).

Now, so far as I know, the iTunes library file, by default, is always in the root of the media folder. At least, I’ve never seen it anywhere else.

So you’d think iTunes would be smart enough, when it senses that it’s not accessing an iTunes library file, to go looking in the root of the media folder. But no, it’s too stupid to do that. Instead, it just creates a new, empty iTunes library file, points itself at it, and declares “hey, you’ve got a lot of music up in the cloud; too bad none of it’s down here!”.

With all due respect to Apple — and I’ve told people for years that they’d have to pry my iPhone out of my cold, dead fingers to get me to relinquish it — this is nuts. Plus, it’s lousy software design. Defaults are a good thing, but they have to be smart defaults. It would hardly take any time at all for iTunes, on launch, to check the root of the iTunes media folder, see an iTunes library file, realize it wasn’t pointing at it, and ask “hey, is that the library file you want me to use?” It could even parse the iTunes library file, see it was pointing to a whole bunch of media files in the drive system…and realize that this iTunes library file was the real McCoy.

Are you listening, Apple?

 

Into the Wayback Machine

My buddy Connell runs an election/campaign advisory firm. He has a database that he uses to track voter profiles, create outreach lists, and the like. To minimize annoying voters — never a good idea when you’re running a campaign — he needs to stay on top of whoever has already voted, so he can take them off the outreach lists (who has voted is a matter of public record; how you vote is not).

Unfortunately, his database/computer guy isn’t available, and Connell wasn’t sure how to import the voting data. Worse yet, he couldn’t find the manual for the software. Between the two of us we came up with a number of different ways to try and work around these problems…but none of them succeeded. So, as always, when subtlety doesn’t work, it was time to turn to brute force :).

[Read more…]

Murphy’s Law

Okay, deep breath. I’m going to write about this because, well, it is funny, in a dark, ironic way. At least now that a few hours have passed.

[Read more…]

IISExpress 8, SSL and Ports 80/443

IISExpress 8 is a nice little component of Visual Studio 2012. It allows you to test and debug websites without having to learn how to administer IIS. You just hit F5 and go.

But it does have some limitations. For example, by default it doesn’t run on ports 80 and 443, the latter being for SSL-enabled http interactions. That’s apparently to maintain system security. Unfortunately, it makes testing in a real world environment — virtually everything on the web runs over 80 and 443! — more challenging than it should be. I hope Microsoft changes the installation routine for Visual Studio so that it defaults instead to a lower security, but more realistic, environment which has IISExpress run over 80 and 443.

You can get there today, but it takes a little command line work and config file modification. Because many of the explanations I found online were both confusing and somewhat misleading I thought I’d detail the steps I went through in this posting. I’ve included some explanatory commentary, but be advised I’m no IIS or SSL expert, so that material could be wrong.

All the instructions here refer to IISExpress8, Visual Studio 2012, and were performed under Windows 8 64 bit.

The game plan for enabling 80/443 support in IISExpress8 is to create a security certificate to identify your development machine as an authorized machine, make IISExpress available over ports 80/443, and then configure IISExpress to serve up those ports for your development site.

Some online instructions call for you to create a single certificate to do this (e.g., Scott Hanselman’s blog entry). But that involves moving the certificate you create into your trusted root certificate store, and that relocation process can cause problems, depending upon how you do it (see the comments at the end of Scott’s post). In addition, configuring Windows http processing to use an SSL certificate seems to require the certificate be located in your personal certificate store, not trusted root. This caused me no end of trouble trying to figure out obscure error messages about specified logon sessions not existing. The problem was solved in yet another comment to Scott’s post, located part way down the page.

In what I outline here you avoid the issue entirely because you create two certificates, one that you put into the trusted root store and one which you put into your personal certificate store. The first certificate is a trusted root certification authority — a certificate used to sign other certificates. It’s used to sign a second certificate you create to identify your IISExpress server, which is in turn placed, and left, in your personal certificate store.

I found instructions on how to do this at Raj Rao’s blog. His post is intended to configure IIS, not IISExpress, so I had to adapt it a bit. It also references a more in-depth discussion which you may find helpful to review.

Here are the commands I used (you’ll need to be doing all of this at an administrative command prompt; I ran ‘Developer Command Prompt for VS2012’, created when Visual Studio 2012 is installed). So you’ll have the certificate and key files available for later use you should do this in a directory you’ve set up to store them.


makecert -r -pe -n "CN=Kensei Root CA" -ss Root -sr localMachine -a sha1 -sky signature -sv KenseiCA.pvk KenseiCA.cer

“Kensei Root CA” is the common name for the certificate authority I’m creating. I called mine ‘Kensei Root CA’, without the quotes, because my development system is named Kensei. I also adjusted the file names for the key and certificate files accordingly (i.e., the .pvk and .cer files at the end of the command line), calling them ‘KenseiCA’ to signify that they’re a certificate authority for Kensei.

You’ll have to assign some passwords when this command executes. Don’t forget them! To keep things simple I used the same password for all the password requests.

This installs a newly-created certificate in your trusted root store. As Raj points out, you can check this by firing up MMC.exe, installing the Certificate snap-in and looking in the trusted root store. Remember to choose ‘Computer Account’ and ‘Local Computer’ when it prompts you for which set of certificate stores to access, assuming you’re working directly on your development machine. The certificate’s uses should be “all”:mmc-cert

Next we create the server identity certificate. You do this by creating certificate and key files, signing them with the trusted root certificate authority you just created, combining them in a single PFX file, and then importing the resulting PFX file into your personal certificate store.


makecert -pe -n "CN=Kensei" -a sha1 -sky exchange -eku 1.3.6.1.5.5.7.3.1 -ic KenseiCA.cer -iv KenseiCA.pvk -sp "Microsoft RSA SChannel Cryptographic Provider" -sy 12 -sv KenseiServer.pvk c:KenseiServer.cer

pvk2pfx -pvk KenseiServer.pvk -spc KenseiServer.cer -pfx KenseiServer.pfx

The first command creates a pair of files, KenseiServer.pvk and KenseiServer.cer, which containing a private key and the server identity certificate. Notice the references to the certificate authority you just created (KenseiCA.cer and KenseiCA.pvk in my case; use whatever names you specified in creating your certificate authority). You’ll have to supply the certificate authority’s password when the makecert command runs. The second command packages the key and certificate files into a PFX file. It doesn’t assign a password to the resulting file.

I’m pretty sure the common name (“CN=…”) used is important. It should match your development computer’s name. In my case that’s kensei.

Import the PFX file into your personal certificate store using MMC, by right clicking on Certificates -> Personal -> Certificates -> All Tasks -> Import..:

cert-import

When using the import wizard, you want to import into Local Machine (although actually I don’t think you’ll be given any other choice). Also, when you select the file to import, remember to pick the PFX file you created, not any of the CER files which may first show up in the dialog; use the file filter dropdown to show the PFX file. Finally, when the ‘supply password’ screen shows up, remember the PFX file has no password, so leave the password field blank.

When you’re done you should have a newly-minted security certificate in your personal certificate store. You’ll need its thumbprint — it’s “ID” — which you can get by double-clicking the certificate name, selecting the Details tab and scrolling down to Thumbprint:

cert-thumb

Copy and paste the thumbprint information from the textbox below the field list into Notepad, and delete all the spaces from the thumbprint. It’s this compressed version of the thumbprint which acts as the certificate’s ID.

We’re done with certificates and MMC, so you can close it down. The next commands finish off the Windows http configuration:


netsh http add sslcert ipport=0.0.0.0:443 appid={214124cd-d05b-4309-9af9-9caa44b2b74a} certhash=107c0a5f8baf14ebe204ac2b7247b8ad7069e6b8

netsh http add urlacl url=http://kensei:80/ user=Everyone

netsh http add urlacl url=https://kensei:443/ user=Everyone

netsh firewall add portopening TCP 80 IISExpressWeb enable ALL

netsh firewall add portopening TCP 443 IISExpressWeb enable ALL

The first command tells Windows to use the server identity certificate as an SSL certificate on port 443. The appid is a GUID, which you can generate using uuidgen.exe or just use the one here (the odds are vanishingly small that this will duplicate a GUID on your system, or so I’ve been led to believe).

The certhash parameter is the id you created in Notepad from your server identity certificate’s thumbprint. If you copy and paste it from within Notepad, be aware that an invisible UTF character is often present at the very beginning of the id string. This will show up as a square box in the command prompt window when you paste the id, and will cause the command to complain bitterly. Just recall the command, cursor back to the square box, delete it and hit enter again.

The next two commands tell Windows to listen to ports 80 and 443 when processing http requests. By default, these are turned off (i.e., the ports are ignored). Note that these particular commands allow access by every user. If you want to restrict access to only particular users adjust the commands accordingly. Note that the server name specified here should match the common name (CN) you used to create the server identification certificate. In my case I kept things simple by using my computer’s name, kensei, throughout the commands.

The last two commands tell your firewall to let other IP addresses access ports 80 and 443 on your machine. This is useful if you want to see how your site looks and works from, say, a smartphone tied in to your LAN. These particular commands enable access from any other IP address, so they’re pretty insecure. That’s okay in my case because my LAN is behind a firewall, but your mileage may differ.

By the way, the firewall editing commands (i.e., the last two) are obsolete/deprecated. They still work on Windows 8, which is why I include them here, but technically I believe you’re supposed to use the following commands in their place:


netsh advfirewall firewall add rule name="IISExpressWeb" dir=in protocol=tcp localport=80 profile=private remoteip=localsubnet action=allow

netsh advfirewall firewall add rule name="IISExpressWeb" dir=in protocol=tcp localport=443 profile=private remoteip=localsubnet action=allow

I haven’t tried these myself. Also, note the profile parameter, set to private. I believe that refers to the firewall zone you’re modifying, of which there are three: private, public and domain (assuming your system runs inside a Windows domain). You may well have to tweak the profile parameter to get things to work, and may have to issue the commands multiple times, for different zones, as well.

Now, modify IISExpress’ configuration file. It’s located at C:\Users\[your user name]\Documents\IISExpress\config\applicationhost.config. Open it (Notepad will do fine, or use Visual Studio) and locate the entry for the website you’re debugging (note: you’ll have to have run the website at least once from inside Visual Studio for the entry to exist):

iisexp-config

 

You may have any number of websites listed, each with its own <site> section, so be careful to modify the correct one. You want to change the <binding protocol…> lines so that the bindingInformation parameters refer to ports 80 and 443, each on a separate <binding…> line (there may only be one entry present if you haven’t debugged the site under SSL). The format, as you can see from the sample above, is:

bindingInformation=”*:[port number]:[server name]”

In this case the quotes are important. You want to use the same server name as you specified in the “netsh http add urlacl” command you issued a few moments ago. In my case, that’s ‘kensei’, without the quotes.

Now it’s time for the final step, tweaking the Visual Studio project file for your website so it knows to run SSL over port 443. You do this by opening the project file in Notepad. Don’t just double-click the file; if you do, it’ll launch Visual Studio. Instead, right click the file in a directory window and select Open With -> Notepad.

Search for and modify the <IISExpressSSLPort> line so that it refers to port 443 (it’ll have some high port number, like 44300, by default):

proj-file

 

If you don’t do this then when you enable SSL on the site’s property page within Visual Studio it’ll use whatever default port value was in the project file. That won’t be compatible with the port 443 approach you just set up.

That’s it! Happy debugging!