Daniel Lyons:

If you’ve ever wondered what Apple will be like after Steve leaves and they’ve got visionaries like Phil Schiller and Tim Cook running the place, well, now you’ve seen the trailer. The movie will be worse.

Remember “industries of middlemen”? Here’s a striking case in point:

“I’ve never made a dime from a record sale in the history of my record deal. I’ve been very happy with my sales, and certainly my audience has been very supportive. I make a living going out and playing shows.”

Lovett, 50, has sold 4.6 million albums in the United States since 1991, the year when SoundScan sales data were introduced.

Maybe the labels’ talk of “protecting artists” should be understood in the sense of the honoured society.

One of my evening projects has been setting up OpenNMS to monitor a network primarily comprised of VENDORNAME switches. OpenNMS is being put in to replace a bundle of Nagios, Cacti, Smokeping, and Groundwork Fruity for Nagios configuration management. The existing system worked well enough, but the lack of autodiscovery of services/nodes along with the poor integration between cacti and nagios was getting a bit annoying.

After setting up and trialling OpenNMS for a bit, we deployed it on this network. And then the switches started failing. They’d still switch packets, and I believe still responded to SNMP, but you couldn’t connect to them via any of the management interfaces.

So, we started looking at the differences between OpenNMS and Nagios/Cacti/Smokeping. Both do SNMP and ICMP queries, and some TCP port availability checks. The combined stack actually does more SNMP traffic because both Cacti and Nagios ended up querying the same OIDs. I’ve often noticed that Cacti sends individual requests for OIDs however, rather than grouping them, whereas OpenNMS defaults to requesting 10 OIDs per PDU. I changed this in the configuration (and later on changed it for real, as it was being set in a different config file as well), and let OpenNMS run against some test switches… and they locked up.

Perry suggested that it could be a memory leak due to the service polling, and set up a test where he polled the SSH server once a minute forever. This test got cancelled after 4 days or so, but the machines hadn’t died at that point, so we decided it wasn’t anything fundamental about the service checks.

I set up a range of services that were being monitored on 10 switches, and let them go for a bit. Due to power outages and equipment moves this step ended up taking longer than it needed to, but the end result was that no matter which services were being monitored, all the switches all locked up at around the same point.

And then I noticed that the switches had a growing number of stale “telnet-d” connections. These switches have capacity for up to 4 concurrent administrative logins - once all 4 slots are full, you can no longer log in. So, the theory is these stale connections were blocking real connections, and never timing out, thus causing the lockout of the management stack. They don’t time out, and you can’t kill them from the switch console short of rebooting the switch. Most of the switches weren’t being actively monitered for telnet, but OpenNMS does do service discovery periodically (I think once a day, and perhaps under other situations too), and this would probe each service. So I firewalled telnet out, and had the switches restarted, thinking this would solve it.

The switches still locked up.

The switches still had stale telnet connections appearing in them.

I turned off the telnet service on each switch, thinking that perhaps there was something else on the network that was talking to them, and restarted them.

Within 5 minutes of rebooting each switch, there was a stale telnet connection listed. Awesome.

So, we’re down to a service that is being misreported as a telnet service. I go through all of them, and discover that none of the other services - FTP, HTTP, HTTPS - even show up as an active session. Which leaves telnet - firewalled out - and SSH.

The OpenNMS plugin which handles discovery of SSH servers is a bit smarter than a basic “is a service listening on port 22″ sort of discovery - it waits for the SSH banner from the server, then sends it’s own SSH banner back, and verifies that it gets a response back. This means it’s getting part way through the SSH establishment, and then canning the connection.

As a quick test, I telnetted to port 22 on a switch and checked the login listing. With the banner is being displayed, nothing even shows up. When I pasted a valid looking SSH banner back, I got a bunch of binary data echoed into my telnet session, and ssh session to the switch locks up. On reconnecting and checking the login listing, sure enough - a stale telnet session was there.

Further tests reveal that if you ssh to one of these switches, but don’t type your password in, the session gets reported as a telnet session. Furthermore, if you kill your ssh process or shell window while the ssh session is waiting for your password, the session never disappears.

So, we have a very live DOS exploit against VENDORNAME switches here, assuming anyone is unwise enough to allow SSH access from random networks and VLANs to their switches that is. I have to point out that while it’s a particular “feature” of OpenNMS that triggered this problem for us, this isn’t a bug in OpenNMS at all, given that it’s trivial to trigger the same problems with the switches directly.

In regards to the actual problem at hand, OpenNMS is quite configurable, so at least I can change the way it does SSH service discovery to revert to a simple “is the port up” check. I’ve left this running for nearly two weeks now, and the switches on my test bed are all still behaving properly.

I held back from posting this until I could get a response from the vendor. They’ve acknowledged the bug, and a fix will be out in the next firmware release apparently. I might update once they have released a new firmware; I’ve edited out the vendor name from this post because I don’t believe it’s responsible to publish denial-of-service vulnerabilities without giving the vendor a chance to fix them.

I also noticed this post on the OpenNMS blog. The author there had similar problems with monitoring a firewall device, and while the scenario seems different, VENDORNAME makes firewalls as well as switches; I wonder if it’s the same vendor in his case.

Too much overloading is perfidious and too little is oppressive.

On hardy after the latest round of updates:


matt@krypton:~$ dpkg -s flashplugin-nonfree | grep Version
Version: 10.0.1.218+10.0.0.525ubuntu1~hardy1+really9.0.124.0ubuntu2


Granted this package is in hardy-backports not hardy proper, but still, what on earth?!?!

[Note: what I’m writing here isn’t news. However, I’ve not seen a set of fire-and-forget instructions so far.]

1. You can get the new fonts (except Segoe UI) independently of Vista: they are bundled with the PowerPoint Viewer 2007.
2. This is a self-extracting cabinet archive, extractable with cabextract 1.2 or later.
3. The fonts are in the ppviewer.cab archive within PowerPointViewer.exe.
4. System-wide font installation is system dependendent (and should be managed by the package manager anyway), but contemporary Linux distributions are configured to allow you to install fonts privately to your home directory in ~/.fonts.

In summary, we get the following script:

#!/bin/sh
set -e

exists() { which "$1" &> /dev/null ; }

if ! [ -d ~/.fonts ] ; then
    exec 2>&1
    echo 'There is no .fonts directory in your home.'
    echo 'Is fontconfig set up for privately installed fonts?'
    exit 1
fi

# split up to keep the download command short
DL_HOST=download.microsoft.com
DL_PATH=download/f/5/a/f5a3df76-d856-4a61-a6bd-722f52a5be26
ARCHIVE=PowerPointViewer.exe
URL="http://$DL_HOST/$DL_PATH/$ARCHIVE"

if ! [ -e "$ARCHIVE" ] ; then
    if   exists curl  ; then curl -O "$URL"
    elif exists wget  ; then wget    "$URL"
    elif exists fetch ; then fetch   "$URL"
    fi
fi

TMPDIR=`mktemp -d`
trap 'rm -rf "$TMPDIR"' EXIT SIGINT SIGQUIT SIGTERM

cabextract -L -F ppviewer.cab -d "$TMPDIR" "$ARCHIVE"

cabextract -L -F '*.TT[FC]' -d ~/.fonts "$TMPDIR/ppviewer.cab"

( cd ~/.fonts && mv cambria.ttc cambria.ttf && chmod 600 \
    calibri{,b,i,z}.ttf cambria{,b,i,z}.ttf candara{,b,i,z}.ttf \
    consola{,b,i,z}.ttf constan{,b,i,z}.ttf corbel{,b,i,z}.ttf )

fc-cache -fv ~/.fonts

Update: based on feedback I made some small improvements:

• The script can now download PowerPointViewer.exe with any of several different utilities (the BSDs and MacOS X bundle fetch and curl, respectively, rather than wget). Note that you can download PowerPointViewer.exe manually and then run the script from the directory where you saved the downloaded file, and it will use that file – this functionality was already present in the previous version.

• It will check whether a .fonts directory exists in your home, and abort if that’s not the case. I don’t know whether I can actually assume that fontconfig is generally configured to pick up privately installed fonts in ~/.fonts, so I don’t know whether it’s reasonable to create that directory if it does not exist. I’d rather the script refuse to work in dubious circumstances than that it do something without any effect.

Also, be aware that cabextract version 1.1 or earlier will not work. These versions cannot process the PowerPoint Viewer installer’s cabinet format (although Fedora Core 6 apparently ships a patched 1.1 that can), nor do they support the -d switch I used to keep the script simple. I would have added a version check to the code, but comparing version strings properly is difficult so the check would have bloated the script.

Update: previous versions of the script omitted the regular weight of Cambria because for some reason its extension is TTC rather than TTF. This is now fixed.

Share and enjoy.

#!/usr/bin/perl
use XML::Liberal;
XML::Liberal->new->parse_fh(STDIN)->toFH(STDOUT);

That way you can stop caring about the fact that a lot of people are incorrigible bozos. ¹ 

The way the module works is quite an amusing hack, by the way.

I just got a pingback on my earlier post, which was from a blog that indexes posts and articles with a particular word - acceptance - in it. Kind of an odd premise for a blog.

UPDATE As per the comment, the author/owner of More Lyrics updated his blog to remove the quote. My original comment was tongue in cheek, but it’s only fair to remove it I think :)

It seems that the original subject of my post yesterday caught the eyes of much of the virtualisation community, including Simon Crosby, formerly from Xensource, and now working for Citrix.

He’s written a typically well thought out response, which covers off a lot of points:

• HP have a multi-hypervisor management tool already which signs off on Xenserver, VMWare and Hyper-V support
• Xenserver Platinum, which is comprised of Xenserver Enterprise and Citrix Provisioning Server, can already provision VMs to not only physical hardware and Xenserver, but to other hypervisors as well
• He covered off again the ecosystem building around the Xenserver product range, specifically in HA areas - products like Marathon Everrun and Stratus Avance.

He also wrote up a good bit on the position of Xen with regards to KVM. I haven’t really looked into KVM much, due to not ready access to test hardware with VT capable chips (the test hardware I do have is tied up with testing Xenserver), but I’ve always been wary of various claims that it’s a better VM stack than Xen is. (That might just be because I’ve not spent the time looking into it, and it might be because of the general not-invented-here feeling the “linux kernel” community seems to have about Xen. Again, not something I’ve spent a lot of time on). A lot of the stuff Simon writes is high level and enthusiastic of course, but it paints a clear picture - Xen already has massive uptake in mindset, and not just with traditional linux vendors either (Sun xVM and Oracle VM having Xen based stacks as well). I guess the jury might still be out on which technology actually is technically superior, but as history demonstrates, it’s not always the technically superior technology that lasts.

Simon also claims that Xen will be in the BIOS hypervisor offering from Phoenix, which is something I haven’t heard before. It certainly makes some amount of sense for Phoenix to not rewrite an entire hypervisor stack and then stick it some place that’s inherently difficult to upgrade - your BIOS, but I’m not sure how it works out regarding Xen’s requirement for a privileged Xen-aware guest to provide hardware drivers.

Simon also makes another point that I must have heard before from him, because it’s stuck with me and I agree entirely with the premise:

The founding thesis of XenSource, and the continued strategy at Citrix, is to promote fast, free, compatible and ubiquitous hypervisor based virtualization. If the hypervisor is free, why worry about who delivers it? Let the customer pick the implementation method that they want - the real money is in the up-sell with products that make virtualization valuable for customers.

Whether you like it that companies are in this to make money or not, this approach seems a good one. Piggybacking their moneymaking on an opensource product, an action which drives development, acceptance and that horrible word “mindshare”, doesn’t have to be a bad thing.

I won a copy of this not so long ago and thought I'd give it a bit of a whirl. Overall I think it it's actually pretty good and can see how it would help home users.

There have been a few gotchas though which are worth mentioning if you end up supporting it for your customers/relatives/friends.

• My VPN to work totally stopped working. Found out that it locks most networking protocols. With programs OneCare pops up a lot and asks you what you want to do (reminds me of how ZoneAlarm was originally). No such luck with protocols though and I had to resort to "what did I last do" to solve this issue - then I realised it was OneCare
• Tells me that my security is only rated "fair" because I don't do backups. I store all my data on work servers or in the cloud (obligatory buzzword there!) so I don't care
• Turned on IE7 phishing filter and on the 10% of the time I use IE instead of Firefox 3 it runs like a dog. So phishing is now turned off

I think it is good overall but still would probably use AVG, Avast or Avira as you can get free home licenses for all of these.

Seems there’s a bit of debate at the moment about the future of Xen within Citrix’s product range, all sparked by this article by Brian Madden, which he clarified later on.

Brian’s followup clarifies his point:

When I say that Citrix will drop Xen, I mean that Citrix will drop the open source Xen hypervisor. I do not believe that Citrix will drop their XenServer product.

When you consider that Citrix Xenserver is a hypervisor based virtualisation stack (Xen on CentOS), and a virtualisation management tool (XenCenter), then sure, it’s possible for Citrix to change XenCenter so that it manages Windows Hyper-V instead. Xenserver, the product and brand, becomes a Windows 2008 Hyper-V install, and XenCenter manages that instead. It’s possible. Scott’s comments about porting Xen to windows missed the mark - Citrix only need to port the management stack and change the virtualisation layer to windows. RedHat are in the process of doing something similar with their recent move away from Xen to KVM. It’s not as radical a shift as from Xen to Hyper-V, but it’s as radical as you need to be - it’s a completely different virtualisation stack.

I’m still not sure I agree with Brian though. Citrix just dropped $550M on purchasing Xensource, and then promptly rebranded their flagship product to match. Granted, Citrix have a great track record for rebranding every couple of years, but it seems like a colossal waste of money given that Hyper-V, while not released at the time, was defintely public knowledge.

Citrix also have no need to drop the Xen out from under Xenserver. Citrix Workflow Studio already handles some automation tasks for both Xenserver and Hyper-V, and it’s no stretch to see this working on VMWare systems as well. Moreover, XenCenter itself could be modified to manage both Xen-based Xenserver systems as well as Windows Hyper-V systems. The reverse will definitely happen from Microsoft’s point of view - integration with XenServer in Microsoft’s Systems Center Operations Manager has been talked about for months now.

One prediction that is worth making is that cross-VM management stacks will flourish and improve. The example of Hyper-V and Xenserver was mentioned earlier, but they will grow to cover other assorted Xen based stacks from Virtual Iron, Novell, Sun etc, KVM stacks like RedHat, and of course VMWare. Citrix Workflow Studio makes a start in some ways, and products like VMLogix’s Lab Manager. Enomalism is already much of the way there, and goes a step beyond into cloud computing. The hypervisor (or at least, some kind of virtualisation) will be ubiquitous, and the winners will be the management stacks.

I was reading this article from Ars about Ubuntu 8.10 features and saw some links about NetworkManager at end. These links are here, here and here and link to Dan William's blog.

This looks like the start of NM being more useful, than annoying - sure it worked in the past but it seriously screwed things up a lot. I like the idea of a 3G connection being able to be shared easily - means Jayne and I can sit in a hotel room and both surf the web.

It's also good to see progress on the WiFi scene. Wireless on Linux has been a very mixed bag for years with mostly out of kernel drivers. This has been sorted out via moving to the ex-Devicescape code in the kernel and drivers all shepherded by John Linville. Unfortunately 2.6.24 for Ubuntu 8.04 was worse for many Atheros cards and people reverted for madwifi. I believe that this is getting largely fixed so Ubuntu 8.10 might be the one for road warriors (as also mentioned in the article at start)

I’m very interested in the topic of network deployments of operating systems, specifically the various Microsoft OSs, as I can already install linux via PXEboot. There’s two main groups of software in this field - unattended or scripted installs, and imaged installs.

A while ago I found a tool called Unattended, which is a network based unattended installation tool for Windows. If it works, it looks very promising. It’s basically a DOS boot disk which mounts a network share and executes the windows installer. Simplicity. The basic install seems to require you to enter a number of responses to questions (such as administrator password, timezone and Microsoft product key), but the documentation explains how to customise the script to meet your business needs, including examples. Once the OS install is done, Unattended can be configured to install third party packages, as long as the packages (eg, MSI bundles) also support some level of unattended installation procedure.

Today I discovered Free Online Ghost, or FOG. FOG is network based computer imaging tool, designed to both read images from, and write images to hosts on your network. I’ve used tools like partimage in the past for exactly this purpose - creating a golden image of a lab machine and then reimaging the entire lab every couple of months to keep everything clean. FOG seems to be more polished than partimage does, as it claims to support things like creating AD accounts for the machine and so on.

The Unattended documentation includes a concise explanation of why the approach adopted by FOG, partimage, and commercial tools like Acronis and Ghost is bad, however I think this is really a case of using the right tool for the job. I can see a system like FOG being used with great success in a lab environment, or for periodic backup of individual host OSes to near-line storage, providing bare-metal restore functionality without requiring major investment in tape backup expansion. And Unattended makes a lot more sense for initial deployments, especially for my workplace, as we use such a wide range of hardware that an imaged install would be fairly problematic.

There are other commercial systems for doing these deployments of course - IBM Director, HP ICE, Citrix Provisioning Server are just a few of them, but these systems invariably make more sense for in-house deployment control.

One of the servers I maintain is the jabber server at jabber.meta.net.nz. This is a free public service, anyone can use it, and it does get quite a wide range of use - for a long time we seemed to be very popular for south american users, possibly because of the web based clients and the range of transports to other protocols we support. We typically see between 50 and 100 concurrent users, depending on time of day and week, but the active account base is normally in the low thousands.

The transports themselves cause me a lot of problems. In the past they’ve been downright buggy, crashing all the time, but with the current codebase for all four protocols in use (AIM, ICQ, MSN and Yahoo) all being in python, we don’t seem to have as many outright crashes. We do have slow memory leaks however, which prompted me to move the services to a new server a while back. Part of me was hoping that the memory leaks were caused by the gentoo system I was using initially, but this doesn’t seem to be the case.

So, I needed to either fix these memory leaks, or to work around them. Enter monit. I’ve heard about monit quite a bit, but never really looked into it other than thinking it might be interesting. I really wish I’d looked further ages ago. It’s easy to set up, is designed specifically to monitor and restart services, and it solved my memory leak problem in about 5 minutes.

Here’s a snippet from the config file:

This is pretty self explanatory really. If CPU usage of this process gets too high, alert, then restart if it stays high for 5 cycles. And if the ram usage is over 300 MB for 5 cycles (a cycle is 2 minutes by default), restart the process. Problem solved. Or rather, the symptoms are solved, but that’s good enough for me at this stage

NoteThis is old, but somehow didn’t get posted

Stephen Adkins:

I have been promoting the idea of a community project to get Perl supported on Google App Engine.

I have made various contacts with people inside Google, and it seems that the core App Engine team is busy with plenty of other things. However, this does not stop the community from getting something started. We will likely attract perl advocates within Google as we go along, even if they are not on the App Engine team. I would like to see the effort advance until the point where it becomes a simple matter for the App Engine team to embrace Perl.

I have started a project for all people interested in following this effort or contributing toward it.

http://code.google.com/p/perl-appengine/

Please visit the website, sign up for the mailing list, spread the word, and start contributing.

Rogers Cadenhead:

Asking [AP] to concede there's a way people can share [headlines and story leads] for free is like asking the RIAA to pick its favorite file-sharing client.

By now I’ve had two backlinks, two emails, and several people pinging me on IRC, all pointing out the irony in my Don’t Go Dark, Go Distributed note suggesting DVCS as a way out of the going-dark problem, when in fact the original article that Atwood quoted from pointed at them as a problem magnifier. But of course, distributed source control making it easy to go dark is just one of the problems that article points out; and as I said, it remains to be seen how much of the problem is social.

Sussman takes off from there to throw in a few unsubstantiated conclusions about the limitations of Subversion somehow keeping the going-dark effect at bay, and tops them off with some comical FUD about git, contrasting with Mercurial. [Note: both Pedro Melo and an anonymous commenter on the posting do a better job of tearing apart the argument than I could be bothered to. You should probably read those instead of the rest of this entry.] To wit:

Another example: when it comes to pushing and pulling changesets, Mercurial’s default behavior is to exchange all history with the remote repository, while git’s default behavior is to only push or pull a single branch — presumably one that the user has deemed fit for sharing with the public. In other words, git defaults to all work being private cave-work, and is happy to destroy history.

Mercurial philosophy may have changed, but it used to be that to create a branch you should simply clone the repository. In-repository branch management was seen as a heavyweight approach. In such a constellation, Mercurial will obviously exchange no more history with a remote repository than will git.

The most mind-boggling part, though, is this:

Clearly, the growing popularity of distributed version control shows just how thrilled people are to hide their work from each other.

The fact that they help people go dark is the reason for their growing popularity‽ Well, Ben Collins-Sussman is one of the designers and core developers of Subversion, so maybe I should not be surprised at such resounding nonsense. For what it’s worth, I have sat on a 5000-line patch – yes, in my working copy –, precisely because I was using Subversion. A number of mistakes coincided to create that situation, but had I been using a DVCS at the time, I would, at the very least, now have an actual commit history of that work. I might even have been able to avoid going dark.

Anecdote vs anecdote.

But really, I don’t care to argue. I honestly wish I would have clicked the link first – that would have kept me from wading into this pointless argument at all. The article reeks of the smell of fear that is emanating from the Subversion camp these days, and I have no interest in getting too close to that.

Jeff Atwood argues that open source projects are in real danger of programmers “going dark,” which means they lock themselves away silently for a long time, then surface with a huge patch that implements a complex feature.

It seems to me that this is as much a technological problem as a social issue… and that we have the technological solution figured out: it’s called distributed version control. It means that that lone developer who locked himself in a room need not resurface with a single huge patch – instead, he can come back with a branch implementing the feature in individually comprehensible steps. At the same time, it allows the lone programmer to experiment in private and throw away the most embarrassing mistakes, addressing part of the social problem. (Git scores particularly highly on both counts as it provides tools for extensive history rewrites. This means that the revision history can edited retrospectively both for readability and for preservation of pride. As Assaf Arkin quips, that way, we’ll think you wrote test cases ahead of the code!)

Now that the tools have started to gain real traction – more so in the open source world than in corporate, of course, but it’s in open source that the problem is most dangerous, as corporate can use policy to cope with it –, it remains to be seen how much of the problem is in fact social.

While reading Tim Bray’s Deletionist Morons note, it struck me:

Maybe the deletionists should be relegated to arguing not about whether an article should be deleted, but about whether it gets voted out of the Notable club (and any number of other such clubs that they care to come up with). This label could have an actual effect, such as excluding the article from the Random Page and other kinds of show-this-by-default lists. It should just not make articles any less accessible in a serious way – like make them harder to link, or exclude them from search results, or such. After all, if I’m writing or editing an article on an actually notable subject, such as the Ruby programming language, why should I not be able to put in a link to _why for anyone who wants to know more about his role? The only reason I can see for ever actually deleting an article is if it’s spam.

Then the deletionists can have their Notable Wikipedia playground and the rest of the world can use Wikipedia for what it does best, that is, be a repository for any and all trivia and factoids, no matter how obscure, that you might ever want to find out about.

My paper as will be presented at this conference is now available from my website here for those that are interested.

When I have been submitting to IEEE conferences my papers have been coming back as failing format checks due to not all fonts being embedded. Anyway I had a quick look and hacked it originally. I took my PDF file from Linux, rebooted into Windows and then used CutePDF to print my PDF back to PDF but with embedded fonts.

Well I thought I'd get this fixed for two reasons - I don't like hacks as I want to know what caused it, and I also don't want to keep rebooting between operating systems.

As it turns out when generating PDFs on Linux Ghostscript embeds all fonts except the standard 14. So I found out how to fix this and documented it here. Unfortunately this didn't work out straight away so just fired up gvim on the pdf file to examine the source. Turns out that I had embedded graphs as pdf so regenerated all those and then it worked out fine.

I've been accepted for the IEEE FMN workshop and so Jayne and I will head to Wales. We've both got Welsh ancestry so are quite pleased about it.

I'm also pleased for my PhD as getting accepted at such an event helps heaps for being examined (when I finally finish writing up)

Day 7 began with a stroll around Battery Park and Castle Clinton, trying to buy Statue of Liberty tickets.  Turns out we (well, I) had underestimated the popularity just a little - by 11:30am, they had stopped selling tickets with monument access, meaning we would only be able to wander around the island.  We shelved that for tomorrow, and headed back out.

Where there's tourists, there's a buck to be made, and a very friendly old Chinese chap named Sam convinced us (well, Fern) to sit for a sketch. The price starts low but rapidly rises when he starts adding second people and a frame (which is required for a charcoal sketch), so some haggling is required.

We went back home to drop the sketch off and have a break, and contemplate our next move.  Fern took some photos out the window, of the back of the building we were staying in - when they get around to tearing it down, it will be worth some serious money. It's not much to look at at the moment though! As you can see though, it's very close to the Empire State Building, but we won't get there today. Instead, we took a trip uptown to Columbus Circle, at the bottom-right corner of Central Park.

Columbus Circle is the home of Time Warner/CNN, and also the Trump International Hotel and Tower, made famous by The Apprentice (back when it was good).  From the corner of the park, hawkers try and sell you pedicab (bicycle rickshaw) rides, or just hire you a bike.  We tried to find a New Yorker to take us around the park, but we (well, I) got suckered in by a pushy Nigerian gent, who did us a demon deal on a trip around the park.

Central Park is iconic, and we were treated to famous sights from dozens of movies I hadn't seen.  Our guy treated us to the story of how the producers of Friends rebuilt this fountain in LA, to film the splashing around in the intro: turns out it's complete crap, and the Friends fountain looks nothing like it. He also told us how Manhattan got its name, and its reputation: Manhattan schist is a shiny rock, which is well suited to laying the foundations for skyscrapers in.  After the Friends story, it's probably crap.

John Lennon was shot not far from here, and ceremonial flames still burn outside the building he lived in.  Yoko Ono raised a large sum of money and dedicated a corner of the park to Lennon's memory, which houses the famous "Imagine" mosaic, to which flowers and memorials are laid every day by well wishers.  (And today, a ukelele).

We walked back around the Bethesda Fountain area, where small musical groups had to battle for attention with larger musical groups. There's no contest really, when the audience includes a guy who clearly put on his robe and wizard hat...

Earlier in the week, we (well, Fern) had been suckered into buying tickets to a comedy show at Stand Up NY (standard Times Square street-hawker routine: it's where the comedy segments in Seinfeld were filmed, Chris Rock plays there all the time, etc etc). The cover isn't extravagant, but they impose a $15/person drink minimum, which thankfully they didn't enforce in the 1/3rd full club.  I can't say it was particularly memorable - there were a number of comedians on the bill, and there was about one who was funny.  They didn't watch each other's acts, in the most part, so the poor Australian guy near the stage got asked who he was and what he did by each guy in turn.  I've seen much better comedy even here in Kitchener.  But, when in Rome, they say.  I was more impressed by the pizza store next door, with photos of the owner and many famous people beforehand.  And less impressed still with the broccolli pizza.

Take a virtual tour of Central Park in the Day 7 gallery.

A couple of our clients have HP AiO1200 iSCSI systems. These are nice enough units, especially for entry level iSCSI SANs. They’re in a slightly modified HP DL320s chassis, and run Windows 2003 Storage Server, as well as some custom built HP management tools.

I’ve never had an easy run when dealing with their iSCSI target and the open-scsi stack used by Citrix Xenserver. The first problem I had was that the management tools don’t support multiple initiators connecting to the same target LUN. They don’t actually stop you, but it seems that you need to hold your tongue just right to allow multiple initiators connecting to the one target. If you don’t hold it just right, the admin tools will let you do it, but it won’t actually work.[1]

The second problem I had is that Xenserver just refuses to connect, saying “Your Target is probably misconfigured”. There’s not really a lot of configuration you can do with an iSCSI target, so I’m perplexed here. Digging deeper:

It seems that iscsiadm can see everything fine!. I tried adding the target via the cli:

Error code: SR_BACKEND_FAILURE_107
Error parameters: , The SCSIid parameter is missing or incorrect, \
?xml version="1.0" ?>
iscsi-target
LUN
vendor
HP
/vendor
LUNid
0
/LUNid
size
42949672960
/size
SCSIid
360003ff646c289389ea2e31c1d419930
/SCSIid
/LUN
/iscsi-target

Unlike the error messages you get in the GUI, that one is quite helpful[2] It tells us we’re missing a SCSIid parameter, and then lists a SCSIid parameter to try:

And our iSCSI target was happily added. I’m not entirely sure the LUNid parameter is required, this post suggests it isn’t. I found a couple of other posts on the forums which suggest that using the CLI for these tasks should be your first attempt.

[1] Where “work” means “actually let you connect more than one initiator at a time”
[2] Although, being full of less than and greater than signs, doesn’t want to display nicely in wordpress. So it’s a bit sanitised

I’ve used a few hardware based iSCSI stacks for Xenserver shared storage backends, but never had spare hardware to run up software based stacks. This is rather backwards from the usual way people would test things I guess, but it’s how it worked out for us.

However, we’re now getting some new hardware for internal use - a couple of frontend servers and a storage server apparently, and we’re going to use a software based iSCSI stack on it. We’ve had a look at some of the commercial offerings - SANmelody, Open-e etc, but I’d much rather not spend money where it’s not needed. This iSCSI backend is going to have one or two LUNs, shared to a static number of hosts, and that’s it.

I’d steered away from the various open-source iSCSI target stacks, because it wasn’t clear whether they supported multiple initiators to access a single LUN concurrently. This surprised me somewhat - it seemed like it should just work, however we kept getting caught by people asking about the “MaxConnections” parameter for IETd, which sounds like it means “Maximum number of initiators to this LUN”, and has a rather depressing note beside it stating that the only valid parameter is “1″ at this stage.

This didn’t sit right with me though - surely there are lots of people using fully opensource iSCSI systems. All the talk about iSCSI being a cheap (free!) SAN alternative can’t just be referring to people consolidating disks but still allocating single-use LUNS. I’ve found lots of references to people even talking about using software iSCSI targets with Xen as a shared storage backend.

And, of course, it’s not right[1]. The IETd “MaxConnections” parameter refers to the number of connections a single initiator can make with respect to a single target, which boils down to whether multipath IO is supported via the iSCSI stack or not. And it’s not, as far as IETd is concerned. This post to iscsi.iscsi-target.devel clears things up quite nicely, but it took me a damned long time to find. So, hopefully, this will help someone else answer this question.

1) multiple ini access different targets on same iet box at same time.
no data concurrency issue. the performance totally depends on your HW.
of course, IET can be improved to support large # of ini better

2) multiple ini access same targets on same iet box at same time. has
data concurrency issue here, so need a clsuter file system or similar
system at client side to coordinate.

3) one ini access different targets on one iet box. it will create
multiple sessions and no data concurrency issue here. performance issue
depends on HW.

all these are MS&OC/S (Multiple Sessions& One Connection per Session)

4) one ini access same target on one iet box.

it might try to use multiple connection in one session (MC/S, Multiple
Connection per Session), but iet doesnot support it and in parameter
negotiation, iet stick to MaxConn=1.

it might try to create multiple sessions with same target (still one
connection per session), which is allowed. usually this is controlled by
client software, for example, linux multi-path.

I read “multiple ini[tiator] access same targets on same iet box at same time” to mean exactly the problem I’m looking at, and the only cavaet is the filesystem issue, which Xenserver deals with. And it clarifies the point about MaxConnections too.

[1] That said, I haven’t tested this properly yet. I ran up IETd on a test box and connected OSX and linux to it concurrently, but while I could format the disk via linux I couldn’t mount it for some reason. OSX saw it fine. I’m not sure if this wasn’t just some transient weirdness on my test boxes or not.

UPDATE: Matt Purvis emailed me to confirm that it does all work as expected. Thankfully. I hope other people find this post useful - if only because it means I’m not the only one that spent hours trying to find a definitive answer on this topic.

I just saw some good advice on choosing passwords posted on the main Google weblog. It made me feel guilty for a moment. I have, at last count, about 140 accounts on a bewildering variety of services. A dozen of them have strong passwords, the rest share one of a handful of weak passwords. I would like to do better than this and have, over a period of time, tried to work out variously elaborate schemes for how to assign passwords.

I haven’t implemented any of them.

Many of them turn out to be impracticable as soon as one starts trying to implement them, but much more mundanely, just changing 140 passwords is a huge heap of work: if it takes 5 minutes to change one password on average, that’s almost 12 hours. Realistically, it would take almost a week to work through the entire list. This means that coming up with a few different password assignment schemes and trying them out in sequence is a non-starter.

In the meantime, despite the absence of such a scheme, I still have a moderately frequent need to log into some of these accounts from computers where my (encrypted, with yet another passphrase) password list file is not available. And thus, I continue to use weak passwords.

I think it is clear at this point that I need to forgive myself for my bad security practice and stop feeling guilty. The problem is clearly not with my willingness to be a model netizen. It is the very concept of passwords that is flawed.

This is far from an original thought, of course. But today, as the faint pang of guilt came and passed, I wondered what we are collectively going to do about this. Passwords are appealing in the small for many reasons (the main ones I see being no prerequisites for participation from users and some extent of plausible deniability for system creators). None of the alternative proposals seem to scale down to the same extent. Worse than all of this, none of the alternatives even try to address the problem of migrating a huge number of accounts away from weak passwords.

This needs fixing. And the only viable solution will be a brutally simple one – as simple as passwords are brutal.

The footprints of the Rockefeller family are all over New York City.  They purchased 17 acres of land in downtown Manhattan in the late 1940s, and donated to the City, who then donated it to the world in the form of the United Nations.

A tour of the UN leads you into the actual rooms where war is declared (or isn't), shows you gifts from its member countries, usually in the form of interpretative works of art, and contains portraits of bodacious people. It even shows you the obligatory Nobel Peace Prize. (What, you don't have one?)

Even though it has its own security force, the NYPD look after the border, which is guarded by Jon and Ponch.

We found a nice place in Café Metro to have lunch, and Det. Stabler from SVU wandered past the window. This wasn't entirely unexpected, as the neighbourhood was plastered with signs suggesting that the show was shooting, and trailers blocking off streets. Unfortunately, we couldn't find him again, which means we're not very good stalkers.

Instead, we took a little more time to look around the famous Grand Central Station, which we last visited on Day 2.

Next stop: Wall Street, via Trinity Church.  You could see a lot more here on September 10, but these days you can only wander around the outside of the stock exchange, under armed watch. You can still go into the Federal Hall National Memorial, which is run by the Department of Conversation (or National Park Service as they're called here). It is the place where George Washington was inaugurated, and the Bill of Rights was signed.

There wasn't much to see at the World Trade Center site - not much of a memorial or anything.  Saw a Jewish guy on a skateboard, though.  Of more interest to people in the area is Century 21, a discount department store.

With a big day under the belt, we wandered down towards the South Street Seaport, to have a look around. From here you get a good look on the Brooklyn and Manhattan Bridges.

There are a great number more photos on the day's gallery.

The American Museum of Natural History is full of mystery and wonder. Or at least, big-ass turtles and dinosaurs!

Cool things we found in the museum:

• Boobies (really)
  
• Pieces of home
• A crazy lady, who thought she was a bird
• Walrus (note: no bukkit)
• The Spitzer Hall of Human Origins. (Those that gave the hall also gave the world ex-Gov. Spitzer, who, according to the news, knows a thing or two about where humans come from)

There were more shiny rocks than you could shake a stick at.

A must-do while in New York is a Yankees game. This is the final year for the famous Yankee Stadium, workplace of George Costanza, with its famous bat-shaped-chimney. They're building a new stadium in a park across the street, and then demolishing the old one to replace the park!

I like baseball, because it's a little like Twenty20 cricket. Except it's a lot colder. Dinner at baseball is hot dog and fries (it's the rules), which requires up to 20 minutes of hard core queuing. The Yankees won 6-1 and we headed back home by the subway.

In today's gallery, we bring you lots of pictures of shiny things and action baseball shots.

Today I decided to upgrade to Vista SP1. I decided that since I am GM at one of NZ's largest IT companies (Datacom) I owe it to my staff to test some of the painful stuff.

Anyway my machine is dual boot Vista and Ubuntu 7.10 and it had bits of Windows Server 2003 lying around in the bootloader as well from an earlier install. I had removed Server 2003 ages ago as it fought badly with Vista - probably the way I set it up and after that I had also rearranged and extended/shrunk different partitions. In other words it was hanging together with #8 wire. Of course I expected Vista SP1 to install seamlessly!

Anyway I have been reading up on SP1 for a while now and came across this article at APC. I then read this article from Microsoft about potential SP1 blockers and saw that my Intel 965 video card was a bad version. So ran Windows Update, found it as an optional update. Windows Update said it installed it AND it failed (said Installation Failure: Windows failed to install the following update with error 0x80070103: Intel Corporation driver update for Mobile Intel(R) 965 Express Chipset Family.). But my video driver was updated to a February 2008 version that was in the range Microsoft specified so that was fine. I also checked my Sierra Wireless card and it wasn't the model mentioned.

I also read this article from APC that explains SP1 won't install on dualboot systems due to BitLocker even if you don't have it installed. So I went and installed the latest version of EasyBCD. I already had this on but latest version has SP1 support. I then backed up my configuration in this and reinstalled standard Vista MBR (which would remove grub for Ubuntu I presume).

I turned off my antivirus and I installed Vista SP1 from Windows Updates and it was a 66 MB download as already had other updates on. This all seemed to go fine and it chugged away for about 40 minutes and rebooted into Vista sans grub. All fine - Vista SP1 worked perfectly.

And then I killed my system :-( I decided to use EasyBCD to restore the settings. There were a couple of problems here - one was that I backed up into c:\boot and then when I went to restore it wasn't in the file chooser. That was easy - just manually typed it in at top of file chooser dialog. The second was that I said remove Vista MBR. It said that this would enable XP. I shouldn't have done that really as it did more than remove Vista MBR as I would find later.

Anyway I went to reboot and it gave me a boot menu choice for Windows Server 2003 and Vista Business. Neither option worked - Server 2003 was no longer on my machine and Vista didn't work. At this stage I briefly panicked.

Anyway I then decided I would first attempt Ubuntu working and then Vista.

So I booted using Ubuntu 7.10 CD and typed the following from a shell prompt:

• sudo -s
• ls - l /dev/sd* (to try and find my partitions)
• mkdir /a
  
• mount /a /dev/sda6 (I had to try a few until I got the right one)
• grub-install --root-directory=/a /dev/sda (note the drive here, not the partition)

I then rebooted and my Ubuntu was all back to normal. At least I had a working machine now. My Vista appeared on the grub menu and selected it but the machine just locked.

Back into Ubuntu and looking around I read this support page from EasyBCD. So I downloaded the Vista recovery DVD by going to this page.

Ran that, selected install, and then repair and Vista worked fine. What's more it didn't blow away Grub which really surprised me.

So now have a fully functioning system again. What a relief.

Go the Commerce Commission. Good to see the news here today (from Stuff).

Just reading an interview with Fedora project leader. Quite interesting. Also saw mention of FreeIPA project that Fedora/RedHat have started that is tying together Kerebos/Samba etc to compete with Active Directory. Quite interesting. Also mentions speed up of X. I also like their focus on pushing things upstream - something I believe in strongly.

From the articles I'm seeing at present I get the feeling Fedora is on the rise.

Larry Wall:

It will always be too early, and too late. There will always be reasons not to do it till next year, and reasons you’re hosed because it wasn’t done years ago.

We needed some groceries, so we wandered around our adopted neighbourhood a little in the morning. On the way to the Korean Supermarket on 32nd St we found Jim Hanley's Universe, a Mark-1-on-steroids where Fern could relive her childhood through Tintin Asterix. Every other shop in the neighbourhood is a souvenir store, but this one wins for most statue-ettes.

Some things at the grocers were a little reminder of home, and others were just a little strange!.

I thought the British liked queueing, but this beat it hands down.  We never found out why. At least they all had Bubble Guy to entertain them.

Greenwich Village, home of the rich and famous, location of Friends, although I never saw Monica or Ross picking up a copy of the PKWARE user manual from a street vendor!  We walked around the outside of the NYU campus, around Bleecker St, the home of music and culture immortalized in song by Paul Simon. I particularly respect their order of national prioritisation!

The Village is the home of the narrowest building in Manhattan, at 2.9m wide (number 75½), and next door at 77, the oldest building in the borough. Even regular width houses are hard to send mail to here. Parking is difficult also.

It's also home to a loud and proud gay district, which, as always, has its finger on the pulse of culture - only one week after the event.

We thought we'd check out the TKTS booth down in Times Square, and got half price tickets for the Monty Python music "Spamalot" literally 2 minutes before it started. Last minute tickets are invariably in the nosebleed section, but we enjoyed the show nonetheless. When Spamalot first premiered it was full of talented actors Tim Curry, Hank Azaria and Niles from Frazier.  All we got was Clay Aiken from American Idol. Afterwards, he attracted a large stage door audience (and was probably the owner of the limo), but we left before he did.

The counterfeit bag crew were out in Times Square also, but we managed to make it past the throngs of crazed bargain-hunting women and back to the subway toward home.

Breakfast, then off to the subway to head down to Chinatown.

The back streets of Chinatown are full of childish laughs, arguments, odd looking food and things to buy.

I was on a mission to find a plate of kwai teow, the fantastic food described as "cholesterol on a platter". Possibly my favourite food ever. Not a lot of hope, before finding a Singaporean restaurant which served me up this piping hot platter of awesome.

Canal St, the main street of Chinatown, is full of counterfeit everything. While one store was noticeably shut down, there were dozens more where that came from. Guys on the street offered to sell you Rolexes and iPods, and every store was full of sunglasses, perfumes and handbags. Dolce and Gabbana seemed the brand of choice - all manner of brandless bags were labelled with the same little triangular logo, and no-one hid those. If you knew what to ask for, they'd let you go out the back and look at the Chanel bags, but asking for Louis Vuitton got you looked at suspiciously. We had to take two attempts, with a little Chinese lady telling us "no pictures! $10 for picture!", to sneak this Kanye West inspired photo.

From here, we headed down to Battery Park, grabbed sweet refreshment, and stared out at the lady of the harbour. From the free ferry out to Staten Island, one of the five boroughs that makes up New York City, you get a great view of the Statue of Liberty and boats on the harbour. (And birthday girls.) The return trip gives ample opportunities for photographs of the Manhattan skyline.

Day 3's gallery has many more harbour shots.

Day 2 started late, and we went to the famous Dining Concourse in Grand Central Station for lunch - or, seeing as I'd managed to pick up a nasty and painful sore throat a few days before leaving, a soothing smoothie.

The Metropolitan Museum of Art is on east side of Central Park, near the famous reservoir. The outlook of the day was a little bleak, but still impressive.

Entry to the museum is by donation, and you can pay as much or as little as you like. However, we didn't know you actually did have to pay something, because we pretty much just walked right in! Apart from the standard wing of paintings, the museum has a lot of interior design, sculpture, armoury, musical instruments, and other forms of art. I spent a lot of time looking for the unicorn tapestries, having seen copies being traditionally made at Stirling Castle last year. I eventually found out that these were at the Cloisters, a gallery of medieval art we would have to see another day.

Personal highlights of the Met:

• an Egyptian tomb (and the entire Stargate mythology)
• Decorative stairs - strange...
• 14th century Backgammon
• Very early music manuscripts
• Obligatory Jesuses
• A pig thing
• Naughty statues
• Another Jesus Dali - this one a bit more space-age. (I'm collecting them for Tony).
• The only known Apollo Lyre
• The first piano, ever. (Previously they were harpsichords.)
• A Stradivarius
• A buddy Christ'ing African statue
• Artistic androgyny - one is a boy, one is a girl. Can you tell which?

(As you can tell, I have wicket good descriptive arty lingo.)

I love the description plate that comes with this piece of art. Puts "oil on canvas" to shame.

Day 2's gallery is full of art.

Fern's birthday was in early April, and she had always wanted to go to New York City. What better birthday present than ten days in the big apple?

First, the logistics: driving to Buffalo, NY takes about 2½ hours, and shaves several hundred dollars off the cost of flying. A couple of JetBlue tickets (a low-cost US carrier) ran about $200 each, including tax, compared to the ~$600 each Air Canada would have charged out of Toronto.

Accommodation was an interesting dilemma, also neatly solved by the Internet. We found a listing on Craigslist put up by a woman named Lamia, who wanted to rent out her apartment while she was away for 10 days - almost exactly the 10 days I had wanted to take off - and made quick arrangements by e-mail, followed by slow arrangements by money order and post.

We had a little problem driving across the border. You know when you pass through the US, and get given one of those little green cards, part of which you're expected to keep? No-one ever explains why, or what it actually does. It's a visa waiver, which lets you enter the country without explicitly being issued a visa, and it's valid for 90 days. If you re-enter the country less than 90 days after leaving it, and don't have the green card, you're in trouble. They actually make you pay $6 for its re-issue after they're convinced you're not going to be bad. Hint: don't say "we've rented an apartment", because even though you mean "for 10 days", they hear "forever".

Buffalo Airport was pleasantly simple - you still get the checks on US domestic flights, but they'll let you on even with just a strange looking drivers' license, The flight was peaceful (the seat-back TV, instead of being on-demand, was live satellite TV, so I watched a little Battlestar Galactica and followed the US election on Faux News) and after just over an hour, we arrived at JFK.

In a complete contrast to Toronto's useless transit, JFK has a simple link with the NYC subway. Strangely, you pay $5 for the 10 minute AirTrain ride from the terminal to the subway station, and then $2 for the 45 minute subway ride into Manhattan.

The subway was stereotypical - by the time we got to the line we needed, both carrying suitcases and carry-on bags, we were abused by New Yorkers for taking up too much space. After a couple of changes it was a two-block walk to the apartment.

Lamia's room-mate is a painter named