One of the incidental consequences of having the Squeezebox 2 — a recent acquisition which I’ll have to describe in detail another time — in the bedroom is a sudden blinding awareness of the craptacular nature of PC clock hardware.

I should explain.

The Squeezebox 2 is a very clever device intended to act as an interface between your stereo and your massive digitized music collection. Connect it to your receiver via RCA cables, connect it to your computer — running the server software — via an Ethernet cable, and you’re off to the races. It sports a dimmable, eminently-legible vacuum-fluorescent display, comes with a thoughtfully-designed remote, and has no moving parts of its own, making it utterly silent and thus something you can put on your nightstand without fearing that it will trouble your dreams.

Part of its cleverness lies in its reliance upon the power and versatility of general-purpose hardware, running easily-modified software — the thing on the far end of the Ethernet cable, in other words — to do the heavy lifting. It has just enough intelligence of its own to ask someone smarter for help.

It never really shuts off, either, unless you pull the plug, which allows it to do the gratifying trick of starting up instantly when called upon. Instead it goes into, at best, a light doze when you hit the Power button on the remote. It can, depending upon your preference, be configured to do any number of things while snoozing, from playing convincingly dead to displaying RSS feeds to showing a clock.

This brings us back to the approximate neighborhood of the original point. In keeping with its “let the server do the work” philosophy, the Squeezebox 2 doesn’t actually keep time on its own, being wholly dependent instead upon the server’s clock. It so happens that I already have a clock on the bedroom wall — an Oregon Scientific unit that tells the temperature and synchronizes itself to WWVB nightly — and thus it is that I conspired to present myself with inescapable evidence of just how awful a job the PC does of keeping time without help.

When I first plugged in the Squeezebox 2, I noticed that its clock was eight or so hours off. “Right”, thought I, realizing that the server’s clock had never been properly set, and proceeded to build and run ntpdate pool.ntp.org so as to prime the hourglass. The Squeezebox 2’s clock and the wall clock were in perfect sync, and I was happy — except that next morning I woke up and noticed that the Squeezebox 2 had gained about six seconds on the wall clock. By evening that amount had reached ten seconds.

Ten seconds. In a day!

Fine. Time to go whole hog and set up ntpd to run long-term so as to determine and eventually correct for clock drift. Only it turned out that ntpd never seemed to get around to calculating a reasonable drift value, let along correcting the clock. Eventually, after recompiling the ntp package with the debug flag set — easy, thanks to Gentoo and Portage — and quite a bit of confused poking around, I figured out what the problem was. My ntpd.conf contained the notrust directive as a parameter to the default configuration, meaning that the daemon rejected each reply it was busily soliciting for lack of cryptographic authentication. Brilliant. (To be fair, Portage tried to warn me about this at build time. Alas, that warning went the way of all Portage warnings, off the far end of stdout. That’s something else I need to correct, and soon.)

With that fixed, ntpd seems to have finally figured out what time it is. At some point I’ll go back and figure out if there’s a way for it to communicate securely with the servers in pool.ntp.org, at least. (Still and all, I have to admit that “spoofed time” is not foremost on my list of personal anxieties.)

Morals of this story:

1. PC hardware sucks on an intrinsic design level, irrespective of manufacturer or vendor. (“Sure, it sucks, but it’s an industry-standard sucking!”) I have cheap plastic made-in-China quartz clocks that do a better job of keeping time.
2. pool.ntp.org is your friend. It’s really nice to be able to specify a server name in your ntpd.conf without having to feel like a leech lest you approach the server administrator cap in hand.
3. ntpd is your friend. Sort of. It does the job admirably once it’s configured, but good Lord, could its error reporting use some work. Anything that can spend several hours busily querying the world at large for the time of day, only to consign every answer it receives to the bit bucket for lack of a cryptographic signature, without making so much as a peep in either the system log or its own, has… self-expression issues. Perhaps this is controllable via an ntp.conf configuration directive or ntpd command-line argument, but if so, I haven’t found the trick of it. Which brings me to my next point:
4. The NTP documentation is not your friend. All honor and respect to David Mills for his perseverance in creating and maintaining a system that does a more-than-decent job of synchronizing clocks over a variably-laggy packet-switched network, but someone needs to sit the man down and explain the basics of good manual design to him.

   One: scrap, for God’s sake, the Tcl/Tk-inspired color scheme. Two: ditch the Pogo cartoons. Cuteness is tolerable when your documentation is otherwise up to snuff. When this is not the case, cuteness is nothing but an aggravating reminder to your reader that your attention was elsewhere than where it should have been. Three: focus on making the package as a whole simple, straightforward, and accessible. For openers:

   • Make sure it’s sanely searchable. That means one or more of the following: indexing the site with your own search engine; paying the few bucks a year to buy your own domain so that searches using Google’s “site:” keyword work properly; just flattening the entire documentation down into a single page so that I can use my browser’s built-in search facility to scan the whole thing. Under no circumstances should I have to open the five different subsections of the ntpd manual, each on a separate page, in order to track down the description of the configuration directive I’m looking for. (Of course, if you just had something as radically newfangled as an index for configuration directives, or even just a coherent organization for said directives, maybe I wouldn’t have to rely upon searching so much.)
   • Try to make it reasonably portable. HTML is nice and even justifiable if you’ve got a lot of cross-references, but if you’re going to bother providing a man page at all, make sure that the meaty bits haven’t been badly truncated by your HTML-to-man conversion. (This may in a sense be the distribution’s fault rather than yours: maybe you didn’t provide a man page at all, and they just did the best they could with what they had. This doesn’t really get you off the hook, though. Failure to provide man pages is itself a capital crime, or should be. The fact that you’re not alone in comitting it — the GNOME folks come to mind, as does the GNU project’s dedication to info pages — does nothing to exonerate you either.)

And I thank you, Lord Almighty up above,
Just for sending down the ‘F’ train to me.
— Mike Doughty,
“Thank you, Lord, for sending me the F train”

Sketches from the notebook of a man walking around in a daze, trying to come to terms with just how hosed he is in terms of the data he has suddenly lost access to:

I left reiserfsck running overnight, having disabled DMA for the wounded drive. (Since reiserfsck failed with lots of ominous-sounding warnings about DMA timeouts and lost interrupts, it seemed worth a shot. Of course, it slowed the process by about a factor of ten — hence the “overnight” part.) In the morning, no joy.

I had to run some errands this afternoon, partly in preparation for John and Jody’s wedding later this week. While I was out and about, I picked up a low-end router, a D-Link DI-604, so that I could restore some reasonable level of network connectivity to the apartment. (I’d have gotten a DGL-4100, but no one had them in stock. It’s probably just as well.)

Walking through the software section at Fry’s, and glancing at the various data-recovery programs on display there, reminded me of the existence of a tool I hadn’t thought about in a while, and had never had occasion to use before: Steve Gibson’s SpinRite. I resolved to look at it more closely once I got home and had re-established web access.

When I got home, I plugged in the little D-Link router, and realized that I should have acquired it, or something like it, a long time ago. Having your own highly-tweakable Linux-based router is nice, but having a foolproof, solid-state box as a backup makes for amazing peace of mind. I will go back to the Linux-based approach in short order, of course, but it will be nice to know that the D-Link is waiting in the wings, ready to pinch-hit the next time I find myself having to juggle hardware. Being able to browse the web for tools and tips did wonders for my peace of mind, to say nothing of having the phone working again.

The first thing I did was scrutinize SpinRite, which looked sufficiently promising that I decided to try it. (Possible salvation for $89 a pop? I’ll take that action!)

Before loosing SpinRite upon the drive, I thought I’d let reiserfsck --rebuild-tree have one last crack at it. In the morning, after the unsuccessful overnight reiserfsck attempt, I had noticed that the drive was a bit dusty, and that some of the dust appeared to be lodged under the integrated-controller PCB. Having a screwdriver nearby, and too much time on my hands, I unscrewed the PCB and blew it clean with a few blasts from a can of compressed air before reassembling the whole thing.

Desperate. Pathetic. Everyone knows that that sort of blind, ritualistic hardware voodoo never does any good.

Except when it does. Except when it does. Because this time around, reiserfsck --rebuild-tree plowed right past the point where it had stopped dead during the five previous recovery attempts, and left me with a successfully-rebuilt and mountable filesystem. I have no idea what convinced the drive to venture back from the sunless lands — for all I know, it was knocking it about eight inches to the carpeted floor when I bumped the box I’d rested it on — and I’m not inclined to care. I’m just grateful for the undeserved third chance I’ve been given. (Yes, tar is chugging away as I write this. I may be stupid, but I’m not criminally stupid.)

All the files I really care about have already been backed up to another disk; I’ll be burning them to optical media in the morning, just to be safe. At this point, I’m down to saving the data I could afford to lose, but would rather not have to recreate. Having paid for SpinRite, I find myself not needing it at the moment. I am utterly unconcerned. I’ll probably unleash it upon the old disk once the backups are complete, just to see what it finds and reports.

I am insanely lucky. My father is fond of saying that it is better to be lucky than good, but I will strive not to push my luck quite so aggressively in the future. Next up: an actual backup strategy.