Stevey's Blog Rants

Hacker News Fires Steve Yegge

2011-07-27T12:33:00.001-07:00

I woke up this morning...ish... to discover that Hacker News had finally had enough of me being at Google, so they forced me into early retirement.

On Monday I was honored to be able to deliver a keynote talk at OSCON Data. In the talk, I announce at the end that I am quitting a project that I had very publicly signed up for, one that I am not passionate about and don't personally think is very important to the human race. Though others clearly do, and that's a legitimate viewpoint too.

But the power of suggestion can make you see and hear something entirely different. If, for instance, someone tells you that I gave the talk wearing a gorilla suit, then when you watch it, I will magically appear to be wearing a gorilla suit. It's actually a gray jacket over a black shirt, but you will perceive the jacket as the back-hair of a male silverback gorilla! And to be honest the talk could have benefited from the judicious application of a gorilla suit, so no harm there.

Similarly, if someone on Hacker News posts that "Steve Yegge quits Google in the middle of his speech" and links to the video, then you will watch the video, and when I say the word "project" at the end of my speech, a magical Power of Suggestion Voice-Over will interrupt -- in a firm manly voice totally unlike my own quacking sounds -- with "Gooooooogle". And then you will promptly sink into a 15-minute trance so that the voice-over can occur in the middle of my speech where Hacker News said it happened, instead of 96.7% of the way through the talk where it actually happened.

I am going to harness this amazing Power of Suggestion, right here, right now. Here goes.

You are going to come work at Google! You are going to study up, apply, interview, and yes, you are going to work there! And it will be the most awesome job you've ever had or ever will have!

I hope for your sake that this little experiment works, because Google is frigging awesome, and you'll love it here. And they'll be happy to have you here. It's a match made in heaven, I'm tellin' ya. It might take you a couple tries to get in the door, because Google's interview process -- what's the word I'm looking for here -- ah yes, their process sucks at letting in all the qualified people. They're trying to get better at it, but it's not really Google's fault so much as the fault of interviewers who insist that you're not qualified to work there unless you are exactly like them.

Of course, there are interviewers like that wherever you go. The real problem is the classic interview process, which everyone uses and which Google hasn't innovated on, not really. It's like deciding whether to marry someone after four one-hour dates that all happen on the same day in a little room that looks kind of like a doctor's office except that the examining table is on the wall.

The reason I haven't been blogging lately is that working at Google is so awesome that I just don't feel like doing anything else. My project is awesome, the people are awesome, the work environment is over-the-top-crazy-awesome, the benefits are awesome, even the corporate mission is awesome. "Organize the world's hardline goods in little brown boxes delivered straight to your doorstep" -- that's an awesome mission, yeah?

Wait, sorry, that was a flashback to the Navy or something. "Organize the world's information" -- that's the one. It's a mission that is changing the course of human events. It is slowly forcing governments to be more open, forcing corporations to play more fairly, and helping all of us make better decisions and better use of our time.

In that vein, the part of my brain that makes Good Decisions was apparently broken a few weeks ago, when I allowed myself to be cajoled into working on something that I wasn't passionate about. I am an eternal optimist, and I figured I could teach myself to be passionate about it. And I tried! I spent a few weeks pretending that I was passionate about it -- that's how I got through my Physics classes in college with A grades, so I know it's a mental trick that can sometimes work.

But then I wrote my OSCON Data speech, in which I basically advise everyone to start working on important problems instead of just chasing the money. Or at the very least, go ahead and chase the money in the short term, but while you are doing that, prepare yourself to help solve real problems.

And after writing the speech I realized I'd completely failed to follow my own advice. I'm getting old and I only have so many "big projects" left that I can actually participate in. So in my mind it's a complete cop-out for me to take the easy path and work on a project that my company is excited about but I am not.

Now, as it happens, I am in fact working on a very cool project at Google. It's not important in the same sense that curing cancer or getting clean water to impoverished cities are important. But it's a project that has the potential to revolutionize software development, and NOT through some new goddamn dependency-injection framework or web framework or other godawful embarrassing hacky workaround for a deficient programming language. No. It is a project that aims to turn source code -- ALL source code -- from plain text into Wikipedia. I've been on it for three and a half years, and I came up with the idea, and the team running with the idea is fantastic. The work may not be directly important, but it is an enabler for important work, much like scaling infrastructure is an enabler.

So I am happy to continue working on that project for now. Yes, at Google. I may even blog it up at some point. But I'm very serious about brushing up on my math and statistics, some of which I haven't applied directly in 20 years, and start focusing on machine learning problems. Particulary, if I may be so fortunate, the problem of curing cancer. I may not be able to participate directly for a few years, as I need to keep working and paying the bills just like you. But I'm studying hard -- I started up again a few days ago -- and I've demonstrated to myself quite a few times that if I do anything daily for a few years I can get pretty good at it.

Anyway, I'm late for work. Isn't that nice? I like the sound of it. It has a nice ring to it: "I'm late... for my job."

So come work with me! Unless you are curing cancer, of course.

eBay Patents 10-Click Checkout

2011-07-22T03:36:00.000-07:00

San Jose, CA (Reuters) — Online auctions cartel eBay (NASDAQ: EBAY) and its collections and incarceration arm PayPal announced that on July 21, 2011, the two companies had jointly been awarded United States Patent No. 105960411 for their innovative 10-click “Buy it Now” purchasing pipeline.

The newly-patented buying system guides users through an intuitive, step-by-step process of clicking “Buy It Now”, entering your password, logging in because they signed your sorry ass out again, getting upsold shit you don’t want, continuing to your original destination, accepting the default quantity of 1 (otherwise known as “It”), committing to buy, clicking "Pay Now", entering a different password than your first one, clicking "Log In" again god dammit, declining to borrow money from eBay’s usury department, reviewing the goddamn purchase details since by now you’ve completely forgotten what the hell you were buying, and finally confirming the god damned payment already.

The 10-click checkout system, known colloquially as 10CLICKFU -- which many loyal users believe stands for “10 Clicks For You” -- was recently awarded top honors by the National Alliance of Reconstructive Hand Surgeons. 10CLICKFU incorporates a variable number of clicks ranging from eight to upwards of fifteen, but eBay’s patent stipulates that any purchasing system that lies to you at least nine times about the “Now” part of “Buy It Now” is covered by their invention.

The patent award came as a surprise to many analysts, since several of eBay’s related patent attempts had been rejected on the basis of prior art. In one well-publicized filing, eBay had tried to patent a purely decorative, non-operational “Keep me signed in” checkbox, but Sony’s PlayStation Network already had one just like it. And another eBay patent claim for excruciating page load times was rejected because the iPad App Store is still loading.

But eBay’s boldest and potentially furthest-reaching patent attempt was for “100% Inaccurate Button Text”. The invention claim was based on several of their UI elements, but rested primarily on the “Buy It Now” button, which eBay claims contains enough inaccuracies to render it "complete bullshit." Their patent was rejected by the US Patent Office review committee on the grounds that the Firefox browser’s “Do this automatically from now on” checkbox has been complete bullshit for over fifteen years. eBay says they will appeal the ruling because the checkbox is not technically a button.

eBay’s spokesperson Paula Smugworth announced that eBay will continue to innovate on ways to remind their users that monopolies can do whatever the hell they want. “Not that eBay is a monopoly,” she added. “But if we were a monopoly, then we could do whatever the hell we wanted. I’m just sayin’.”

eBay’s stock rose on the news, driven largely by anonymous shill bidders.

Haskell Researchers Announce Discovery of Industry Programmer Who Gives a Shit

2010-12-01T13:16:00.000-08:00

The worldwide Haskell community met up over beers today to celebrate their unprecedented discovery of an industry programmer who gives a shit about Haskell.

On Wednesday, researchers issued a press release revealing that 27-year-old Seth Briars of North Carolina, a Java programmer at Blackwater accounting firm Ross and Fordham, actually gives a shit about Haskell.

"Mr. Briars has followed every single one of our press releases for years," the press release stated. "Probably even this one."

Haskell researcher Dutch Van Der Linde explained how they had stumbled on the theoretical possibility of Briars and his persistent interest in Haskell. "We knew that there are precisely 38 people who give a shit about Haskell," said Van Der Linde, "because every Haskell-related reddit post gets exactly 38 upvotes. It's a pure, deterministic function of no arguments -- that is, the result is independent of what we actually announce. But there are only 37 of us on our mailing list, so we figured there was a lurker somewhere."

"That, or it was an off-by-1 error not detectable by our type system," Van Der Linde added. "But we don't, uh, like to dwell on, I mean with good unit testing practices we can, um... sorry, I need to get some water."

As Van Der Linde stumbled off in a coughing fit, his fellow researcher Bonnie MacFarlane outlined their basic dilemma: "Finding a person who gives a shit about Haskell is an inherently NP-complete computer science problem. It's similar in scope and complexity to the problem of trying to find a tenured academic who didn't have the bulk of his or her work done by uncredited graduate students. So even though we suspected Briars existed, we needed a strategy to smoke him out."

She explained the trap they set for Briars: "We crafted a fake satirical post lampooning Haskell as an unusable, overly complex turd -- a writing task that was emotionally difficult but conceptually trivial. Then we laced the post with deeper social subtext decrying the endemic superficiality and laziness of global industry programming culture, to make ourselves feel better. Finally, each of us upvoted the post, which was unexpectedly contentious because nobody could agree on what the reddit voting arrows actually mean."

"And then we waited to see who, if anyone, would give a shit," she said.

MacFarlane concluded, "Our elegant approach didn't work, so we hired a Perl hacker to go dig up the personal details on all 38 accounts that had ever upvoted a Haskell post, and the only one we didn't know was Seth Briars. So we reached out to him, and thankfully so far he hasn't threatened to sue us."

Briars says he is pleased to have been recognized for his apparently unique shit-giving about Haskell. "I've been giving a shit about Haskell for a long as I can remember. I follow all their announcements and developments closely, just in case I ever get the urge to use the language for something someday."

"It's a beautiful, elegant language," Briars observed as he busied himself cleaning a fingernail. "You'd be hard-pressed to find a more expressive and composable core. And they've made astounding advances over the years in performance, interoperability, extensibility, tooling and documentation."

"I'm kind of surprised I'm the only person on earth who gives a shit about it," Briars continued. "I'd have thought there would be more people following the press releases closely and then not using Haskell. But they all just skip the press releases and go straight to the not using it part."

"People see words like monads and category theory," Briars continued, swatting invisible flies around his head for emphasis, "and their Giving a Shit gene shuts down faster than a teabagger with a grade-school arithmetic book. I'm really disappointed that more programmers don't get actively involved in reading endless threads about how to subvert Haskell's type system to accomplish basic shit you can do in other languages. But I guess that's the lazy, ignorant, careless world we live in: the so-called 'real' world."

Haskell researcher Javier Escuella remains hopeful that one day they may be able to double or even triple the number of industry programmers who give a shit about Haskell. "I believe the root cause of the popularity problem is Haskell's lack of reasonable support for mutually recursive generic container types. If we can create a monadic composition-functor wrapper that is perceived as sufficiently sexy by hardened industry veterans, then I think we will see an uptick in giving a shit, possibly as much as a full extra person."

Haskell aficionado Harold MacDougal is not quite as sanguine as his colleague Escuella. "I doubt Haskell will ever be appreciated by the uneducated natives of this industry. As exciting as it is, the discovery of Briars should be considered an anomaly, and not as a sign that more people will ever give a shit. Programmers only seem to pay attention to things when there is humor involved."

"We do have an experimental humor monad," added MacDougal. "But it doesn't seem to be getting much adoption. Haskell fans just don't see the need for it."

MORE NEWS

Previous article: Perl Community Debating Adding Monads The Perl lists are brimming with discussions about the value of adding monads to Perl. "We don't really know what they do, but it doesn't make sense _not_ to have something in Perl," said Perl hacker Landon Ricketts. Read more

Next article: Microsoft to Introduce Mutually Recursive Error Messages Software giant Microsoft announced today the launch of their new REDRUM platform, an elegant system that allows Windows system error messages to shuffle blame around indefinitely by using continuation-passing. Read more

Wikileaks To Leak 5000 Open Source Java Projects With All That Private/Final Bullshit Removed

2010-07-28T13:40:00.000-07:00

EYJAFJÖLL, ICELAND — Java programmers around the globe are in a panic today over a Wikileaks press release issued at 8:15am GMT. Wikileaks announced that they will re-release the source code for thousands of Open Source Java projects, making all access modifiers 'public' and all classes and members non-'final'.

Agile Java Developer Johnnie Garza of Irvine, CA condemns the move. "They have no right to do this. Open Source does not mean the source is somehow 'open'. That's my code, not theirs. If I make something private, it means that no matter how desperately you need to call it, I should be able to prevent you from doing so, even long after I've gone to the grave."

According to the Wikileaks press release, millions of Java source files have been run through a Perl script that removes all 'final' keywords except those required for hacking around the 15-year-old Java language's "fucking embarrassing lack of closures."

Moreover, the Perl script gives every Java class at least one public constructor, and turns all fields without getters/setters into public fields. "The script yanks out all that @deprecated shit, too," claims the controversial announcement.

Longtime Java programmer Ronnie Lloyd of Austin, TX is offended by the thought of people instantiating his private classes. "It's just common sense," said Lloyd, who is 37. "If I buy you a house and put the title in your name, but I mark some of the doors 'Employees Only', then you're not allowed to open those doors, even though it's your house. Because it's really my house, even though I gave it to you to live in."

Pacing and frowning thoughtfully, Lloyd continued: "Even if I go away forever and you live there for 20 years and you know exactly what's behind the doors — heck, even if it's a matter of life and death — plain old common sense still dictates that you're never, ever allowed to open them for any reason."

"It's for your own protection," Lloyd added.

Wesley Doyle, a Java web developer in Toronto, Canada is merely puzzled by the news. "Why do they think they need to do this? Why can't users of my Open Source Java library simply shake their fists and curse my family name with their dying breaths? That approach has been working well for all the rest of us. Who cares if I have a private helper function they need? What, is their copy/paste function broken?"

Wikileaks founder Julian Assange, who coined the term "Opened Source" to describe the jailbroken open-source Java code, fears he may be arrested by campus security at Oracle or possibly IBM. The Wikileaks founder said: "Today the Eclipse Foundation put out a private briefing calling me a 'non-thread-safe AbstractKeywordRemovalInitiatorFactory'. What the fuck does that even mean? I fear for my safety around these nutjobs."

The removal of '@deprecated' annotations is an especially sore issue for many hardworking Java developers. "I worked hard to deprecate that code that I worked hard to create so I could deprecate some other code that I also worked hard on," said Kelly Bolton, the spokesperson for the League Of Java Programmers For Deprecating The Living Shit Out Of Everything.

"If people could keep using the older, more convenient APIs I made for them, then why the fuck would they use my newer, ridiculously complicated ones? It boggles the imagination," Bolton added.

The Eclipse CDT team was especially hard-hit by the removal of deprecation tags. Morris Baldwin, a part-time developer for the CDT's C++ parsing libraries says: "We have a policy of releasing entire Java packages in which every single class, interface and method is deprecated right out of the box, starting at version 1.0."

"We also take careful steps to ensure that it's impossible to use our pre-deprecated code without running our gigantic fugly framework," the 22-year-old Baldwin added. "Adding public constructors and making stuff non-final would be a serious blow to both non-usability and non-reusability."

The Agile Java community has denounced the Wikileaks move as a form of terrorism. "It was probably instigated by those Aspect-Oriented Programming extremists," speculates Agile Java designer Claudia Hewitt, age 29. "I always knew they wanted to use my code in ways I couldn't predict in advance," she added.

Many Java developers have vowed to fight back against the unwelcome opening of their open source. League of Agile Methodology Experts (LAME) spokesperson Billy Blackburn says that work has begun on a new, even more complicated Java build system that will refuse to link in Opened Source Java code. The new build system will be released as soon as several third-party Java library vendors can refactor their code to make certain classes more reusable. Blackburn declined to describe these refactorings, claiming it was "none of y'all's business."

Guy Faulkner, a 51-year-old Python developer in Seattle, was amused by the Wikileaks announcement. "When Python developers release Open Source code, they are saying: Here, I worked hard on this. I hope you like it. Use it however you think best. Some stuff is documented as being subject to change in the future, but we're all adults here so use your best judgment."

Faulkner shook his head sadly. "Whereas Java developers who release Open Source are code are saying: Here, I worked hard on this. I hope you like it. But use it exactly how I tell you to use it, because fuck you, it's my code. I'll decide who's the goddamn grown-up around here."

"But why didn't they write that Perl script in Python?" Faulkner asked.

MORE NEWS

Previous article: San Francisco Airport Announces That All Restrooms Near You Are Now Deprecated
The SFO port authority announced today that all airport restrooms located anywhere near you are now deprecated due to "inelegance". The newer, more elegantly designed restrooms are located a short 0.8 mile (1.29 km) walk from the International Terminal. Read more

Next article: Eclipse Sits On Man's Couch, Breaks It
New Hampshire programmer Freddie Cardenas, 17, describes the incident: "We invited Eclipse over for dinner and drinks. Eclipse sat down on our new couch and there was this loud crack and it broke in half. Those timbers had snapped like fuckin' matchsticks. Then my mom started crying, and Eclipse started crying, and I ran and hid in my bedroom." Read more

Blogger Finger

2010-07-15T03:06:00.000-07:00

Well! I've sure had a nice relaxing blog-free year. No worries, no haters, no Nooglers wandering by my office and staring at me through the window as if they expect me to crap in my hand and hurl it at them. Not that I wasn't tempted.

Nope, it's just been peace and quiet and reading and coding and practicing my guitar and stuff. It's been awesome.

And now that everyone's completely forgotten who I am, or whatever exactly I'd said that made them feel all butthurt inside -- as measured by my incoming email rate, which is finally near-zero -- I figure it's probably safe to get back in the water.

I'm not really sure what my plans are going forward, other than staying employed at Google until the day comes when I need one of their comfy, brightly-colored caskets. Other than that, my plans are flexible. I'm feeling downright leisurely at the moment.

I realize now that I was trying way too hard to change the world via blogging, and it made me care maybe just a little too much. This was bad for my mental and emotional health. Caring is fine. Lots of things are worth caring about. Very few of them merit sacrificing your health.

Fortunately during my ad-hoc sabbatical I was able to gain some new perspectives by distancing myself a bit from the constant storm going on in the tech world.

One nice perspective I gained is this: There is nothing on this earth that can make everyone happy. Reddit is a huge, living, breathing demonstration of this, since essentially no reddit post ever goes above maybe 80% approval, and a "good" post seems to hover around 65%-70% liked.

That made me feel better about the haters. Haters abound. They're just a fact of life, part of the human condition. There's no need to waste energy hating haters.

Another perspective I gained was that decorating your mansion with works of art you know nothing about is amazingly rewarding, as long as you can mix it up by leaping across rooftops and assassinating bad guys and hanging with your buddy Leonardo. I swear, if they ever make a movie about my life, the handsome and dashing actor who plays me, when asked on his deathbed which of life's pleasures had given him the greatest happiness, will say something cheesy that makes the audience ooh and aww with appreciation, but it'll be total Hollywood bullshit, because what I really will have said was "gaming".

Yet another perspective I gained is that I now actually agree with everyone who complained that my blog posts were too long. Reddit has ruined my attention span for online material. There seems to be no such thing as too frequent, but there's definitely such thing as too long. So I'll be better about that.

I used to have this pet theory that the length of my blogs is a big part of why they've been noticed at all. I mean, look at this dude. If he'd written only one or two crazy things, he'd be just another nutjob, but by dint of almost superhuman persistence he's managed to get the entire world to laugh at him.

I was sort of aiming for getting people to laugh with me, but I used the same basic recipe as Time Cube Dude. And the formula seemed to be working, modulo the haters.

However, Dave Barry -- my Personal Childhood Hero (66% liked!) -- always wrote his columns in chunks of 800 words, even if it necessitated inserting filler words such as "booger" and "legislative session" into his articles about wine tasting or car engines or bat guano, or whatever it was that caught his fancy that week.

Overall it seems likely that post-length is less important than factors such as quality, consistency, passion, relevance, and legislative booger session.

So I was originally thinking of writing up to a maximum of 800 words today, and I'm at about 500 now, but I've been really successful at my Not Caring Too Much Initiative, so... later! Nice chatting with ya. Cheerio!

HAHAHA DISREGARD THAT...

Just to ensure this post isn't entirely devoid of content I'll share something important that I learned last year.

Here's what I learned: after Carpal Tunnel Syndrome, the second most common hand ailment is known as Trigger Finger.

Its more formal name is digital tenovaginitis stenosans, which is ancient Latin for "electronic hand inflamed vagina without writing", which I believe is why most people prefer to call it Trigger Finger.

I have it, you know. Trigger Finger, I mean, not an inflamed vagina.

Although Trigger Finger is "idiopathic", a fancy word meaning that doctors don't have a fucking clue what causes it, it is widely known in musical circles as a musician's injury. It happens to musicians who overpractice, usually in preparation for a recital, performance or recording session.

I found all this out after being diagnosed with it.

It is not idiopathic in my case. I have the benefit of hindsight, and I know exactly what caused it. It turns out that if you play a certain right-hand arpeggio on a classical guitar enough times -- where "certain arpeggio" here refers to Hector Villa-Lobos' Etude No. 1, and "enough times" is approximately 650,000 times in a 5-month period^[wtf?] -- you acquire Trigger Finger. That's not precisely what I was playing, but it'll serve.

Trigger Finger is a painful, debilitating, demoralizing injury. I highly recommend not letting it happen to you. Your body will begin telling you when it's time to ease up on the practice sessions. Listen to your body when it says that.

As for specifics, there's not much to tell. My hand started hurting. Then it hurt real bad for a month. Ibuprofen and cold/hot packs didn't help. It got steadily worse. Even quitting guitar altogether for another month didn't help. I could no longer use my right hand, and it was beginning to feel permanent. I wasn't even sure why it was happening. I was terrified and I began to despair.

My Google doctor was great. She referred me to a specialist -- a hand surgeon. I told her I didn't really want to see a hand... S-word. I could barely say it aloud. She reassured me that seeing a specialist didn't necessarily mean surgery. They might have other tricks up their sleeves. So I decided to brave it.

My first trip to the specialist only took about 15 minutes. She listened to my disoriented bleating, asked me a few questions, gently felt my hand here and there, and informed me that I had Trigger Finger. She said she was going to give me a cortisone shot. She was pulling out a giant needle as she told me this. It just sort of materialized from under the table, the way a knife appears in a bar fight. It was a very large needle. She explained calmly that the cortisone is a steroid that stays wherever you inject it. They use it on athletes to reduce inflammation from certain injuries.

Then she stuck the giant needle all the way into the base of my right middle finger and squeezed. Compared to the pain of my trigger finger, the injection felt like a mosquito bite.

She told me that I'd start feeling better in a week, and in a month I'd be pretty much all cleared up. If not, I should come back and see her for more treatment. And no, I wasn't going to lose my hand.

It was kind of weird, but on my way back to my car I think someone had been cutting onions in the elevator. A lot of onions. In the last fifteen minutes my whole life had been handed back to me with an almost casual lack of concern. I was overwhelmed with onions.

A month later I was back to see her. The cortisone had helped a lot. I gave it a 66% approval rating. She said she could give me another shot, or do surgery. This time I'd done my homework. I elected for surgery. That was back in September. It was an interesting story in its own right, but the upshot is she did great. And then after that there was a lot of physical therapy.

I typed the word "September" in the previous paragraph three times before I got it right. The word doesn't even have letters that need my right middle finger. My right hand, which had shaped itself into an unusable, agonized claw between March and July, is still afraid to flex and extend my middle finger. It's up to, oh, a 95% approval rating now, which I believe is phenomenally successful. Who could ask more of a hand surgery? It could have been much worse. Much much.

But that last 5% is rough. The haters in my hand are a constant reminder of the old pain. When I type, or play piano or guitar, my right-hand fingers twist and curl in elaborate, incomprehensible dances to avoid a pain that is for the most part no longer there.

Yep, I think it'll be easier to keep my blog posts shorter going forward.

Ironically, some good came out of the experience. I've switched to a sustainable new guitar style and a new repertoire, one I enjoy greatly. And I now pay much more attention to economy of motion in my typing. And I spend more time finding pain-saving Emacs shortcuts. It makes me wonder what I might have achieved had I focused on it sooner.

Surgery notwithstanding, on the whole I still think it was a great year. The year was in fact more complicated and more painful than I've let on here, but that's life for ya.

And now that I'm rested up, I believe I'm ready to start tech blogging again... in moderation, anyway. The rest and relaxation and research did wonders for me. I used to have lot of open, long-standing concerns about the future of programming and productivity, but my sabbatical last year finally brought me some clojure.

[1] Talk about caring too much. I may explain this 650k figure in a future blog post if I can ever get over my embarrassment.

A programmer's view of the Universe, part 3: The Death of Richard Dawkins

2009-05-18T02:47:00.001-07:00

We're getting close to the end of my blog. After today's entry, I only have three left to write. After that, I'll only blog anonymously or (more likely) not at all.

This is part three of five in my "Programmer's View of the Universe" series. I struggled for a while with how best to introduce the ideas in this installment, and ultimately opted for a short story.

This is a science fiction short story. It's different from many other sci-fi stories in that it is set in the "near future", but it has realistic schedule estimates. So unlike 1984, 2001, The Singularity is Near and all the other sci-fi stories that grossly underestimated their project durations, this one is set 1000 years in the future. I.e., right around the corner.

The story is disrespectful to pretty much everyone in the world. It will create a fantastic shit storm. This is probably a good time to point out that I don't speak for my employer. [Edit: Yay for fiction! Apparently marking something as fiction placates people. Nice to know.]

The story is 18 pages (PDF from Google Docs print preview). That's not unusual for my blog, but I went ahead and published it as a standalone document.

I'd encourage you to enjoy it, but I'm old and embittered enough to know better. You probably shouldn't even read it. Just wait for someone to summarize it for you.

Installments 4 and 5 will not be short stories; they will be regular old blog rants. In them I will further develop these ideas, and I will also attempt to clear up any gross misconceptions about the story, of which there are bound to be many.

My final blog-rant entry is the only one I care about anymore. I've been working on it so hard that my fingers have started to fail. It's been tons of fun, aside from the chronic pain. It's about a neat programming language, and Emacs, and lots of other stuff. I can't wait!

Oh yeah. Here's the link to the story. I've never done a read-only Google Docs link before, so it's probably broken. Or editable. I don't know. We'll see.

This week is going to suck. People are going to be mad. Maybe I should take a vacation and come back when the whining is finished. Can someone email me and let me know when it's all blown over?

Have you ever legalized marijuana?

2009-04-09T02:07:00.000-07:00

Over the holidays I read a neat book called Predictably Irrational: The Hidden Forces That Shape Our Decisions, by Dan Ariely. The book is a fascinating glimpse into several bizarre and unfortunate bugs in our mental software. These bugs cause us to behave in weird but highly predictable ways in a bunch of everyday situations.

For instance, one chapter explains why bringing an uglier version of yourself to a party is guaranteed to get you more attention than other people who are arguably better-looking than you are. I personally do this all the time, except that I'm usually the ugly one. The same principle explains a ploy used by real-estate agents to get you to buy ugly houses.

Another chapter explains the bug that causes you to be a packrat, and shows why you desperately hold on to things you own, even if you know deep down that they would rate lower than pocket lint on eBay.

In any case, well, good book. I'm going to harsh on it a teeny bit here, but it's only one tiny part towards the end, one that actually has little to do with the rest of the research presented in the book. I still highly recommend it. It's only about a 4- or 5-hour read: beyond the reach of most social-network commenters, perhaps, but you can probably handle it just fine.

So: about that harshing. Dan Ariely, who seems like a pretty fascinating guy in his own right, independent of his nifty book, says something that's kinda naïve towards the end. It doesn't seem naïve at all when you first read it. But naïve it is.

Towards the end of the book — and I apologize here, since my copy is on loan to a friend at the moment, and you can't search inside the book on Amazon.com no-thanks to the book's publisher, so I can't double check the exact details — but towards the end, Dan works himself into a minor frenzy over what seems like a neat idea about credit cards.

Credit Card Buckets

Dan's idea is simple and appealing: let's partition credit limits into "buckets". People are always maxing out their credit cards, and it leads to all sorts of financial misery, since the rates are always by definition just epsilon short of legal usury, so most people can never, ever pay down the debt.

Dan's idea is more or less as follows: you divide up your credit card available balance into "buckets", where each bucket represents a type of expense. You might, for instance, have a bucket for rent and utilities, a bucket for alimony, a bucket for chocolate and desserts, a bucket for sports and leisure activities, a bucket for dining out, a bucket for home improvement, a bucket for groceries, and a bucket for discretionary spending, or "misc".

Each bucket would have its own credit limit, and the sum of all the individual limits would be your credit limit. Let's assume you pay off your credit card entirely every month — not typical, but it simplifies the explanation. Each month, you'd have a certain amount of money to spend on each bucket, and you would not be allowed to spend more than the limit for that purchase type.

OK, so that's the way I understood the proposal. There were several pages devoted to it, as I recall, so I may have missed a few nuances here and there, but I'm hoping I've captured the essence of it.

As an aside, I read a short diatribe many years ago by a working mom whose kids were always asking her why they couldn't spend more money on entertainment purchases like video games. She was having trouble getting through to them, so one month she took her paycheck, went to a bank, and got it issued to her in 1-dollar bills. She took the bills home and piled them up on the table in front of her kids, who were amazed at the giant pile of money she had made. She then went through her budget with them, stacking the bills into piles by expense type: this many dollars for rent, this many for utilities, this many for groceries, this many for soccer uniforms, etc. At the end there were only a couple of dollars left, and the kids soberly realized that they needed to wait about 20 years and then start downloading games illegally online.

Anyway, Dan's idea was kind of similar. In order to train consumers not to overspend in a given category, leading to overall overspending, they would be able to opt-in to a program that partitioned their credit, and presumably it would lead to much wiser, more deliberate spending.

It seemed logical enough to me! It sounds similar to calorie-counting: I've found that explicitly keeping track of my calorie consumption each day does wonders for lowering my overall calorie intake. Along the same lines, I am 100% sure that if I had an explicit budget, rather than just a vague gut feel for how much I'm spending, then I would spend less money each month. We don't really have a proverb for this concept, but we do for the opposite: "out of sight, out of mind". If your budget is in plain sight, well then...

Plus the idea of having types in my credit-card accounting was all the more alluring since I'm a programmer, and I "get" the idea of types. Types are great. Dan is effectively suggesting a strongly-typed approach to credit-card spending, so the programmer in me was all for it.

Evil Banks (as if there's any other kind)

Unfortunately, this story has a sad, bitter ending. Normally I would want to add: "like all overly-strong-typing scenarios", but that would just be mean. So I'm not saying it!

Dan goes on to explain that banks are far too evil, or at any rate far too self-serving to implement his incredibly cool idea. He actually took the idea to at least one bank, meeting with their board of directors and presenting the idea. How cool is that?

Dan says the bank executives seemed to like his idea, and indicated that it might be a great way to get new customers. Credit cards are all pretty much the same (i.e., loan sharks), so they need to find ways to differentiate themselves. A nifty credit-bucketing program seemed like something marketing could run with. They all said they'd look into it and see about maybe implementing it.

And then... nothing. They never implemented the idea! Not even a little prototype of it. Nothing.

Dan theorized that profit margins, as always, are the culprit here. Even if banks could potentially sign up more customers on the promise of better spending control, there's a fundamental problem here, which is that credit cards make money for the banks based on spending. If consumers aren't spending as much, the banks won't make as much profit!

Banks make money off credit cards in at least three ways: they charge the merchant a fee at point of sale, they charge you interest on the loan, and they charge you fees such as the overdraft fee when you inadvertently overspend your limit. All those ways require you to make purchases, and the last way actually requires you to overspend your limit — exactly what Dan's idea is trying to prevent!

I suppose a truly evil bank might look at buckets as an opportunity to screw you on fees for each individual bucket. But Dan seems to think that on the whole, the fear of decreased margins — induced by the suddenly more rational consumer spending — is what is preventing banks from implementing his idea.

At the time I was reading the book, I thought, well gosh, I hate banks. In fact, I don't even use a bank — I now use an investment brokerage that has banking services on the side. You don't have to be rich to do this, and it saves you from ever having to walk into a bank again. And if you choose this route, then whenever you walk into a bank you will immediately be struck by what amazing ghettos they are: little brick buildings with little vaults holding your little dollars, little lines to talk to little tellers who provide you with little help... they're awful. They stink. I detest banks; I've found the whole notion loathsome for at least ten years before hating them became globally fashionable a few months ago.

So yeah. Dan had me. The banks are evil. That's why they aren't implementing his idea. Case closed.

The little winged nagging programmer angel on my shoulder

So just like Memento Guy in L.A. Confidential, my mind wouldn't let the case close forever. (Why can I remember Rollo Tomasi but not the actor's name? Oh wait, Guy Pearce. Him.)

For the next couple of weeks, Dan's situation replayed in my head like a bad song. I myself have given presentations to boards of executives in the past, usually presentations that had come to naught, and I felt a certain empathy with him.

As it happens, I also served time in Amazon.com's Customer Service Tools group for four years, leading the group for the latter half of my stay, and I know a thing or two about credit-card processing. Not a whole lot, but definitely a thing or two.

And I'm a programmer. Just like you. (You might not know it yet, but you are. Trust me.)

The programmer part of me started wondering: how would I implement Dan's idea? What would it take to add "bucketization" to credit cards?

And the programmer part of me started to get a sinking feeling in the pit of his... uh, its... stomach. It got the chills. And a fever. At the same time. Why? Because it, by which I mean "a part of me that wishes I could forget it", has been on software projects like that before.

The little nagging voice in my head started enumerating all the things you would need to do, like counting so many sheep. First I imagined I worked at the bank, some poor schmuck of a programmer wearing a suit, working bankers hours and golfing every day at 3pm. So, you know, pros and cons. Then I imagined my boss coming in and saying: "Steve, we gotta implement Buckets. The board just approved it. Make it happen. Yesterday."

Aw, crap. OK, what to do. First, we need a spec. So, like, I ask my boss a few preliminary questions:

Can customers control the buckets, or are they fixed?
If fixed, how many are there? What are their names?
Let's assume for the remaining questions that they are NOT fixed, since a predefined set of buckets would be "insanely stupid" and rejected by customers. So, how many buckets can a customer make? Min and max?
Can customers give the buckets names? If not, do they have to use numbers?
What characters can they use in the name? What's the maximum length? If we need to truncate the name in a printed statement, how do we truncate it?
Can a customer change their buckets mid-month?
Can a customer change their buckets between months? What if their balance is nonzero? Can they transfer balance between buckets?
Can a customer change the name of a bucket? Do names have to be unique?
Exactly how does a customer name a bucket? Online? Over the phone? By snail mail forms? Talking to bank teller? All of the above?
Same question for all other configuration settings. How? Where?
Do credit-card customer service reps have to know about the buckets? How much do they have to know? (hint: everything) Is there training involved? (hint: yes)
Do the customer-service tools have to be redesigned to take into account this bucketization?
What about the bank's customer self-service website?
What about the phone interactive voice-response tree?
What about the software that sends email updates to the customer?
What about the software that generates printed billing statements? How exactly does it represent the buckets, the individual spending limits and balances, the carry-overs from month to month, the transfers, the charge-backs, the individual per-bucket fees?
What about the help text on the website? What about the terms and conditions? What about the little marketing pamphlets? Should they try to explain all this shit, or just do some hand-waving?
Can a customer insert a new bucket into the list? How are the credit limits of the remaining buckets re-allocated? What if adding a new bucket puts one or more of the older buckets over the limit? Do we charge fees? Do we tell the customer they're about to be charged a fee right before they create the bucket? Is it, like, OK/Cancel? Do we send them a follow-up email telling them they just fucked themselves over? What exact wording do we use?
Can a customer delete a bucket? What if there's money in it? What if it's overdrawn? How do we represent the overdraft fee in the database? How do we show the deletion event in their bill?
Can a customer merge or consolidate buckets?
What if a customer has an emergency situation, plenty of limit in other buckets, and they really really need to charge to a couple of buckets, but they want to avoid an overdraft fee? What do they do? Are the buckets mandatory or discretionary?
How the hell do we even tell if they're buying "chocolate", anyway? The vendor doesn't tell us the purchase type. How do we know how to charge the right bucket? What if it's ambiguous? What if the buckets overlap? Does the customer need a point-of-sale interface for deciding which bucket to put the charge in? Can they do "separate checks" and split the charge into several buckets?
Where are you going? Answer me!
WHAT THE EVER-LOVING *FUCK* ARE YOU PEOPLE SMOKING? HUH? HAVE YOU EVEN THOUGHT ABOUT THIS PROJECT FOR MORE THAN A MILLISECOND? THE SPEC FOR THIS PROJECT WILL BE 5,000 PAGES! IT WILL TAKE THOUSANDS OF MAN-YEARS TO IMPLEMENT, AND *NOBODY* WILL UNDERSTAND HOW IT WORKS OR HOW TO USE IT, EVEN IF WE SOMEHOW MANAGE TO LAUNCH IT! IT'S FRIGGING IMPOSSIBLE! IT'S INSANE! __YOU__ ARE INSANE! I QUIT! NO, WAIT, YOU'RE FIRED! ALL OF YOU! AAAAAAAAAUUUGH!

The little nagging white-robed behaloed programmer whispering in my ear was getting pretty goddamned irritating at this point. And it asked a LOT more questions than the ones in the list above, which is merely a representative sample. My stress level began approaching what I might call "Amazon levels", and I don't even work there anymore. Thank God.

But for all its downsides (e.g. as voiced by my brother Mike, who was in the Navy on an aircraft carrier in the Persian Gulf during the 1990s Gulf War, and later worked at Amazon, and declared after four years that Amazon was _way_ more stressful), Amazon did teach me a valuable lesson, namely that YOUR IDEA IS INSANE. It doesn't even matter what it is. It's frigging insane. You are a nut case. Because anything you try to do at Amazon these days involves touching a thousand systems, all of which are processing gazillions of transactions a second, and you want to completely redo the database schema, and you don't even know the answers to these fucking questions, DO YOU?

I suppose I should think of it as a valuable experience. If nothing else, I understand Complexity in a way most people will, mercifully, never have to.

Anyway, I hope I've imparted the basic flavor of my thinking after having been totally bought into Dan's idea. Here's how I (now) envision the days just after Dan's meeting with the bank executives:

Day 1: (executives): Managers, we'd like you to look into this incredible new idea from Dan Ariely. We think it could revolutionize consumer credit-card spending in a way that makes everyone love us and sign up for our services, dramatically increasing both our profit margin and our overall customer satisfaction. And it's an incredibly simple idea!

Day 2: (managers): Programmers, project managers and marketers, we'd like you to flesh out this idea from On High, and give us some time estimates. We all know we only have a budget of about 2 months for ideas from the Board this year, so let's try to make it fit. How long will it take?

Day 3: (grunts): A billion years. We quit. Fuck you.

Day 4: (managers): Executives, we think it will take about 3 years. It's surprisingly hard. We wouldn't be able to do anything else at our current staffing levels. Should we move forward?

Day 5: (executives): Two years? Good lord! For a project that _might_ increase our profit margins and customer satisfaction, but could also cause customers to be so confused that we have to triple our Customer Service headcount? We don't think that sounds... well, reasonable. Although it would be a very interesting experiment, it's simply too expensive for us to attempt. Should we tell Dan? Well... it might be patentable, and we might be able to get around to it someday if there's a sudden glut of programmer expertise, so... maybe we'd better just sit on it for now. Who's up for golf?

In reality I'm sure it went down a little different from that. For instance, they may have had the Day 5 meeting late, and then gone to a strip club instead of a driving range.

But I'm pretty sure that aside from the mundane details, that's exactly how it went down. Because that kind of shit happened at Amazon pretty much every week I was there, for almost seven years. (And astonishingly, we actually managed to launch at least half those crazy ideas, by burning through people like little tea lights. But that's another story. Plus, no bank can execute like Amazon can. Banks just don't have the culture for it. Bless 'em.)

Legalization

So. I'm two glasses of wine into this whine. I'm going to go get a third glass, mark the calories off in my spreadsheet, and then wrap up. If you don't know what's coming by now, then you're pretty stupid, but on the plus side you're an amazingly fast reader, so I'll go through the motions anyway.

<gets third glass>

It doesn't actually matter what your stance is on the legalization of marijuana, for purposes of this little essay. You could be radically opposed to it on religious, moral, or "parental" grounds. Or you could be so radically in favor that you've been laughing hysterically and rubbing your hands together incessantly ever since you started reading this post. If you know what I mean. Or you could be somewhere in between, moderate and yet open-minded.

It doesn't matter.

This blog is about complexity, the bugbear that haunts software developers, program managers, project managers, and all other individuals associated with trying to launch new software projects and services.

Dan Ariely would have made a great VP (that is, Vice President). If you think that legalizing marijuana is a black-and-white, let's just decide it and get the frigging thing legalized once-and-for-all issue, then you too have some VP blood in you.

VPs have what my brother Mike refers to as "Shit's Easy Syndrome".

You know. As in, shit's easy. If it's easy to imagine, then it's easy to implement. Programming is just turning imagination into reality. You can churn through shit as fast as the conscious mind can envision it. Any programmer who can't keep up is an underperformer who needs to be "topgraded" to make room for incredible new college hires who can make it happen, no matter what "it" happens to be, even if they have to work 27 hours a day, which of course they can because by virtue of being new college hires, they have no social lives and no spouses or significant others, and they probably smoke a lot of crack from being in the dorms so they can stay awake for weeks at a time.

That's the kind of programmer we need at our venerable institution. And we are completely anti-slavery, for the record.

Shit's Easy syndrome is, well, pretty easy to acquire. Heck, you don't even have to be a VP. Directors sometimes get it if they stay away from the code for too long.

As for the rest of us, well, we ought to know better. YOU, as a frequenter of reddit and a programmer (wannabe or actual, it doesn't matter), you ought to know better.

Let's ask our little naggy angel: what would it take to legalize marijuana?

I don't know the answer, and I'm certainly no expert, but I've been on enough projects like this to know how to start asking the right questions.

What exactly do you mean... "legalization"?

So... one minor, teeny-weeny almost insigificant caveat before I continue: I have smoked marijuana (and inhaled it, deeply) on more occasions than I can count. And yet I'm almost undoubtedly smarter than your kid that you're so goddamned worried about. I skipped three grades (3rd, 7th and 8th), entered high school at age 11 and graduated at age 14, took A.P. courses, had stellar SAT scores, was a U.S. Navy nuclear reactor operator, went to the University of Washington and earned a Computer Science degree, worked at major corporations like Amazon.com and Google for many years as a senior staff engineer and/or senior development manager, and now I'm an internationally famous blogger.

I don't usually dwell on that, but today it's relevant. It's relevant because I've smoked a LOT of pot, and I dare you to prove that it has impaired me in any scientifically detectable way. We would debate, and you would lose; nevertheless I double-dog dare you.

So, well, sure... from that perspective, yeah, I'm in favor of legalization. The laws are stupid. Legalize it, already. For cryin' out loud. Jeez.

However, from a programmer's perspective (and keep in mind that I was also an Engineering Project Lead at Geoworks for 3 years, a Technical Program Manager at Amazon for a year, a Senior Development Manager at Amazon for about 5 years, and now I'm a plain-vanilla programmer with 3.5 years at Google, so I've done it all), the idea gives me the chills. And a fever.

Because laws are pretty much like programs. You have to specify their behavior at almost (not quite, but almost) the same level of detail, using a language that's almost as crappy as our programming languages today — English. Or whatever your native language is: it sucks too. If you don't believe me, ask a lawyer. Or try to write a technical spec in your native tongue that the programmers don't ultimately poke full of holes.

Aw, don't try. Don't even bother. Just trust me on this one. Today's natural languages are completely unsuitable for specificity, and "legalese", as much as we all love to ridicule it, is our collective best effort to permit being logical, specific, and unambiguous.

I have more respect for The Average Reddit Commenter than I have for, well, the average commenter in any other forum, period, assuming that "period" is stevey-legalese for "except for LTU, news.ycombinator and their ilk, mumble mumble."

But the Average Reddit Commenter has gone too far. Everyone these days, when debating the merits and demerits of marijuana legalization, seems to have completely overlooked the fact that it's HARD. It's a project of vast, nearly unimaginable complexity.

Think about it. What kinds of laws do we have about alcohol and tobacco? Is it just one law each, saying "it's legal" or "it's illegal?" Of course not, and you're insulted that I asked such an obviously rhetorical question, yet intrigued by my line of reasoning. Admit it! How is marijuana similar to alcohol? How is it different? How is it similar and different to tobacco?

Let's let the little angel ask a few preliminary questions, just to see where it takes us, shall we?

Is it legal to drink alcohol in a TV commercial? No? OK, what about marijuana, then? Can you smoke it in a commercial? Can you SHOW it? Can you talk about it? Can you show marijuana smoke at a party, without anyone actually being seen smoking it? Can you recommend its use to children under the age of 9? What exactly are the laws going to be around advertising and marijuana?
Do we let everyone out of prison who was incarcerated for possession and/or sale of marijuana? If not, then what do we tell them when they start rioting? If so, what do we do with them? Do we subsidize halfway houses? Do we give them their pot back? How much pot, exactly, do they need to have possessed in order to effect their judicial reversal and subsequent amnesty? A bud? An ounce? A cargo ship full?
Is it legal to sell, or just possess? If the latter, then how do we integrate the illegality of selling it into the advertising campaigns that tell us it's legal to own it?
If it's legal to sell it, WHO can sell it? Who can they sell it to? Where can they sell it? Where can they purchase it? Are we simply going to relax all the border laws, all the policies, all the local, state and federal laws and statutes that govern how we prioritize policing it? All at once? Is there a grandfather clause? On what _exact_ date, GMT, does it become legal, and what happens to pending litigation at that time?
Are we going to license it? Like state alcohol liquor licenses, of which there are a fixed number? What department does the licensing? How do you regulate it? Who inspects the premises looking for license violations, and how often? What, exactly, are they looking for?
Is it OK to smoke marijuana at home? At work? In a restaurant? In a designated Pot Bar? On the street? Can you pull out a seventeen-foot-long water bong and take a big hit in the middle of a shopping mall, and ask everyone near you to take a hit with you, since it's totally awesome skunkweed that you, like, can't get in the local vending machine? If it's not OK, then why not?
Can you drive when you're stoned? What's the legal blood-THC level? Is it state-regulated or federal-regulated? For that matter, what is the jurisdiction for ALL marijuana-related laws? Can states override federal rulings? Provinces? Counties? Cities? Homeowners associations?
What exactly is the Coast Guard supposed to do now? Can illegal drug smugglers just land and start selling on the docks? Are consumers supposed to buy their marijuana on the street? What happens to the existing supply-chain operations? How are they taxed? Who oversees it?
Can you smoke marijuana on airplanes? Can airplanes offer it to their customers in-flight? Is it regulated in-flight more like tobacco (don't get the smoke in other peoples' faces) or alcohol (imbibe as you will, as long as you don't "appear intoxicated"?) What about marijuana brownies? Are you allowed to eat it in areas where you're not allowed to smoke it?
Can an airplane captain smoke pot? A ship captain? A train conductor? The driver of a car? An attendee at a Broadway musical? A politician in a legislative session? What is the comprehensive list of occupations, positions and scenarios in which smoking pot is legal? What about eating pot? What about holding it? What about holding a pot plant? What about the seeds?
Speaking of the seeds, are there different laws governing distribution, sale and possession of seeds vs. plants vs. buds vs. joints? If so, why? If not, why not?
What laws govern the transportation of marijuana in any form into or out of countries where it is still illegal? What policies are states able to enact? Is it OK under any circumstances for a person to go to jail over the possession or use of marijuana? If so, what are those circumstances?
Are there any laws governing the use of marijuana by atheletes? U.S. military personnel? Government employees? Government contractors? U.S. ambassadors, in title or in spirit? What are our extradition laws? What do we do about citizens who are subject to the death penalty in countries like Singapore for the possession of sufficient quantities of what we now consider to be legal substances?
What about derivatives? Are the laws the same for hashish? How do we tell the difference? What if someone engineers a super-powerful plant? How do the new laws extend to a potential spectrum of new drugs similar to THC?
For driving and operating machinery, do we have legal definitions that are equivalent of blood-alcohol percentage, and if so, what are these definitions? How do we establish them? How do we figure out what is actually dangerous? How do we test for these levels? When they are established, do we we put up signs on all roadways? Do we update the Driver's Education materials? How do we communicate this change to the public?
How does legalization impact our public health education programs? Do they have to immediately retract all campaigning, advertising and distributed literature that mentions marijuana? How does legalization interact with the "Say no to drugs" programs? Do we need extra education to differentiate between a drug that is now legal (but wasn't before) and drugs that are still illegal? What's our story here? What about other drugs that are even less addictive and/or less intrusive than marijuana?
Monsanto is eventually going to sue the living shit out of someone for using genetically-engineered pot seeds. Can they sue individuals with a single plant in their windowsill? (answer: yes) Will Oprah step in and help that beleaguered individual? (answer: we'll see!)

I'm not an expert, and in fact I've gone to extra-special effort to avoid all possibility of being accused of having researched this subject. I know NOTHING about it.

But the questions, they're bugging me. How the hell do we implement all this? Sure, it's "legal" in Amsterdam, or so they say. I've never been there, and I suspect their laws are way too vague for the overly-litigious United States of America.

I hope it's obvious that we can't say "it's just like tobacco" (it's not) or "it's just like alcohol" (it's not), or (God help us) "it's just like doing alcohol and tobacco together, so take the intersection of their laws".

Marijuana, whether you like the idea of legalizing it or not, is a project. It requires an implementation, and the implementation is a lot like that of a software project. The US federal government is analogous to "The Company", and the states are analogous to "The Teams" that comprise the company. Some of them have free time; some do not. Some of them agree with the overall goal; some do not. And every single miniscule little detail has to be worked out, and written up, and voted on, and approved, and then specified, and implemented, and enforced.

And there will be bugs. And loopholes. And unexpected interactions. The best-laid plans will go awry.

People will die. It's a certainty. Some people are going to die as a direct consequence of legalization of marijuana. I don't like it, and you don't like it, and most of us would probably argue that it shouldn't hold up legislation or legalization indefinitely... but we have to take it into account. Because if it's YOU who dies, smashed to death on the iceberg by your skunkweed-stoned ship captain, you're going to be REALLY pissed off. I guarantee it.

Shit is NOT easy. Remember that. Shit is NOT easy. If you think it's easy, then you are being naïve. You are being a future VP. Don't be that way.

Try to think about how you would implement it. Yourself. If your boss came to you and said: "Make it happen. Yesterday." Have you ever legalized marijuana?

I haven't. But I wouldn't want to be the people who decide how to legalize it. Their asses are majorly on the line. Even more than ours were at Amazon.

My advice: give it some time. Hell, give _Obama_ some time. Whether you still like him or not, he's not a frigging King, he's a President. He can't make stuff happen overnight by waving his magical sceptre. He just can't. I don't know what you were thinking, but "overnight" is a pipe dream, and "a few months" is definitely "overnight", in presidential terms.

Moral of the story: Shit is not easy. Stuff takes time. Months. Years. Decades. It's OK! You'll still be here. Count your calories. Exercise. And you'll still be here to see it all happen.

Patience. It's a wonderful thing. I can't wait!

Story Time

2009-03-12T23:06:00.000-07:00

So I've got all these fancy blog posts planned. More than planned, actually — they're well underway. But it's also been a busy couple of months, so nothing's really ready yet.

To make my schedule even worse, I kind of sort of got myself a little bit addicted to the writings of this one blogger. Normally I can't frigging stand blogs, including my own. Everyone always asks me what blogs I read, and I reply: "Um... do books count?" Which of course is greeted with blank stares, since nobody seems to read books anymore these days. Such a shame.

Anyway, recently I tried another fling with reddit. I'm always getting addicted to reddit for a little while, and then coming off it cold-turkey for a month or two while I reassess whether I want to be spending my life reading pun threads and upmodded lolcat pictures and conspiracy theories.

Not to mention the comments, which can individually sometimes be quite cool, but in aggregate only make sense when you read them aloud with the voice of the Comic Book Guy from the Simpsons: "I upvoted you for the appropriate uses of 'its' and 'their' in the link title, but downvoted you because your link actually appeared on a little-known German social networking site several hours ago. I feel it is important that you understand that this is not a zero-vote of abstention, but rather a single upvote and a single downvote cancelling one another out." If you read the comments with CBG's voice, a lot of them make a whole lot more sense.

No matter how many times I quit, in the end I always wind up sneaking a peek at the reddit home page in a moment of weakness, and before long I'm hooked again, following the adventures of memes about narwhals and how is babby formed and all this other stuff I have zero context for, but occasionally it's hysterically funny. I'm guessing this is more or less what heroin is like. Some days I don't even shower.

In any case, my latest reddit flirtation ultimately led to a secondary and more severe addiction to the writings of some dude who goes by the name "davesecretary". Yeah, him. He's a fucking genius. I started reading his old re-posted stories, got hooked and basically blacked out and lost about three days where nobody knew where the hell I was. Eventually I made my way to his stories about his trip to China. At that point I became actively envious of someone for I think the second or maybe third time in my adult life. I'm just not the jealous type normally, but damn that guy can write. He has a real gift.

So now I'm flat-out jealous. I'm not going to make a secret of it, either, and pretend he's started some trend and I'm just a bigger light jumping on the bandwagon or some lame shit like that. No, he writes way better than I do, and I'm just going to copy him blatantly, like Terry Brooks rewriting Lord of the Rings line-by-line badly as Shitstorm of Shannara.

I had actually already been thinking of publishing some true stories. In fact before I stumbled on davesecretary's tales, I had written down the story below about my brother Dave taking out the garbage, and I had been planning more. But Dave Secretary's stories were like the Great Pyramids next to the mud pueblo of my own ambition. He's like a force of nature; stories just surround the guy like gnats, and he spews out the funniest, most interesting possible versions of them imaginable.

So while I work on my more technically ambitious blogs-in-progress, I thought I'd kick off a series of light reading. It's all true stories. None of it is anywhere near as cool as trying to find a box bigger than Kyle's goddamn cereal box, and nothing in them is anywhere near as funny as the line "Iodine, Dave." But at least these are my own stories, and they're kinda fun to write down once you get going on them. I probably have about a hundred stories like the ones here. I'll start with nine random stories that managed to get themselves written first, in no particular order.

I hope you like at least one or two of them. If not, well, I'll read your complaints out loud as CBG, so we'll be even.

So this one time I'm when in my early 20s, a fragile time during which I'm as fat as a walrus, I'm spending way too much time thinking about the name of the Star Wars character "Dash Rendar". He's some random Star Wars dude, except he's not from one of the actual movies. He's only in the aftermarket books or games or toys or whatever.

I have no idea where I'd heard the name "Dash Rendar", but I can tell you I absolutely loathed it. Lucas names are always a little hard to believe, but this one was just too much. I guess when you're that fat it's easy to get irritated by little things. At that point in my life I was genuinely annoyed that the famous one in the family hadn't been little Pete Rendar or Floyd Rendar or whoever else with a normal name. No. It just had to be frigging Dash.

I'm telling you all this so you can understand my frame of mind for what happened next.

I'm coming up the elevator from the underground garage because I'm too obese to take the stairs, and I'm in a surly mood on account of this stupidly-named character Dash Rendar. I'm trying to understand basic human nature here: how could anyone, even a nine year old, suspend disbelief for such an idiotic name?

I'm all alone in the elevator, so on the spur of the moment I decide to pretend to be a nine year old and see if it makes any difference. So I make this muscle pose and say in my most ominous nine-year-old voice: "I am DASH RENDAR!"

Right then the elevator doors open and fucking Dash Rendar walks in. I am not making this up. If you lined up ten thousand guys and asked a hundred people to point to the one whose name in real life was most probably Dash Rendar, they'd all point to this dude.

He of course witnessed my little announcement, pose and all, and all I could do was stand there slack-jawed, trying to think of a clever and succinct way to say: "I was just sort of trying to understand what kind of idiot would call himself Dash Rendar, and itjustsortofrequiredmetoawfuck." Dash was staring at me with a look like he'd just stepped in dog shit, except when he lifted up his foot he'd found me there instead. I was embarrassed enough to wish I would die spontaneously, but not quite enough for it to actually happen. So instead I just walked out of the elevator and never came back there again. Ever.

Afterwards I definitely established some new rules about what kinds of things I'll try in elevators.

My brother Dave used to work as a waiter at Applebees. This was back about 2 years after he'd graduated high school as a varsity football player, after which he'd settled into a life of comfort and no small quantity of pizza, and he'd blimped up to about 260 pounds. But it happened so fast that he wasn't entirely in touch with how much he actually weighed.

One day before the restaurant opened they were having the usual employee meeting in the back room, where everyone stood in a circle while the manager ran them through the day's specials and whatever else waiters need to know for the day. Dave was sweating and getting a little fidgety, so he found a chair and sat down.

The next thing that happened was a loud BANG, and everyone looked over at Dave, who was sitting on the floor with his mouth open in a big round 'O'. He had completely flattened the chair. It had once been a strong industrial solid metal chair, but now all four of its legs were sticking in different directions like a baby deer crushed under a UPS truck.

Everyone was a little stunned, and the boss felt like he needed to say something to break the awkward silence, so he said delicately: "Gee, Dave, looks like it might be time to start cutting back a bit, eh?"

Dave said later (after losing like 100 pounds) that at the time he didn't know what the FUCK the boss was talking about, and all he could think was: This chair is DEFECTIVE!

So that reminds me of this time I was watching the Seattle Seahawks play at their (at the time) brand-new stadium. We had sweet seats on the 50 yard line, in the second row back, and we were watching a pretty awesome game between the Seahawks and I think it was the Bears or the Giants. I forget which, but it was definitely one of the two. Probably the Bears, so let's go with that.

Anyway, the row of seats in front of me was occupied by some Bears fans. There were about twelve of them, and every single one looked to weigh well over four hundred pounds. It was seriously the fattest row of people I've ever seen. They were squeezed up against each other in this tangle of arms and legs because they couldn't fit in their assigned seats that had been designed for ordinary three-hundred pound fatass football fans.

Every time the Bears had a good play, they would all stand up and roar "YeaaaaAAAAH!!!!", and then they'd all sit down at the same time, on account of their arms being interlocked due to the aforementioned fatitude. They did this little act over and over for the first half of the game: leaping up, bellowing fiercely, then crashing down again as a unit.

Finally I think just after half-time the Bears executed an especially good defensive play, and they all stood up and screamed wildly as usual. But this time when they crashed their asses down, they ripped the bleacher right out of the fucking concrete.

There was this horrible thundering tearing noise like an earthquake, and the whole row of whales spilled forward onto their faces, with their twelve giant asses sticking up at the rest of us. It looked like a bomb had gone off. There were like twenty iron girders that had been ripped right out of the concrete. To say it was the most wonderful thing I'd ever witnessed would be a gross understatement. I think even now, ten years later, it might still be my favorite whale ass sticking up scene in my whole memory.

To make it all even more sublimely beautiful, our beer splashed all over them in slow-motion because the cup-holders on the backs of their seats were thrown forward with the rest of the wreckage.

Anyway, after a few seconds of general shock and hilarity, the ushers ran over to see if the herd were all OK. They had finally managed to extricate themselves and wipe most of the beer off, and they all stood up and started screaming: "This bleacher is DEFECTIVE!"

Once my family all went to this Chinese restaurant in Seattle's Chinatown. It was one of my favorites, and on this particular occasion I was there with my at-the-time girlfriend Stephanie. Stephanie just happened to be mainland Chinese, from Beijing. But unlike everyone else I've known from Beijing, who by and large seem like pretty laid back people, Stephanie was unusually sensitive to race and culture issues, and she kept a keen eye out for any perceivable slights or offenses. (Everyone else said she was just plain nuts, but I'm trying to be charitable here.)

Steph asks my brother Mike what he's ordering. He ponders the menu a while and answers: "Fly lice." Stephanie instantly blows up at him: "What do you mean FLY LICE!!!1!!?? You're insulting our culture? We have 5000 years of Chinese culture and you Caucasians are always insulting us! I can't believe you made fun of it and called it fly lice! That's very rude! That's not how we pronounce it! I can't believe I'm so offended!" and so on in that vein. Everyone else in the family, including me, suddenly becomes tremendously preoccupied with trying to figure out how to do origami with our chopstick wrappers.

Mike listens patiently, since let's just say this sort of outburst isn't wholly unfamiliar territory when Stephanie's around, and after she finally simmers down a little he says gently: "Uh... ok. Sorry."

So the (Chinese) waitress comes around and is taking our orders, and when she gets to Mike, Stephanie starts hyperventilating a little in case she'll have sudden need of some extra screaming oxygen. Mike says in his blandest and most American possible voice: "I'd like to have the fried rice, please."

The waitress nods happily and says "FLY LICE" really loud and writes it down on her pad. Mike, always an excellent poker player, manages to keep his face pretty straight, but I can tell his eyes are sparkling just a little. Meanwhile Stephanie's eyes have grown to the size of large saucers, and she hisses loudly: "She said FLY LICE!!!!!! Hee hee hee hee hee hee HEE HEE HEE HEE HEE!" Mike just shrugs, like, "hey man, I just want to not get yelled at anymore."

We don't call it fly lice now, though, since you just never know.

When I was growing up the holidays were always crazy because I had over 20 aunts and uncles, and seventy or eighty first cousins, and of course we had to do a massive holiday family party at my grandparents' teeny house every year. Every family would bring a few friends, so there would be well in excess of 100 people running around in pure holiday chaos for a day.

We used to do this secret-santa gift exchange where everyone would put their name in a hat a few weeks in advance, and then we'd all draw names, and that was the person you had to get a gift for. It wasn't split into adults and kids or anything like that.

With an extended family that big, it's kinda hard to keep track of everyone, especially the new additions. It was also kinda hard to know what people actually wanted, and some of my uncles weren't too big on preparation. Uncle Harold was pretty much the worst at it. We loved him to pieces, but he just couldn't get the hang of the holiday party or the secret santa exchange.

To give you the basic Uncle Harold flavor, one year he didn't show up because on the way to the party he decided to stop in a bar down the street, and he stayed there drinking bloody marys until midnight. Another year he was tasked with bringing the turkey, and he showed up at dinnertime with a frozen-solid turkey. And one year his secret santa target was Aunt Celie, who was infamously religious and pretty much nothing else, just religious, so Harold, totally at a loss, got her a picture of Jesus and taped fifty bucks to the back of it. All the adults were pissed off and all the kids thought it was pretty much the funniest goddamn thing in history. Either way, Uncle Harold just couldn't win at Christmas.

But this one year he really outdid himself. I think I was about 11 or 12 years old, just old enough that I was no longer running around screaming like the next generation of cousins. So I got a chance to sit back and watch the party unfold. I'm sitting there watching Uncle Harold give his secret santa gift to one of my cousins, one of Aunt Diana's kids. For some reason Harold looks embarrassed.

My dad had 11 brothers and sisters, all married with like 6 or 8 kids each, and there was no way any of us could really keep track. Usually we'd get in touch with an immediate family member and try to get some direction, but Harold, as usual, just winged it. I don't remember which cousin it was, but last time Harold had seen him, he was a baby, so Harold, thinking it through with his usual clarity, got him a jar of Gerber baby food.

Unfortunately it turns out it's been like six years since Harold had last seen him. So my cousin opens up his present and I hear him saying to my aunt Diana: "Mom. Uncle Harold got me baby food." Diana was busy with two other crying kids, and she's not feeling particularly merciful, so she says: "Well go tell him thank you."

So my cousin walks over to the tree where Harold is sitting, and he says: "Thank you for the baby food, Uncle Harold."

This story is more grim than funny, but it's a story. With sequels, no less!

In college I lived in a big four-bedroom apartment with 3 roommates for about a year. It wasn't air-conditioned and during summer it was hotter than hell, and we all kept our windows open most of the time.

I was just starting to drift off to sleep one night, laying on my back, when I heard this loud clicking noise. It sounded like snapping a ball-point pen out of its plastic cap, and the clicks were coming at roughly one per second in irregular bursts.

The clicking noise was coming from between my eyes. Not, like, a foot away, or an inch away, but from directly between my eyes, near the bridge of my nose.

I remember thinking: "That's odd. Whatever could be making that loud clicking noise coming from between my eyes? I've never heard anything like that before."

Part of me was falling asleep, and was thinking "mufbmlflfsleeep go sleep go sleep mblfjust don't worry abouuuuuut it", or something like that.

But there was this other little part of me, a tiny voice, that was busy thinking out loud to its tiny self: "It could be a spider."

My falling-asleep part woke up a little at that point, and pondered it for a few dozen clicks. Then it replied: "No. No, it's not a spider, because that would just be TOO horrible to contemplate. So that's not what it is."

And the tiny voice was like, "Well you come up with a non-spider explanation then. You can't just ignore loud clicking noises coming from BETWEEN YOUR EYES, dude."

Falling-asleep voice: "Spiders don't make noise. So there."

Tiny voice: "Small ones don't. Medium-sized ones don't. But we don't really know what BIG spiders do, now, do we? I bet a really big one could make a noise just like that."

At this point, opinion-wise, I'd say I was coming down solidly on the side of falling-asleep voice, but neither the clicking nor the tiny voice would shut up, so I rolled over onto my side so I could think about it from a different angle.

As I rolled over I felt something huge jump off my face. It landed on my arm and started dragging itself along with what felt like a dozen scrabbly legs, and of course I was like WAAAAAAAAAAAH!!!! and I jumped out of bed faster than any human being has ever done before, and turned on the light.

Falling-asleep voice and tiny voice were both, like, "Oh, shit!" Sitting there next to my pillow was the biggest fucking wolf spider I've ever seen in my life. It had a good four inch legspan, and it was very specifically glaring at me. There was no mistaking its expression. It was really pissed off. At me.

So I do the only thing I know how to do in Large Spider situations, which is to scream for help. "AAAH!!! A FUCKING SPIDER CRAWLED ON MY FACE!" My roommates' doors all started opening and they came out in their bath robes, all keenly interested to see the spider that had crawled on my face.

Jacob is first to arrive. He takes one look at this massive glaring spider and says: "Holy shit dude, that thing's as big as a horse!" and leaves as fast as he came in.

Dave shows up next. "Holy crap, that thing crawled on your face? You must be psychologically damaged!" Dave opts to stand and goggle at the giant spider, from a safe distance.

Then Eric comes in and says: "Oh, it's just a hobo spider. I'll take care of it." He walks into the bathroom and grabs ONE square of toilet paper. Not a whole handful, no, just one teeny slice, like he's going to wipe the spider's butt with it or something, and he goes and grabs the spider with it. We can clearly see the spider's legs waving around outside the toilet paper, and Eric says: "I'll just put it outside your window."

I of course was having none of that crap. "How the hell do you think it got on my face in the first place? Flush it! Now now now! Flush!" Eric, clearly disappointed, flushed the thing, and I'm certain that it now lives in the Seattle public sewer system, feeding on rats and stray dogs and plotting its revenge on me.

My actions that day clearly angered the Spider God, because over the next few years I had several more spider-on-face incidents, at least one of which was way more dramatic than this one. It's a bit traumatic to tell both stories at once, though, so I'll wait a bit on that one.

When my brother Dave was around 14, our family lived in a house in Southern California. It was kinda rainy at the time, which is sort of unusual for where we lived. On this particular California winter day it was Dave's turn to take out the trash.

Our city-issue trash bin was out in the carport, this sort of concrete alley next to the house where our big Dodge van was parked. The trash bin was on the other side of the van, next to a six-foot brick wall separating us from the neighbor.

Dave grabs the trash bag from the kitchen and heads outside. He walks over to the trash bin, opens the lid and sees that it's completely full. There's a plastic garbage bag right at the top, and the lid just barely closes over it. Dang.

It's starting to rain pretty hard, and Dave just wants to get his bag into the can and get back inside, so he figures he'll do the old Human Trash Compactor trick. He puts down his bag, climbs up on the brick wall, aims for the bag on top of the bin, and jumps.

Usually this works pretty well, right? Your plummeting body weight smushes the bag down just far enough to make room for another bag, and you hop out unharmed.

Unfortunately for Dave, it turns out that this time the bin was completely empty except for the bag on top. That one bag was full of trash, but it was so bulky that it hadn't fallen down into the bin.

So Dave jumps off the wall, plunges all the way into the bin, and the lid slams shut on top of him. His momentum makes the bin tip over, and it wedges itself solidly between the van and the wall, trapping Dave inside with the garbage.

Dave's no more claustrophobic than the next person, but the whole thing caught him a little off guard. In his mind's eye he'd been envisioning some sort of heroic plunge that would compress the garbage about a foot or so, followed by a dashing bounce onto his feet on the driveway. Instead, in just under one second he'd materialized sideways in a dark stinky fallen-over garbage can getting pounded by rain, and the lid wouldn't open.

So he did the first thing that came naturally to him, which was to panic and kick and scream and thrash and flail and try to claw his way out. Pretty much what you or I would have done.

However, he was really stuck in there pretty good, and nobody could hear him because of the rain, so it took about ten minutes of violent side-to-side heaving before he finally rolled the can out from between the van and the wall.

So we're all sitting in our nice, comfy living room, watching TV. Dave has gone to take the garbage out. Ten or fifteen minutes later, after we've all totally forgotten about him, the front door bangs open and Dave barges in, wild-eyed and soaking wet and yelling at us at the top of his lungs. He's completely covered in garbage: coffee grounds, smears of leftover food, pieces of dirty paper, part of a banana peel on his shoe, brown slime all over his head and arms and legs and clothes, and he's screaming: "DIDN'T ANY OF YOU HEAR ME? I WAS STUCK IN THE GARBAGE CAN!"

Back when I started working at Amazon.com in 1998, the company was in this little building in downtown Seattle in kind of a bad part of town. I mean it wasn't terrible, but there were definitely some issues. There was a needle exchange across the street, which was cool and all, but a fair number of the drug users did their thing in the alley behind the building we were in. So you didn't really want to walk out the back door, or you'd run the risk of stepping on some dude with a syringe dangling out of his arm.

The nearest place to eat was the Scaryaki joint across the street, next to the needle exchange. It was this teriyaki place that we all called the Scaryaki, since even though the food was pretty good almost nobody ever went there because it was scary. Usually we'd walk a couple blocks past it, which took us to Pike Place Market, which is also scary in its own way but is much less overtly threatening. Plus there's more culinary variety.

One day my friend Jacob and I left the 'zon premises to eat at Matt's in the Market, which was this really awesome hole-in-the-wall joint that was Zagat rated and nobody knew about it and it was delicious.

On the way back to work, we're crossing the street in front of the Scaryaki, and I can't help but notice Jacob's doing an unusual amount of looking at my ass and my legs. We're both straight, and as a consequence we usually try to avoid staring at each other's asses as a general rule, so I'm a little annoyed. But he's definitely checking me out with waaaay too much interest.

I finally give him the Glare, and he says: "Where'd you get those pants from?"

I shrug. "I don't know."

Jacob is now really keenly eying my pants, which are these dark blue cotton slacks I've been wearing to work on and off for at least a few months now. He starts being more demanding. "No, really, where'd you get those pants from?"

I'm like, "Dude, I don't know! Stop looking at me!"

Jacob suddenly announces, in a really loud voice that everyone within a block or so can hear: "You're wearing my pants!" A couple of people in the vicinity, including some druggies and some coworkers, perk up with some interest as the argument unfolds.

Me: "No I'm not! What are you talking about?" <walking faster>

Jacob: <running to catch up, pointing at my ass> "Those are my pants! Where'd you get them from? Those are mine!"

Me: "I don't know where I got them! I just had them! They're mine, OK? Leave me alone!" I'm now running full bore for the doors, since I want to get back to work and out of this sudden surreal nightmare.

And all the while I'm thinking to myself: "Where DID I get these pants? I never buy blue slacks. I found them in my closet one day, so they MUST be my pants. They fit pretty well, even though they're a little long. So they're mine! I must just not remember buying them!"

We burst into the lobby, where everyone in the frigging company is on their way back from lunch, and Jacob is running after me yelling: "I recognize those pants you're wearing! Those are MY PANTS!" There was chaos for a while. It was pretty messed up.

The funny thing is, it turned out they actually were his pants. He and our mutual friend Jeff and I had all gone out to El Gaucho earlier that year. It's the awesomeest steakhouse in Seattle, arguably in the whole country, and we all dressed up after work one day to eat there. I drove, and they threw their work clothes into the trunk of my car, which was almost completely filled with random crap. When they got their clothes out later, Jacob didn't see his pants or something, and I must have thrown them in my laundry when I finally cleaned my car, months later. One day I started wearing the pants without really thinking too deeply about their origins, and the rest was history.

He probably could have picked a better way to tell me, though.

I remember when I was 23 years old, my dad decided to have one of those Dad to Son talks. He'd clearly thought seriously about it, because he sat me down and gave me one of those This Is Important looks. He said: "You know how sometimes they lose your file?"

"Uh... what?"

"You know, like when you call up to make a dentist appointment, and then when you get there they have no record of the conversation? Or you set up an account with the cable company, or whatever, and they can't find it later?"

"Er... yeah, sure. That happens to everyone sometimes."

My dad is like, "Nope. It happens to us way more often. And when it happens, tell them to look under 'W'."

"What? 'W'?"

"Yep. Ask them to look under Wegge instead of Yegge. 9 times out of 10 they'll find it."

"WTF?"

"Yep. It's weird. But when you spell our name, Y-e-g-g-e, a lot of people write or type 'W' instead of 'Y'."

I'm thinking, "You waited until I'm 23 years old to tell me this?" But I'm also thinking: "Damn, are people really that stupid? And if so, how the hell didn't I notice this myself?"

"Uh, thanks Dad. I'll keep an eye out!"

So I watch. And listen. And sure as shit, he's absolutely right. The percentage is pretty high, like maybe 10% to 20% of the time. Someone (in person or on the phone) will ask me to spell my name, and I'll say 'Y', and they'll enter 'W'. A lot of the time I'll actually be in a position to watch them as they do it -- I'll be looking over the rental-car counter or whatever, and when you're looking at a keyboard upside-down from the side, you can see the 'W' as they type it.

I have no idea how many years it took my dad to figure this out, but he's a pretty perceptive dude, and he was 43 when he told me. So we're talking about half a lifetime of watching people fuck up, and eventually realizing there's a pattern to it. Bravo.

Nowadays I'll say "Yellow Echo Golf Golf Echo" if I'm on the phone, since it slows them down just enough to think about what they're typing. In person, when I can see what they're typing, I still say "Y", because I'm always dying to know if they're one of Those People. If they get it wrong, I'm like, "No, 'Y', not 'W'", and they always say: "Gosh, I have no idea why I did that!"

Me neither. But I think it's because when you pronounce the letter Y, it starts with a W-sound, as in "WHY"? Sometimes it even seems as if the slower and more deliberately I say 'Y', the more likely they are to get it wrong, because my lips started off with this big 'W' sound.

Anyway, the best one ever was at a Jiffy Lube. The Lube Supervisor Dude was asking me for my personal information and writing it all down on a form on his clipboard. He apparently felt he was better qualified to write my personal data on the form than I was, and to be fair, he had pretty good handwriting.

He asked me to spell my name, and I said: "Y". He wrote "W". So far, so good. I really didn't want these fuckers to have my personal information just because they gave me an oil change, anyway.

I said "e", and he wrote "i". Wow, this was new.

I said "g, g" and he wrote "jj". Cool!

I started to get kind of excited to see if he'd get every letter wrong. I said very carefully: "e", and he wrote "i", completing my last name as "Wijji". It was Jiffily Lubriciously Awesome. I told him "THAT'S EXACTLY RIGHT!", like he'd just won on Who Wants to be a Millionaire, and he gave me this big shit-eating grin with several missing teeth. It was nice to make him feel happy.

But check it out: I told this story to my friend Gayle, whose last name has a bunch of doubled letters in it, including ending in "nn". She said that she'd found empirically that in order to get people to write "nn" at the end of her name, she had to say as many as four or five N's. She's gotten into the habit of spelling it with "enn enn enn enn enn" at the end so they'll write two of them.

Go figure.

OK, last story for today.

I've been a snowboarder for almost 20 years, but waaaay back in the old days I actually used to ski. I learned in my late teens, and I spent 2 or 3 seasons skiing, mostly using Navy rental equipment. I have all sorts of Navy stories to tell, but they'll have to wait for another day. Suffice it to say the Navy provided the rental equipment for this story.

This was back when I was at the US Navy S5G Nuclear Reactor Prototype training plant in the middle of Idaho. "S" was for "Submarine", 5 was the reactor model number, and "G" was for General Electric, who had made the reactor. Hmmm, maybe I should tell some of my reactor stories someday... they're definitely interesting. Anyway, the reactor was at a Department of Energy site in the desert in the middle of Idaho, with a 60-mile bus ride from the nearest city, Idaho Falls. I'll leave it to your imagination as to why they located the experimental and training reactor facilities out in the middle of a desert, but your first three guesses are probably all more or less correct.

Anyway, the downside to the program was that we were in Idaho, where the fun thing for residents to do is follow potato trucks in their cars, trying to hit dislodged potatoes. That's pretty much the pinnacle of entertainment in that fugging dustbin wasteland. The upside was that there were some really nice ski resorts nearby, so during winter we got to do a lot of skiing.

This one time we were either at Jackson Hole or Grand Targhee. Both of them are in Wyoming, but only about 2 hours from town, so we usually went to one of them. The way the Navy Nuke program works during the prototype training phase is that you work 7 days on, 1 day off, 7 days on, 2 days off, 7 days on, 4 days off. This is actually a pretty doable schedule, and the once-a-month 4-day weekend was always something you planned in advance, since that's a lot of days for screwing around.

A lot of my friends were skiers, and snowboarding was still pretty new back then. There was one dude, I think his name was Lundberg (everyone goes by last names in the Navy), who was learning to snowboard, and he pretty much spent all his time falling on his ass. So I decided to ski. The Navy pretty much takes care of all your needs, including ski rentals, so I went to the local Navy ski rental place and got hooked up with some rental skis, boots and poles.

Unfortunately this one time, which was probably no more than my seventh or eighth trip ever, the rental guy adjusted my bindings too loose. Bindings are the things on your skis that hold your boots in place, and you don't want them to be too tight or you could break your legs in a bad crash. You want them to be tight enough to hold during an aggressive turn, but loose enough to pop out in a crash. And mine were way too loose.

We ski for a few hours, and I've warmed up a little, so I want to work on my turns instead of snowplowing down steep stuff like a sissy. So I follow my friends to a black diamond. When I was skiing I would actually follow my friends pretty often, which was completely idiotic because they've been skiing since they were in the womb, and they look like frigging Olympians, and they're always dragging me off to slopes I have no business being on. But for some reason I still did it.

This slope is a pretty steep black -- nothing for me today on my snowboard, but back then it was way out of my league. But it doesn't hurt that much to fall, so I went ahead and tried a real turn. Of course as luck would have it my binding popped out and I went flying straight down the slope on my face. It was so steep that I actually started picking up speed, and I wasn't experienced enough to know how to stop myself, so I just started sliding faster and faster downhill, face-first on my stomach.

I remember being vaguely aware of people stopping and looking at me and pointing as I accelerated by them, but after a hundred feet or so I had picked up enough speed that I couldn't really do anything except this emit gurgling screaming noise. The snow was bunching up in my face and going in my eyes and up my nose and down my throat, not to mention down my shirt and pants, but all this plowing wasn't really slowing me down, so I wound up sliding about three hundred yards on my face. I didn't have any really good ski gear at the time, though. I was actually wearing baggy jeans, if you can believe it. So when I near the bottom stopped I was caked in snow and ice.

Someone was nice enough to bring me my ski from way up the slope, and I hobbled into the lodge to grab a beer. I was sitting there and everyone was laughing and having a great time, but I was FREEZING. I went and sat by the big fire, but it wasn't helping. I started shivering badly. Finally I stood up and lifted up my shirt and about ten pounds of snow fell out, which sort of explained it.

My brilliant friends told me that I should go back up for another run before we left for the day. They said you stay warm on the slopes, which is more or less correct if you're actually skiing and not being a human snowplow. I really wanted to warm up so I went back up for one more run.

At the top I took a cat-track between slopes -- just a road, basically. The cat-track was running along the side of a steep roped-off drop-off with a bunch of trees. The cat track headed for the drop-off, then made a sharp turn to the left. When I got to the turn, I turned too hard, and my right binding snapped open and my ski shot out from under me, and of course I wiped out.

I'm laying there on my back, thinking "man, I'm just not sure about this whole skiing thing", and an older guy skis up next to me and asks me if I'm OK. I'm still laying on my back, with my right leg bent awkwardly under me, trying to catch my wind. I tell him I lost my ski, and that I think it went over the edge. I ask him if he can see it.

He skis over to the ledge. I lay there, sun shining on my face, snow on my back, freezing my balls off, just kind of trying to enjoy the moment as best I can while he assesses the situation. I look over and he's at the ledge, scanning the mountain. A bird flies by. Some time elapses -- probably no more than 30 seconds, but it seemed like eternity.

I finally ask the guy, "Hey, can you see my ski?" He looks at me, then back down the mountain. Then he takes a deep breath as he decides how best to tell me. Finally he speaks.

"Yep. She's still on the run."

A programmer's view of the Universe, part 2: Mario Kart

2008-12-27T03:32:00.000-08:00

This is the second installment of a little series of discussions. They're not much more than that, just discussions. And I hope I'm inviting discussion rather than quenching it. But I'll be honest — the goal of this series is to pound a stake through the heart of a certain way of thinking about the world that has become quite popular. If my series fails in that regard, I hope it may still provide some entertainment value.

Part one, The fish, was about a twisty line and a fish's examination of it. Today we move to a twisty plane.

Embedded Systems

There are many kinds of computer programs, and many ways to categorize them. One of the broadest and most interesting program categories is embedded systems. These systems are the centerpiece of today's little chat.

Embedded systems are a bit tricky to define because they come in so many shapes and sizes. Loosely speaking, an embedded system is a little world of its own: an ecosystem with its own rules and its own behavior. So an embedded system need not even be a computer program: a fish tank is also a kind of embedded system.

We call them embedded systems because they exist within the context of a host system. The host system provides the machinery that allows the embedded system to exist, and to do whatever it is that the embedded system likes to do.

For fish tanks, the host system is the tank itself, which you may purchase from a pet store. A tank has walls for holding the water in, filters and pumps for keeping the water clean, lights for keeping the fish and plants alive a little longer, and access holes for putting your hand through to clean the tank. There's not much to it, really. The embedded system is everything inside: the water, the plants, the rocks, the fish, and the little treasure chest with bubbles coming out.

For computer games, another popular kind of embedded system, the host system is the computer that runs the game: a PC, a game console, a phone, anything that can make a game exist for a while so that you may play it.

Programmers have been building embedded systems for many decades: a whole lifetime now. It is a well-studied, well-understood, well-engineered subject. Gamers understand many of the core principles intuitively; programmers, even more so. But in order to apply all that knowledge outside the domain of embedded systems, we will need some new names for the core ideas.

The most important name is the One-Way Wall. I do not have a better name for it. It is the most important concept in embedded systems. In lieu of a good name, I will explain it to you, and then the name will stand for the thing you now understand. It's the best I could come up with.

But first let's dive into an embedded system and see what this wall looks like from the inside.

Mario Kart

I will assume you've played Mario Kart, or you've at least watched someone else play it. Mario Kart is the ultimate racing game in terms of sacrificing realism for fun. It bears so little resemblance to reality that it's a wonder it tickles our senses at all. The Karts skid around with impossible coefficients of friction, righting themselves from any wrong position, and generally making a mockery of the last four centuries of advances in physics.

It's really fun.

Mario Kart, like all games, has a boundary around the edge of the playing area. In Mario Kart you bump into it more often than in most other games, which is part of the reason I chose it to be our Eponymous Hero. If you want to win a race, you will need to become quite good at accelerating around corners, which means you will spend a fair amount of time bumping up against an invisible wall.

You know the wall I'm talking about, right? It's invisible, so you can see right past it to the terrain beyond. But the wall is there, and you are not permitted to venture beyond it.

In slower-paced games, when you arrive at the invisible map boundary, you will sometimes be told by the game: "You can't go that way. Turn back!" And since that is your only option, you comply. These invisible boundaries are non-negotiable.

In other games, you may stop on contact with the boundary, or perhaps bounce off. But the boundary is always there.

Imagine Mario and Luigi chatting about the you-can't-go-that-way wall. Their conversation might go something like this:

Mario: "Luigi, my brother!"

Luigi: "Maaaarioooooo!"

Mario: "Yes, Luigi. I am a here. Tell me my brother, why is it that every a time I a spin around the cactus in the third a bend of the Desert a Track, I get a stuck and have to start accelerating from nooooothing?"

Luigi: "Whaaaaa?"

Mario: "Brother, the Desert a Track! It's Number a Three! You know the big a bend, where you have to slow down? I am always a forgetting to slow, and I just a stop. Just a like that!"

Luigi: "I don't know, Brother. That Bowser, he is always a squishing me right before I hit that turn, and I am a flat like a pan a cake for a looooong a time."

Mario: "What about that a time two races ago, where Wario hit you and you a spun around and you a headed for the hill, and you got a stuck and wailed for me?"

Luigi: "Ah, that Wario! I will get him next a time!"

Mario: "Why did you not just turn around then?"

Luigi: "My Kart, it did not a move, no matter how I wailed."

Mario: "That is a what I am a speaking of, Brother. Your Kart moves in all other places, but if you head for the hills, it just a stops a suddenly!"

Luigi: "I a hate a stopping a suddenly."

Mario: "Yes, Brother. So do I. But why can we not traverse that a part of the hill? What is on the other a side?"

Luigi: "I think it is Track 4, Brother. They do not want you to wind up in the lake."

Mario: "Whaaaa? Who is this a 'they'? And why can we a not see the lake from a Track a 3? I think there is a nothing there, Brother. Just a more hills."

Luigi: "No, my brother. I think it is a Track a 4. Or maybe Track a 2. There must be a something there."

Mario: "Perhaps you are a right, my brother. We should get a back to a racing now. We can talk a more about a this after the next a race."

Luigi: "Yes, brother. I will a get that Wario this a time!"

Well-formed nonsensical questions

In our Highly Realistic Dialog, Mario is wondering about the Invisible Boundary at the edge of the track. He asks his brother the seemingly obvious question: "What is on the other side?"

As a gamer, if you pause to consider the question at all, the answer seems to be: "Nothing I care about." The invisible (or sometimes visible) Boundary seems just like any other wall. It is designed to keep you in where you're at, or out of where you're not, depending on your point of view.

But the gamer's view is that the boundary does have another side. You have no way to get there, but it exists. For maximum gameplay immersion a game universe needs to appear consistent. Thus, when you peer through the wall it appears that the other side is just more of the same.

To an embedded systems programmer, Mario's question is complete nonsense, like a fish asking the temperature of the water on the other side of the glass. There is no water on the other side, and for that matter a fish can't ask questions, so the question itself is based on nonsense premises.

From a perspective within the Mario Kart universe, the "other side" of the Invisible Boundary is... a kind of nothingness. The embedded world quite literally ends at the boundary. The level designers usually try to make it appear as if the current world continues on, but this is an illusion. They discontinue the model's polygons beyond the line of sight. Put another way, the water stops at the edge of the tank.

The nothingness beyond the Invisible Boundary of an embedded system is much deeper and more significant than simply not having objects there. In that nothingness there is no programmatic substrate upon which to place objects. If you were to ask: "What lies between Mario's head and Luigi's head," the answer may well be "nothing", since no game objects may overlap the line between their heads at the particular time you ask the question. But that "nothing" is qualitatively different from the "nothing" on the other side of the Invisible Boundary. Between Mario and Luigi there is a space – a place in their universe where objects and forces and logic apply, even if they are Mario Kart's twisted versions of physics and logic. That universe ends abruptly at the surface of the boundary.

The question "What's on the other side" is well-formed in a strictly mathematical sense. You could project a line from Mario to the nearest boundary, and ask the more precise question: "If Mario is at distance D from nearest boundary B, what overlaps the point P obtained by extending the Mario-boundary line L to a distance D beyond B?" ("Whaaaaa?")

The new formulation of the question is more well-formed, but it is every bit as semantically nonsensical.

What's really on the other side

In the context of the Mario Kart universe, the system boundary truly only has on side, and Mario and Luigi are on that side. From their perspective, we can't meaningfully ask the question "What's on the other side?" However, there is a semantically significant interpretation of "the other side" of that invisible boundary. To get to this better answer we must leave Mario's universe.

From the perspective of an embedded systems programmer, the entire Mario Kart universe is data structures laid out in the memory space of some machine. There is a program — the game engine — that interprets the data in a way that makes it appear to be a universe that is similar to ours in various intuitively appealing ways, yet different from our universe in various fun ways.

It is very unlikely that the polygons and other level data are laid out in strictly contiguous order in the host machine's memory space. It is more likely that they are spread out, with gaps, like dominoes spilled on a tile floor.

The question "What's on the other side?", when viewed from the perspective of a systems programmer, might be phrased more precisely and meaningfully as follows: "What occupies the memory spaces adjacent to the memory used by the Track Three Desert Level?"

This is the same as Mario's question, but we had to step outside the Mario Kart universe in order to ask the question in a way that made sense. The Mario Kart universe, like most game universes and in fact most embedded systems, is designed to appear complete. There is apparently "stuff" beyond the boundary; you just can't go that way.

When we step up into the host system and ask the analogous question about the "other side", both the question and the potential answers become much more complex: many orders of magnitude more complex, in fact. Fortunately, due to the Mario Kart system being so simple, increasing the complexity still gives us a vaguely accessible problem.

Let's try to cook up an answer to this new, more complex question regarding the contents of the program memory space on the "other side" of the invisible wall.

In terms of atomic program units (say, bits or bytes), the amount of memory used by a game like Mario Kart is actually high enough to defy our sense of scale. A game with hundreds of megabytes or gigabytes of runtime data has billions of discrete elements, which is too many for us to keep track of individually. One of the key jobs of a programmer is to organize their code so that the elements can be managed at human-CPU scale: up to seven "things" at a time. But this organization can't mask the fact that billions of individual elements exist, each with its own role to play in supporting the embedded system.

Hence, even for a game as "simple" as Mario Kart, we're stuck imagining how it works internally. Even the programmers who built the game have only a dim and incomplete mental model of the game. It's like building a skyscraper, or a city: you build it a piece at a time, and it's rarely fruitful to try to picture everything inside of it simultaneously.

Anything that goes wrong or is out of place in the program could take days to track down with state of the art tools. That's just how it is. We're not able to comprehend large problems all at once; we can only attempt it in small increments.

Bearing this necessarily incomplete understanding in mind, what exactly would we find in the machine memory between the elements of Mario and Luigi's track mini-universe?

The answer is familiar to programmers and perhaps surprising to non-programmers. The answer is: almost anything. It could be elements of a different program, or random garbage not being interpreted by any program, or supplemental data for the Mario Kart universe, such as language translation strings. Or Luigi could be right: it could be Track 4, pre-cached for the next race. Perhaps not exactly the way he's imagining it, but... close.

Moving beyond the Wall

Our little visualization of the host system's memory raises another natural question: what would happen if you "zapped" Luigi's Kart across the boundary?

This question isn't entirely as nonsensical as "what's on the other side?" With the proper programming tools, you might be able to observe changes in the machine memory as Luigi's Kart moves, and these changes might follow an observable pattern that leads to a relevant prediction of sorts.

As just one possibility, Kart motion might be represented as shifting a Kart memory structure along the sequential indexes of an in-memory array. This arrangement is unlikely for performance reasons, but it's certainly possible, and should serve to illustrate the point. If you were to find that moving Luigi's Kart ten meters (in the scale of Luigi's track universe) resulted in a structure motion of 1000 memory addresses, then you might make the reasonable prediction that in another ten meters his representation would move another 1000 addresses.

This might well put him beyond the end of the physical memory array. In most real-world scenarios this would be a bug, and would result in some sort of nasty response from the machine, such as a game crash. Or in a more user-friendly environment, the game engine might simply prevent his motion in that direction and move him back into his universe. This might put his Kart in an uncomfortable position, but it will at least be a real position according to the logic of the Mario Kart universe. At least the Kart won't have disappeared.

However, it is infinitesimally possible that Luigi's Kart could be moved into another physical machine process in the host system — say, another instance of Mario Kart running on a time-sharing operating system or virtual machine — in such a way that Luigi and his cart physically disappear from the old universe (a process address space) and appear in the new universe (another process address space).

Even if this infinitely remote possibility were to occur, chances are high that the sudden appearance of Luigi and his Kart would wreak local or even global havoc in the new universe. He might get lucky and crush Bowser into a pan-a cake, but more likely he would wind up stuck in a stone wall or a hill, unable to move. Or even more likely, his Kart's memory structure would be copied into a non-game area, such as the translation string section, and Luigi's sudden mysterious appearance might manifest as nothing more than garbled speech from one of the other characters.

There are many possibilities, too many to imagine. The most important takeaway here is that no matter how unlikely Luigi's safe intra-universe migration may be, it is possible. With some external help from the host system, it would even be straightforward: perhaps no more than a few days work from a skilled programmer.

In real-world programs sharing adjacent address spaces, it's improbable (but possible!) that migrating data from one process to another in a randomly chosen destination spot would have semantically meaningful results in the destination address space.

To put it in simpler terms: under just the right circumstances, Luigi could teleport to the other side, and wind up in a different world.

Ghosts in the machine

An embedded system is a little environment. My Betta in the previous installment of this little series lived in a very simple embedded system. Its most obvious difference from my own environment is that the tank was filled with water, while my room was filled with air. The differences from the host system can yield different behavior and different rules. In the fish tank, the rules are only a little different — for example, most things are more buoyant inside the tank. In a virtual embedded system, the rules and behavior might be completely different from those of the host system. It all depends on how the embedded system was designed to work.

Every embedded system has a containing surface, which might be called its frontier. The frontier is one-sided in the sense that the rules of the embedded system may simply stop at the boundary: there is no "other side". In some embedded systems (such as a Euclidean space embedded in a non-Euclidean space), even the supposedly intuitive notion of extending a line across the boundary only has meaning on one side, the "inside" of the frontier.

If there is a formal mathematical term for this One-Sided Frontier idea, I've yet to come across it, and I've spent quite some time looking. If you have any suggestions, let me know. The closest I could find are the spaces that are undefined in noncontinuous functions, such as the Tangent function for a value of 90 or 270 degrees. If you ask a question f(x) at these values of x, the answer is something like "You can't go that way."

So the other side of the frontier is undefined. This statement has a deep, almost philosophical meaning to programmers. It does not mean "nothing"; no, no no! It means anything. It is a modern-day Wand of Wonder, a Bag of Tricks, a Horn of Plenty, a Box of Chocolates. You never know what you're going to get.

If a computer program's code inadvertently reaches into undefined, then it has stepped across a mystical wall into another universe, and anything could happen. It's as if Gremlins have taken over the machine. Your PC speaker might beep erratically or continuously. Your video array might burst into random fireworks. Your printer might fire up and print out mysterious symbols. Your program may even continue to operate normally, but with the addition of unexplainable and unrepeatable phenomena.

I have seen all these things happen. All C/C++/assembly programmers have seen bugs like this — program bugs with "crazy" behavior. The bugs are so spooky, and so hard to track down, that computers are now designed to be gatekeepers at the Wall, and when you try to step across they say STOP! It's much better to be blocked immediately than to let your program wander into undefined, where ghosts may take hold of your data in ways that may even cause your legal department to take notice. Stranger things have happened, as they say.

This computerized gatekeeping is commonly called "protected mode". The computer checks every addressing instruction, and any time a program tries to access the memory area outside its own, things halt immediately. When you see the message "segmentation fault", or a blue screen of death, or some other sign that a fatal, unrecoverable program error has occurred, it is almost always because someone or something in the embedded system tried to escape.

Holes in the Wall

From the perspective of a computer game, the system frontier is relatively uninteresting. It's not much different from any other obstacle.

However, in other embedded systems these frontiers are almost mystical in nature. They provide endless fascination for computer scientists working in the domain of programming languages, which are yet another kind of embedded system.

To see how it works, let's consider the situation in Mario Kart, which is simpler.

In Mario Kart, most of the racers are computerized opponents, or AIs. A few of them, usually at least one, are controlled by people, using steering wheels or other physical controllers. Sometimes (e.g. in simulation mode) all the opponents may be computerized.

In order for people to interact with the embedded system, there must be some way to send information back and forth across the system frontier.

Coming from the Mario Kart side, we typically have video and sound signals. The embedded system generates these signals and sends them to your TV or device.

Coming from the people side, we typically have motion input: which way to move the Karts. The signals begin as your physical motions: buttons you press, or in newer controllers, the direction you tilt the controller. They are sent from the host system to the embedded system and they generally produce observable effects in the embedded system. Like magic!

Mario Kart is especially interesting because the external camera is physically present within the game. You can see it in the instant replays of your races: a little guy with a camera, floating behind you on a little cloud. The game designers have arranged things so that you can almost never see him while racing, because he's always flying behind your head, along your line of sight.

But he's there. And in fact that camera is always there, in all video games. It's just that the Mario Kart designers have chosen to reify him as a cute little turtle guy with a camera, floating behind you on a cloud.

The external camera is a one-way hole in our one-way wall: it sends data out of the embedded system. For all intents and purposes it also sends the sound data, since the sounds are usually scaled as a function of distance from the camera.

During a race, there's a lot of stuff going on in the embedded system, and there's a lot of stuff going on in the host system. But they constrain their communication via mostly-invisible channels, and these channels are restricted in the kinds of communication they may convey. Your controller can send motion inputs, but (at least today) it can't send video data. And (at least today) the characters in the game can't control your arms and legs, the way you can control theirs.

Hopefully now you should have a mental picture of this magical wall between an embedded system and the Great Undefined Beyond on the "other side" of its system frontier. The wall may have deliberate holes in it: channels, really. Information may flow across these channels in predefined ways. And the channels are almost always invisible to occupants of the embedded system.

Reflections

I've spent a lot of effort telling you about a rather strange, twisty kind of surface: the frontier of an embedded system. This frontier exists for all embedded systems. It may have "holes" (data channels) in it. The number and nature of these channels is entirely up to the designers of the embedded system. Programmers sometimes call these channels "reflection" or "metaprogramming".

The holes in the frontier may or may not be detectable within the embedded system itself. They may only be detectable within the host system. This, again, is up to the designers. For most of the embedded systems I know of, the channels are "well defined", in the form of application programming interfaces offered to either the embedded system or the host system for communicating across the frontier.

But you need a channel for this kind of communication. Setting it up is usually not cheap. And setting up meta-reflection (in other words, being able to "see" the channel from within the embedded system or host system) is even more work.

So most of the time, channels through the embedded system frontier are completely invisible and undetectable from inside the embedded system. They're quite real, and information flows either one way or both ways, but they cannot be detected from within the embedded system, and their behavior is intrinsically non-repeatable via experimentation.

When data comes across these invisible channels, stuff "just happens" in the embedded system, with no clear indicator nor explanation as to why.

In our discussion so far, I have intentionally blurred the distinction between the host system (such as a fish tank or a game console device) and the host system's host system (such as your bedroom or living room). But you've probably noticed by now that all host systems are embedded in some larger system. This is just the way things work. The fish tank is in your bedroom, which is a system embedded in a house, which is a system embedded in a neighborhood, embedded in a county, a nation, a continent, a planet, a solar system, a galaxy, a universe.

It's perhaps not as clear in the case of fish tanks, but host systems often overlap and even cooperate. A city is composed of many interleaved subsystems. So is your body. It's not always a simple containment relationship. Systems are made of, and communicate with, other systems.

But one way or another, all systems are embedded systems.

There is no reason to assume that our Universe works any differently. Our Universe is a system; that much should be self-evident. It has boundaries; astronomers and astrophysicists have recently even determined that the boundary appears to be a dodecahedron.

We are already painfully aware of the question "what happened before the Big Bang, if in fact the Big Bang occurred in the way all the evidence suggests", and its inherent nonsensicality. What happened before the beginning of Time? What lies beyond the end of the Universe?

Programmers already know intuitively the answer to these questions. The answer is: undefined is there. Undefined is not the same as "nothing", no indeed. It's anything. And we can't go that way.

This has ramifications for the way we think about things today.

I believe I will have more to say about this soon. Right now I think I will go play Mario Kart: a game as fun as any I think I've ever played.

Fable II: Arguably Better than Getting Your Head Crapped On

2008-12-25T06:39:00.000-08:00

I finished Fallout 3 maybe six or eight weeks ago, and it was hands-down one of the best games I've ever played. A game like that gets you in the mood for more gaming, so I thought to myself: "Hey, I should plop down $160 for Fable II!"

Actually that's not exactly what I thought, but it's what happened. I bought the game for $60, fired it up, got up to the part in the intro where a bird craps on your head (yes, this is how it starts), and it locked up hard. Reset the XBox, tried again, and this time got as far as some guy selling snake oil gadgets before it locked up again. Snake oil, indeed.

I tried playing for about an hour, with the game crashing every 3 to 5 minutes, and I finally went online to read about how it kills XBoxes and it's the Game of Death and blah blah blah, all interesting but not especially helpful. Eventually I stumbled across discussions of the "install to hard drive" option. Nobody actually said how to do it, so it took another hour of digging to deduce that you need to purchase a $100 wireless network adapter (or 100 feet of network cable, I guess). So I shut it down for the night, waited for the stores to open, forked over the $100, and installed the game to the hard drive. To Lionhead's credit the game never crashed again, making it significantly more stable than Oblivion or Fallout 3.

I tried hard to like Fable 2. I didn't even need to like it $160. I would have settled for a $60 value. I vaguely remember liking Fable 1, although I can't remember anything about the game except for one neat scene where you had to escort two NPCs through a dark valley. One of the NPCs has been bitten by a Balverine (a werewolf), and the two argue the whole trip about whether he's going to turn. It's a funny conversation and the scene has a funny ending. Other than that, I just have vague recollections of shooting birds on the roof of some guild, and needing to get a 6-foot handlebar moustache for some side quest. The rest of it is basically a blank. But I had set some flag to the effect that "I liked it," and I wanted to like the sequel too.

Unfortunately, with a few noteworthy exceptions that I'll call out in the "Highlights" below, the game is entirely forgettable. It's already fading from memory as we speak. It wasn't as bad as some people make it out to be. It's playable for a couple of days, and it has its fun moments. But it's not a very good game, and it's definitely not a very memorable game. This is sad, considering the amount of effort that went into its development.

The no-spoiler synopsis of Fable 2 is that it's a bad Zelda clone. You can smell the desperation; there are dozens and dozens of direct rip-offs from the Zelda franchise. Heck, I wouldn't have minded a half-decent Zelda clone; they're some of the best games of all time. But Fable 2 misses the mark by a mile. The humor is juvenile bordering on imbecilic, the hints are hamfisted, the areas are small and cramped, the minigames are lackluster, the music is virtually nonexistent, and the story pacing is rushed and breathless. It's a cargo-cult copy of Zelda that winds up having no identifiable soul: forgettable across the board.

I've given up every piece of Microsoft software and hardware I own except for the XBox, which I had been holding onto just for Fable 2. Now that it's come and gone like a bird crapping on my head, I'm giving up. No more XBox or PC games for me. Ever.

Hence, Fable 2 cost me $160. I hope you got it for cheaper than that.

Anyway, here's a quick rundown of the lowlights and highlights of the game, as I see them. Enjoy!

Lowlights

1) Humor. Fable 2 tries hard to incorporate humor into the game — too hard. The writers use the trusty old "stopped clock" approach to humor, in which they inundate you with jokes, and 1 out of every 43,200 of them is funny. Amazingly, this perseverance leads to 3 or 4 genuinely funny ones, mostly near the entrance to the Crucible (arena). But by the time you get there, you've already tuned out all attempts at humor and have probably even tried killing yourself a few times. So they may fail to register.

2) Theresa. The game features an old lady who watches everything you do and talks at you constantly. This starts in the very beginning of the game and lasts until the very end, with no option to turn her off. Your character can't so much as take a crap without Theresa piping in with helpful advice on which hand to use. "That is ancient paper. Be cautious." She uses some magical form of communication system that only breaks down in the fog — probably shortwave radio — and there's no way to turn the fugging thing off.

I really hated Theresa.

3) Expressions. You can't talk to people. Instead, the game gives you a series of increasingly repugnant forms of nonverbal communication. Initially you're limited to belching, farting, giving people the finger and making lewd pelvic thrusting motions, but as you rise in fame Theresa informs you that you've earned the right to use the "kiss my ass" expression. I am not making this up. I tried to avoid using expressions altogether, but the game forces you to do it once in a while. Made me want to take a shower.

4) Retinal blindness. Fable 2 is nauseatingly saturated. They just don't know how to lay off the paint gun. There are a few OK-ish-ly tasteful areas, such as the big trees in Bower Lake, but most of the game is a frightully garish mix of lime greens, oranges, purples, reds, blues, and general oil-spill iridescence. It makes you color-blind fast, even if you didn't start that way. Finding anything onscreen is like trying to spot where someone threw up on a Matisse.

5) Linearity: the game is unrelentingly linear for the first hour or two (a _long_ time), after which it settles into, well, linearity. The gameplay occasionally approaches the smashing-through-lines-of-baddies feel of Gauntlet Legends, which I liked, but mostly it makes you feel like a rat running a big maze, following a neverending golden trail of cheese.

A major contributor to the linear feel, even after the game opens up, is the plethora of tiny little fences and obstacles that you can't hop. It makes it really hard to know where you can walk, and it feels like you're constantly bumping into things, because, well, you are. So the game is linear at all resolutions: high (the plotline), medium (most of the level designs) and low (the path designs). Linearity can cramp even the best of games — Kingdom Hearts comes to mind. It's just a bad way to design things. And Fable's linearity felt especially suffocating after just having finished Fallout 3, which is immense and wide-open.

6) Controls. It's been a long time since I played a game whose controls were so accident-prone. Normally a game's controls take some getting used to, and then it's like driving a car. In Fable 2, even after days of play, I'd still be trying to hop a fence and wind up shooting the front door off a mansion, blowing boards everywhere and scaring the shit out of the villagers.

Hell, even when I was trying to buy my final sword (this was a $50k sword I'd been looking for all day), I tried to bring up the "buy sword" menu for the blacksmith, and I accidentally wound up casting a massive Inferno spell, causing him to literally run screaming across town. It was weeks of in-game time before I saw him again. God dammit. They really should have had different controls in safe zones.

7) Elephantine mammary glands. I don't know what planet these guys have been living on, but giant udders fell out of fashion at least two or three decades ago. Every single woman in Fable 2 had knockers significantly bigger than her head. It reminded me of my trip to Paris, where every statue of a woman is bare-breasted, presumably so that you can tell it's a woman — a practice which unfortunately suggests that there's really no other way to tell. Dumb French statue-making assholes.

I mean, the people in Fable 1 were ugly — the main character worst of all. They all had this "I'm a programmer who never gets outdoors" look, and I expected (and got) no better from Fable 2. But I was really disappointed that every female in the game was a fugging dairy farm. I mean, someone with some taste and maturity should have a talk with these asshats, and explain to them what women actually look like. Or they should pick up a frigging Victoria's Secret catalog or watch a goddamn Target commercial or something. Jesus.

The milk jug thing... it was really just too much. I have zero respect for those jerk-offs at Lionhead.

8) Ass-kissing. This was probably the most serious problem with the game. It was a disease in Fable 1 that went malignant in Fable 2. Whoever designed these games was apparently neglected as a child, because the gameplay revolves around gaining "renown". Lionhead's hopelessly adolescent view of "renown" is that villagers should follow you around and say things like "yay!" and "hurrah!" It's even worse than I'm making it sound.

They spent so much time coding this crap that they forgot to code pushing into the game: you can't push people out of your way. So as soon as you wander into a dead-end alley you're fucked: a bunch of people will crowd in after you asking for autographs and offering you gifts and all this sickening bullshit.

To the game's credit, and I count this as a highlight, if you pull out your six-barreled rifle, take the safety off, and aim right at their heads, it clears everyone out pretty fast. You can imagine how desperate I was by the time I tried that approach. But they coded it correctly, bless 'em.

9) Too Easy. The game just wasn't hard, period. There were no hard fights. I never died. I don't even know what happens when you die in Fable 2. I used a couple of Resurrection Phials, but only because I had become so lazy in combat that I didn't care anymore. This was a serious flaw in the game: it essentially removed the element of fear, which was the only emotion (other than disgust) that the game had a chance of evoking.

10) Demon Doors. Oh man, oh man. These were probably the low point of the whole game. They made me want to puke. I would run past them as fast as I could so I didn't have to listen to their inane drivel. This was some of the worst game writing I've ever seen. I just don't want to talk about it.

11) Misguided innovation. They really should have stuck with copying Zelda, and Kingdom Hearts, and Gauntlet: Legends, and all the other games they copied blindly, and badly. Because whenever they introduced something entirely new, it almost invariably sucked. Examples? OK. Sure. Since you asked, and all.

How about the "innovation" that when you eat nearly any food (and it only takes a few bites), you bloat up to the size of Orson Welles, and the only way to get rid of it is to eat celery. No amount of exercise will make any difference, but eating a few bites of celery makes it go away. Innovative!

Innovation: you can purchase almost any property in the game. Is this realistic? No. In reality, not everything would be for sale (and especially not posted on the front doors). Real-estate transactions wouldn't be instantaneous. You would need the owner present to buy something. Etc, etc. So given that this feature makes the game less realistic, what purpose does it serve? Is it fun? No. Buying real estate is about as fun as attending insurance seminars, so I don't know what the hell they were thinking. It could have been fun in the right setting, with suitable other human participants, in a Parker-Bros. Monopoly kind of way. Maybe. But slapping it on the side of an RPG and calling it innovation? It boggles the mind.

And what about the the busywork jobs (blacksmith, woodcutter, bartender) for making gold? Um, dudes -- busywork jobs only exist in MMORPGs to limit per-player CPU usage. They're not fun. "Innovatively" bringing them into a single-player game was just flat-out brain damaged.

12) Nonexistent target audience. What age group is the target market for this game? If you enumerate the possibilities, you arrive at the inescapable conclusion that the game was either (a) created by imbeciles, or it was (b) created for imbeciles, or possibly (c) all of the above.

It's presumably not intended for kids, or you wouldn't be finding condoms in treasure chests, soliciting and obtaining sex from male and female prostitutes of all shapes and sizes, performing pelvic thrusts to solve quests, and so on.

It's not for adults either, or you wouldn't be bombarded with the constant barrage of scatological humor, beginning with the bird shitting on your head, continuing with warnings about "extending the fart command and messing it up", and going pretty much straight downhill from there.

Is it intended for teenagers, then? Poooossibly, but (a) that ignores their primary demographic, which is 30-year-olds, and (b) I don't know any teenagers that are that stupid, nor so hard-up for attention that they need AI villagers to yell "hurray!" whenever you pass them, even if you're in a graveyard at midnight.

Dipshits. This game was designed by dipshits. The coding was great, the artwork was great, the sound effects were great; the details were for the most part rock-solid. But the creative direction was just inexcusably bad.

Highlights

OK, I've been pretty tough on the game so far. Fable 2 did actually have a few genuine highlights worth calling out. You could even argue that these highlights make the game almost worth playing, in spite of all the crap you have to suffer through in order to get to them.

1) Banshees. Fable 2's banshees are, in a word, awesome. I've been racking my brain trying to think of a VG monster as cool as these banshees in any game I've ever played. I'm coming up with a few ties, but nothing that beats them. The YouTube videos don't come close to doing them justice. Fable II is worth playing just to get to Wraithmarsh.

The only real problem with the banshees is that since none of the combat is challenging (see Lowlight #9), they're nowhere near as scary as they could have been. But they're amazingly stylish. I'd call them an innovation, but I'm inclined to believe Lionhead stole their basic design from some other game, given all the other copying they've done. (The Fable 2 Trolls, for instance, are about as Zelda-clone-esque as you can get without inviting a lawsuit.)

2) Lucien's speech, where he addresses the recruits. Really great speech. Riveting and convincing. Amazing how Microsoft-run studios that are so consistently bad at humor are so good at creating convincingly evil speeches about taking over the world.

Actually the whole centerpiece drama in the tower was very nicely done. I have to give them credit for that part of the game: it was exceptional by any standard. It basically saved the game from being a total loss.

3) Hammer. She's cool. Great voice acting, surprisingly good scripting, neat character, lots of depth. One of the better-realized VG supporting characters I've seen in many years.

4) The dog. Apparently there was a lot of hype about the dog. Or so I hear, after actually having played the game. Whatever the hype, the reality is that it's a very believable dog. I especially liked how it would run ahead of you -- I've seen pets that follow you, but the dog would often anticipate your direction and run ahead, kinda turned back towards you like "c'mon! let's go!" I encountered no glitches with the dog; the coding was rock solid. Overall it was, well, very... doggy. And what more could you ask for in a dog, really?

As a tribute to the believability of the dog, I'll offer a minor spoiler. (Skip ahead if you don't want a spoiler!) At the end of the main storyline, you are granted one wish. Your choices are: (a) get all the people who died back, (b) get your dog back, or (c) get a bunch of money. What I really wanted was a sort of amalgam of the 3 choices: I wanted my money back for this dog of a game. But when push came to shove, I picked the dog. I kinda missed him.

5) Architecture. Overall the architecture was really nice. The only somewhat dubious exception was Bowerstone, which looks almost exactly like Euro Disney. I kept expecting Tigger to come waltzing around, cursing in French under his breath, just like he did on my real-life trip to Euro Disney a few years back.

Other than the Euro Disney influence, which I could take or leave, I thought the architecture was nice throughout the game. I liked the waterfront town of Bloodstone. I liked the manors in Oakfield. I liked the gypsy wagons. I liked the vendor carts. I loved pretty much every creepy structure in Wraithmarsh. The overall look of the game was beautiful, once you got past the color-saturation problem, and the architecture was a huge contributor.

6) Fight music. Unlike in Fable 1, most of the music in Fable 2 is forgettable background/atmosphere music. They didn't get Danny Elfman this time around, and it shows. The theme for Bower Lake is nice as far as it goes, which is exactly 2 chords over and over and over. But it's still OK. The rest of the music didn't leave any sort of impression on me at all, except for the fight music, which almost made up for everything. It was very good. There were at least two fight themes and both of them were cool. If only the rest of the music had been... present. It was like it wasn't even there. It phoned in its performance.

Folks at Dorkhead studios: Zelda's music is one of the top five reasons for its success as a franchise. Same goes for Mario and Kingdom Hearts and Final Fantasy. Their music is always great, and it's always in your face. The music isn't muttering or mumbling; it's shouting. And they can get away with it because it's always great. Even when it's bad or annoying, which is rare, the music still anchors each place and event in the game in your memory, in a way that only music can. You guys really screwed the pooch on this one.

7) Mixed-tactic fighting. They did a great job of setting things up so that you could use melee, ranged weapons and magic effectively in combat. It was refreshing to be able to switch styles in mid-fight: you could use your sword to kill everything near you, then start blasting everything ten feet or more distant with your rifle. Or you could clear a little space and cast a time-slowing spell, and then just start zinging around whaling on bad guys. The combat was never hard, but it was on the whole fairly satisfying.

The downside of ultra-convenient access to melee, range weapons and spells was that you could effortlessly use them all simultaneously while trying to buy vegetables from a produce stall in the main market. I really wish they'd made it just a teeny bit harder to cast spells in public areas.

8) Well, damn. I can't think of a Highlight #8. I thought of some more lowlights, though: long area loads, unresponsive controls during "scenes", only a handful of available spells, months of coding/design effort wasted on useless features like "groin shots" and tatoos...

Oh, and the lack of control over when quest scenes actually unfold — they trigger from proximity to the relevant NPC rather than interacting with the NPC because, oh, that's right, you can't interact with anyone except to fart on them or give them the finger. Oops! So you're always accidentally wandering into a dungeon that triggers some quest, and there's no way out except to back entirely out of that phase of the quest, which may involve losing hours of your time, all because you walked through the wrong door. Damn that pissed me off.

And how the hell do you sleep in an Inn? I never managed to figure it out. I'd wind up spending $10k for some hovel just to get a frigging bed to sleep in. It was amazingly bad UI design, if there even is a way to do it. If not, then their helpful tutorial message lied to me at least a dozen times.

Argh. Well, this highlights section is going downhill in a hurry, so I think I'll end it here.

Better than a crap on the head?

Maybe, maybe. But compared to Fallout 3, Fable 2 pretty much sucked. It had a couple of nice features, but they were drowned in an ocean of painfully adolescent design. Such a shame.

I've tried to be fair here. I don't mean to discourage you from playing the game, since for all I know there's nothing better out there right now.

If you do decide to play it, I hope I've set your expectations very low. That way, well, who knows? You might actually have some fun with it.

But if you open even one of those Demon Doors I'll lose all respect for you.

Ejacs: a JavaScript interpreter for Emacs

2008-11-16T22:19:00.000-08:00

So! I have all these cool things I want to write about, but I broke my thumbnail. Can you tell that's a long story?

See, this summer I got excited about playing guitar again. I usually switch between all-guitar and all-piano every other year or so. This summer I dusted off the guitars and learned a bunch of pieces, and even composed one. I was prepping for — among other things — a multimedia blog entry. It was going to have a YouTube video, and a detailed discussion of a wacky yet powerful music programming language you've probably heard of but never used, and generally just be really cool.

And then it all came crashing down when I busted my thumbnail off. And I mean off — it broke off at least a quarter inch below where the nail and skin meet. Ick. I just accidentally jabbed my steering wheel, and that was that.

I remember reading an interview with some dude who said he had punched a shark in the nose. He said it was like punching a steering wheel. So now I know more or less what it's like to punch a shark in the nose, I guess. There's always an upside!

Anyway, that was going to be my magnum opus (literally: Op. 1) for the year, but it fell through for now. I'll have to revisit the idea next year. My thumbnail's back, but it's been at least 2 months since I touched my guitar, so I'll have to start over again.

Work has been extraordinarily busy, what with having to collect all these Nuka-Cola Quantum bottles and so on. I'm sure you can imagine. So I haven't had much time to blog lately.

But I do like to publish at least once a month, whether or not anyone actually cares. It's been about a month, or it feels that way anyway, and all I have to show for it is this box of Blamco Mac and Cheese.

So I'm cheating this month.

You know how on Halloween how you walk around in your costume holding your little bag and you say "trick or treat", and every once in a while some asshole does a trick instead of dumping half a pound of candy into your bag? And then he has to explain to all the dumbfounded and unhappy kids that "Trick or Treat" means that a trick is perfectly legal according to the semantics of logical-OR, and the kids remember that a-hole for the rest of their childhoods and avoid his house next year?

Yeah.

So I'm doing a trick this time. Hee. It's actually kind of fun when you're on the giving end.

My trick is this: in lieu of saying anything meaningful or contemporarily relevant, I'm writing about something I did over a year ago. And there isn't much to say, so this really will be short.

Ejacs

Around a year ago, I wrote a blog called Stevey's Boring Status Update, mostly in response to wild rumors that I'd been fired from Google. Not so. Not yet, anyway.

In that blog I mentioned I was working nights part-time (among other things) on a JavaScript interpreter for Emacs, written entirely in Emacs Lisp. I also said I didn't have a name for it. A commenter named Andrew Barry suggested that I should not call it Ejacs, and the name stuck.

Ejacs is a port of Narcissus. Narcissus is a JavaScript interpreter written in JavaScript, by Brendan Eich, who by pure unexpected coincidence is also the inventor of JavaScript. Narcissus is fairly small, so I thought it would be fun to port it to Emacs Lisp.

It turns out Narcissus is fundamentally incomplete. It cheats. It's that trick guy on Halloween. Narcissus has a working parser and evaluator, but for its runtime it calls JavaScript. So it's kind of like saying you're building a car by starting from scratch, using absolutely nothing except for a working car.

This meant I wound up having to write my own Ecma-262 runtime, so that the evaluator would have something to chew on. In particular, the Ecma-262 runtime consists of all the built-in properties, functions and objects: Object, Function, Array, String, Math, Date, RegExp, Boolean, Infinity, NaN, parseInt, encodeURIComponent, etc. A whole bunch of stuff.

I basically did this by reading the ECMA-262 specification and translating their algorithms into Emacs-Lisp. That spec is lousy for learning JavaScript, but it's absolutely indispensable if you're trying to implement JavaScript.

I didn't know Emacs-Lisp all that well before I started, but boy howdy, I know it now.

Emacs actually has a pretty huge runtime of its own — bigger than you would ever, ever expect given its mundane title of "text editor". Emacs has arbitrary-precision mathematics, deep Unicode support, rich Date and Calendar support, and an extensive, fairly complete operating system interface. So a lot of the porting time was just digging through Emacs documentation (also extensive) looking for the Emacs version of whatever it was I was porting. That was nice.

I had big plans for Ejacs. I was going to make it a full-featured, high-performance JavaScript interpreter, with all the Emacs internals surfaced as JavaScript native host objects, so you could write Emacs customizations using object-oriented programming. It was totally going to be the "meow" part of the cat.

And then I broke my thumbnail.

Actually, what happened was js2-mode.

js2-mode

After I got the interpreter working, I was at this crossroads. There were two big use cases: a JavaScript editing mode, or a JavaScript Emacs development mode. Both were going to be a lot of work.

It turns out you really want the editing mode first, if possible, so that when you're doing all your JavaScript programming you have a decent environment for it. So I picked the editing mode.

Unfortunately I found the Ejacs parser wasn't full-featured enough for my editing needs, since at the time I was working on my Rhino-based Rails clone and writing tons of JavaScript 1.7 code on the JVM.

I spent a little time trying to beef up the parser, then realized it would be a lot faster to just rewrite it by porting Mozilla Rhino's parser, which is (only) about 2500 lines of Java code. Ejacs is something like 12,000 lines of Emacs-Lisp code, all told, so that didn't seem like a big deal.

So I jumped in, only to find that while the parser is 2500 lines of code, the scanner is another 2000 lines of code, and there's another 500 or so lines of support code in other files. So I was really looking at porting 5000 lines of Java code.

Moreover, the parse tree Rhino builds is basically completely incompatible with the Ejacs parse tree. It was richer and more complex, and needed more complicated structures to represent it.

So after I'd ported the Rhino parse tree, what I really had was a different code base. I went ahead and finished up the editing mode, or at least enough to make it barely workable (another 5000 lines of code), and launched it. It was a surprisingly big effort.

And it left poor Ejacs lying unused in the basement.

So today, faced with nothing to write about, I figured I'd dust off Ejacs, launch it with lots of fanfare, and then you'd hardly notice that I cheated you. Right?

You're not coming to my house next year. I can tell already.

Anyway, here's the code: http://code.google.com/p/ejacs/

There's a README and a Wiki and installation instructions and stuff. I can't remember how to put the code in SVN, and I'm having trouble finding it on the code.google.com site. As soon as I figure it out I'll also make it available via SVN.

Emacs Lisp vs. JavaScript

In the interests of having something resembling original worthwhile content today, I'll do a little comparison of Emacs Lisp and JavaScript. I know a lot about both languages now, and a few folks mentioned that a comparison would be potentially interesting.

Especially since I think JavaScript is a better language.

So... the best way to compare programming languages is by analogy to cars. Lisp is a whole family of languages, and can be broken down approximately as follows:

Scheme is an exotic sports car. Fast. Manual transmission. No radio.
Emacs Lisp is a 1984 Subaru GL 4WD: "the car that's always in front of you."
Common Lisp is Howl's Moving Castle.

This succinct yet completely accurate synopsis shows that all Lisps have their attractions, and yet each also has a niche. You can choose a Lisp for the busy person, a Lisp for someone without much time, or a Lisp for the dedicated hobbyist, and you'll find that no matter which one you choose, it's missing the library you need.

Emacs Lisp can get the job done. No question. It's a car, and it moves. It's better than walking. But it pretty much combines the elegance of Common Lisp with the industrial strength of Scheme, without hitting either of them, if you catch my drift.

Anyway, here's the comparison. Here's why I think JavaScript is a better language than Emacs Lisp.

Problem #1: Momentum

A recurring theme is that Elisp and JavaScript both will both exhibit a particular problem, and there are specific near-term plans to fix it in JavaScript, but no long-term plans to fix it in Elisp.

It's easier to resign yourself to a workaround when you know it's temporary. If you know the language is going to be enhanced, you can even design your code to accommodate the enhancements more easily when they appear.

People are working on improving JavaScript. It's not happening quite as fast as I'd hoped earlier this year, but it's still happening. As far as I know, Emacs Lisp is "finished" in the sense that no further evolution to the language is deemed necessary by the Emacs development team.

Problem #2: No encapsulation

Every symbol in Emacs-Lisp is in the global namespace. There is rudimentary support for hand-rolled namespaces using obarrays, but there's no equivalent to Common Lisp's in-package, making obarrays effectively useless as a tool for code isolation.

The only effective workaround for this problem is to prefix every symbol with the package name. This practice has become so entrenched in Emacs-Lisp programming that many packages (e.g. apropos and the elp elisp profiler) rely on the convention for proper operation.

The main adverse consequence of this problem in practice is program verbosity; it makes Emacs-Lisp more difficult to read and write than Common Lisp or Scheme. It can also have a non-negligible impact on performance, especially of interpreted code, as the prefix characters can approach 5% to 10% of total program size in some cases.

The problems run slightly deeper than simple verbosity. Without namespaces you have no real encapsulation facility: there is no convenient way to make a "package-private" variable or function. In practice there's little problem with program integrity; it's hard for an external package to change a "private" variable inadvertently in the presence of symbol prefixes. However, it makes it annoyingly difficult for users of the package to discern the "important" top-level configuration variables and program entry points from the unimportant ones. Elisp attempts a few conventions here, but it's a far cry from real encapsulation support.

JavaScript also lacks namespaces. They're being added in ES/Harmony, but in the meantime, browser JavaScript code typically uses the same name-prefixing practice as Emacs-Lisp.

However, JavaScript has lexical closures, which provide a mechanism for creating private names. One common encapsulation idiom in browser JavaScript is to wrap a code unit in an anonymous lambda, so that all the functions in the code unit become nested functions that close lexically over the top-level names in the anonymous lambda. This trick is nowhere near as effective in Emacs-Lisp, for several reasons:

elisp is not lexically scoped and has no closures
elisp nested defuns are still entered into the global namespace
CL's `flet' and `labels' are only weakly supported, via macros, and they frequently confuse the debugger, indenter, and other tools.

Some elisp code (e.g. much of the code in cc-engine) prefers to work around the namespace problem by using enormous functions that can be thousands of lines long, since let-bound variables are slightly better encapsulated. Even this is broken by elisp's dynamic scope:

(defun foo ()
  (setq x 7))

(defun bar ()
  (let ((x 6))
    (foo)
    x))  ; you would expect x to still be 6 here

(bar)
7  ; d'oh!

So let-bound variables in elisp can still be destroyed by your callee: a dangerous situation at best.

Emacs is basically one big program soup. There's almost no encapsulation to speak of, and it hurts.

Problem #3: No delegation

One of the big advantages to object-oriented programming is that there is both syntactic support and runtime support for automatic delegation to a "supertype". You can specialize a type and delegate some of the functionality to the base type. Call it virtual methods or prototype inheritance or whatever you like; most successful languages support some notion of automatic delegation.

Emacs Lisp is a lot like ANSI C: it gives you arrays, structs and functions. You don't get pointers, but you do get garbage collection and good support for linked lists, so it's roughly a wash.

For any sufficiently large program, you need delegation. In Ejacs I wound up having to implement my own virtual method tables, because JavaScript objects inherit from Object (and in some cases, Function, which inherits from Object).

Writing your own virtual method dispatch is just not something you should have to do in 2008.

Problem #4: Properties

I wrote about this at length in a recent blog post, The Universal Design Pattern. JavaScript is fundamentally a properties-based language, and it's really nice to be able to just slap named properties on things when you need a place to store data.

Emacs Lisp only offers properties in the form of simple plists – linked lists where the odd entries are names and the even entries are values. Symbols have plists, and symbols operate a little bit like very lightweight Java classes (in that they're in the global namespace), but that only gets you so far. If you want the full JavaScript implementation of the Properties Pattern, you'll have to write a lot of code.

And so I did. Your implementation choice for object property lists has a huge impact on runtime performance. Emacs has hashtables, but they're heavyweight: if you try to instantiate thousands of them it slows Emacs to a crawl. So they're no good for the default Object property list. Emacs also has associative arrays (alists), but their performance is O(n), making them no good for objects with more than maybe 30 or 40 properties.

I wound up writing a hybrid model, where the storage starts with lightweight alists, and as you add properties to an object instance, it crosses a threshold (I set it to 50, which seemed to be about right from profiling), it copies the properties into a hashtable. This had a dramatic increase in performance, but it was a lot of work.

I experimented with using a splay tree. I implemented Sleater and Tarjan's splay tree algorithm in elisp; Ejacs comes with a standalone splay-tree.el that you can use in your programs if you like. I was hoping that its LRU cache-like properties would help, but I never found a use case where it was faster than my alist/hashtable hybrid, so it's not currently used for anything.

And then in the end, after I was done with my implementation, it was a library (at least from the Emacs-Lisp side of the house). It wasn't an object system for Lisp. It's only really usable inside the JavaScript interpreter, where it has syntactic support.

You really want syntactic support. Sure, people have ported subsets of CLOS to Emacs Lisp, but I've always found them a bit clunky. And even in CLOS it's hard to implement the Properties Pattern. You don't get it by default. CLOS has lots of support for compile-time slots and virtual dispatch, but very little support for dynamic properties. It's not terribly hard to build in, but that's my point: for something that fundamental, you don't want to have to build it.

Problem #5: No polymorphic toString

One of the great strengths of JavaScript is the toSource extension. I don't know if they support it over in IE-land; I haven't been a tourist there in a very long time. But in real versions of JavaScript, every object can serialize itself to source, which can then be eval'ed back to construct the original object.

This is even true for functions! A function in JavaScript can print its own source code. This is an amazingly powerful feature.

In Emacs Lisp, some objects have first-class print representations. Lists and vectors do, for instance:

(let ((my-list '()))
  (push 1 my-list)
  (push 2 my-list)
  (push 3 my-list)
  my-list)
(3 2 1)

(let ((v (make-vector 3 nil)))
  (aset v 0 1)
  (aset v 1 2)
  (aset v 2 "three")
  v)
[1 2 "three"]

But in Emacs Lisp, many built-in types (notably hashtables and functions) do NOT have a way to serialize back as source code. This is a serious omission.

Also, trying to print a sufficiently large tree made entirely of defstructs will crash Emacs, which caused me a lot of grief until I migrated my parse tree to use a mixture of defstructs and lists. Note that simply typing the name of a defstruct, or passing over it ephemerally in the debugger, will cause Emacs to try to print it, and crash. Fun.

The problem of polymorphic debug-printing (or text-serialization) is, I think, a byproduct of Emacs not being object-oriented. If you want a debug dump of a data structure, you write a function to do it. But Emacs provides a half-assed solution: it debug-prints lists very nicely, even detecting cycles and using the #-syntax for representing graph structures (as does SpiderMonkey/JavaScript). But it has no useful debugging representation for hashtables, functions, buffers or other built-in structures, and there's no way to install your own custom printer so that the debugger and other subsystems will use it.

So it sucks. Printing data structures in Emacs just sucks.

The situation in Ecma-262-compliant JavaScript really isn't that much better, although you can at least install your own toString on the built-ins. But any competent "server-side" JavaScript implementation (i.e. one designed for writing real apps, rather than securely scripting browser pages) has a way to define your own non-enumerable properties, so you can usually override the default behavior for things like toString and toSource.

And all else being equal, at least JavaScript functions print themselves.

Emacs advantages: Macros and S-expressions

Pound for pound, Emacs Lisp seems roughly as expressive as JavaScript or Java for writing everyday code. It shouldn't be that way. Emacs Lisp ought to be more succinct because it's Lisp, but it's incredibly verbose because of the namespace problem, and it's also verbose to the extent that you want to use the properties pattern without worrying about alist or hashtable performance.

Elisp does have a few places where it shines, though. One of them is the cl (Common Lisp emulation) package, which provides a whole bunch of goodies that make Elisp actually usable for real work. Defstruct and the loop macro are especially noteworthy standouts.

Some programmers are still operating under the (ancient? legacy?) assumption that the cl package is somehow deprecated or distasteful or something. They're just being silly; don't listen to them. Practicality should be the ONLY consideration.

The cl package wouldn't have been possible without macros. JavaScript has no macros, so even though it has better support for lambdas, closures, and even (in some versions) continuations, there are still copy/paste compression problems you can't solve in JavaScript.

Emacs Lisp has defmacro, which makes up for a LOT of its deficiencies. However, it really only has one flavor. Ideally, at the very least, it should support reader macros. The Emacs documentation says they were left out because they felt it wasn't worth it. Who are they to make the call? It's the users who need them. Implementer convenience is a pretty lame metric for deciding whether to support a feature, especially after 20 years of people asking for it.

Elisp is s-expression based, which is a mixed bag. It has some advantages, no question. However, it fares poorly in two very common domains: object property access, and algebraic expressions.

JavaScript is NOT s-expression based (or it wouldn't be a successful language, many would argue), but it does offer some of the benefits of s-expressions. JSON is one such benefit. JavaScript's declarative object literals (or as a Lisp person would say, "syntax for hashes") and arrays provide a powerful mechanism for designing and walking your own declarative data structures.

JavaScript also has all the usual (which is to say, expected) support for algebraic operators. And unlike Java, JavaScript even got the precedence right, so it's not full of redundant parentheses.

Overall Comparison

In the end, it comes down to personal choice. I've now written at least 30,000 lines of serious code in both Emacs Lisp and JavaScript, which pales next to the 750,000 or so lines of Java I've crapped out, and doesn't even compare to the amount of C, Python, assembly language or other stuff I've written.

But 30,000 lines is a pretty good hunk of code for getting to know a language. Especially if you're writing an interpreter for one language in another language: you wind up knowing both better than you ever wanted to know them.

And I prefer JavaScript over Emacs Lisp.

That said, I suspect I would probably prefer Clojure over Rhino, if I ever get a chance to sit down with the durn thing and use it, so it's not so much "JavaScript vs. Lisp" as it is vs. Emacs Lisp.

I would love to see Emacs Lisp get reader macros, closures, some namespace support, and the ability to install your own print functions. This reasonably small set of features would be a huge step in usability.

However, for the nonce I'm focusing on JavaScript. I've found that JavaScript is a language that smart people like. It's weird, but I keep meeting really really smart people, folks who (unlike me) are actually intelligent, and they like JavaScript. They're always a little defensive about it, and almost a little embarrassed to admit it. But they think of it as an elegant, powerful, slightly flawed but quite enjoyable little language.

I tell ya: if you're a programming language, it's a very good thing to have smart people liking you.

It doesn't make me smart, but I kinda like it too. Even though there's (still) a lot of hype these days about Java, and people tootling on about how Java's going to be the next big Web language... I just don't see it happening. There are too many smart people out there who like JavaScript.

So enjoy the interpreter. Ejacs is just a toy, but I think it also shows a kind of promise. Scripting Emacs using JavaScript (if anyone ever actually implements it) could be really interesting. It could open up the world's most powerful, advanced editing environment to millions of people. Neat.

In the meantime, it doesn't actually do squat except interpret EcmaScript in a little isolated console, so don't get your hopes up.

Reminder — here's the Ejacs URL: http://code.google.com/p/ejacs - enjoy!

And with that, I'm off to find some Nuka-Cola Quantum. I just wish those bastards hadn't capped me at level 20.

A programmer's view of the Universe, part 1: The fish

2008-10-28T21:52:00.000-07:00

I write a column for computer programmers called "Stevey's Blog Rants." It's basically a magazine column — I publish to it about once a month. The average length of my articles is about 12 pages, although they can range anywhere from 4 to 40 pages, depending on how I'm feeling. But for precedent, don't think blogs: think of Reader's Digest. The blog format sets the wrong expectations.

Hence, some people complain that my articles are too long. Others complain that I have not given my arguments sufficient exposition, and that my articles are in fact too short on detail to warrant any credibility. This is a lose-lose situation for me, but I keep at it nonetheless because I enjoy writing. Even if nobody were to read my blog, the act of writing things down helps me think more clearly, and it's engaging in the same way that solving a Sudoku puzzle is engaging.

You should try it yourself. All it takes is a little practice.

My blog topics vary widely, and sometimes I even venture outside the realm of programming. Programming is where I'm most comfortable, and it's also where people seem to ascribe to me some level of punditry: I'm not necessarily right, but even my greatest detractors grudgingly admit that I'm entitled to an opinion, by virtue of my having spent twenty years hacking day and night without any sign of wanting to give it up and turn into a pointy-haired manager.

Even though I love both programming and to a lesser extent writing about it, there are also lots of non-programming topics I'd like to write about. Being a career programmer gives you an interesting perspective on issues not directly related to programming. You start to see parallels. So maybe I'll branch out some more and see how it goes.

The programmer's view

The first thing you notice as a programmer is that it trains you — forces you, really — to think in a disciplined way about complex logic problems. It also gives you a big booster shot of confidence around problem-solving in general. Junior programmers tend to have very high opinions of themselves; I was no exception.

In time, though, programming eventually humbles you, because it shows you the limits of your reasoning ability in ways that few other activities can match. Eventually every programmer becomes entangled in a system that is overwhelming in its complexity. As we grow in our abilities as programmers we learn to tackle increasingly complex systems. But every human programmer has limits, and some systems are just too hard to grapple with.

When this happens, we usually don't blame ourselves, nor think any less of ourselves. Instead we claim that it's someone else's fault, and it just needs a rewrite to help manage the complexity. In many cases this is even true.

Over time, our worldwide computer-programming community has discovered or invented better and better ways ways to organize programs and systems. We've managed to increase their functionality while keeping the complexity under control.

But even with such controls in place, systems can occasionally get out of hand. And sometimes they even need to be abandoned altogether, like a dog that's gone rabid. No matter how much time and love you've put into such systems, there's no fixing them.

Abandoning a system is a time of grieving for the folks who've worked on it. Such systems are like family.

And there's a disturbing lesson at the tail end of such experiences. The scary thing is that it's very easy, as a programmer standing at the precipice of complexity, to envision systems that are orders of magnitude more complex, millions of times more complex, even unimaginably more complex.

In the end, programming shows us how small we are.

The fish's view

Long ago, I used to have a Siamese fighting fish, also known as a Betta splendens, or simply a "betta". You can buy these fish at almost any pet store. I kept my betta, who was a deep vibrant red, in a pretty little 15-gallon tank decorated with a resplendence of real freshwater plants. And for a while I think my betta was happy there.

Like many Americans, I went through a phase in which I kept and ultimately killed many, many tropical fish. I didn't kill them intentionally; I wanted them to live and thrive. But keeping them alive for long is a challenge when you don't live in the tropics. So they might live for a few months or maybe a year, but they would always die prematurely. It was sad, and eventually I could no longer bear it, so I stopped keeping them.

Of all my fish, my betta left the biggest impression on me. The betta is a remarkable fish in several ways. For one thing, bettas are physically beautiful, and when they are at full display, their fins expand, peacock-like, into a fluid rose shape that is undeniably dramatic.

Bettas are also remarkable because they fight. They do not fight to establish a pecking order, as other fish do; they fight to kill. The males display their fins and then fight whenever they see another male betta, or even their own reflection, so they have to be kept alone and away from mirrors.

But bettas, I think, are most remarkable for their intelligence. Of all of the hundreds of tropical fish I kept, only bettas displayed anything resembling intellectual curiosity.

This really makes bettas some of the saddest stories in the tropical fish industry. Like other hobbyist fishes, they are stolen from their natural habitat and shipped overseas, or at best farmed in unsavory conditions. But unlike most other fish, bettas are also dyed to enhance their color. They are generally housed in tiny fist-sized bowls because of their ability to breathe air when necessary. And they are bred to express their fighting genes, and are often made to fight by their owners. Whereas other fish are kidnapped and sold, bettas are abused.

But worst of all, I believe their high intelligence endows them with greater capacity for suffering than other fish species. They can suffer physically and emotionally, but as we will see shortly, they can also suffer intellectually.

So bettas are a sad story.

My betta

Here is the specific sad story of my betta, the fish that left such an impression on me.

I had taken to lying on my bed and watching my betta for an hour or longer. The betta was the sole occupant of the tank in my bedroom. I had filled the tank with plants and copious natural light, so the effect was calming and serene. At times I almost envied the betta for the nice home I'd made for him.

One day, after the betta had been in his new home for several weeks, I found him exploring. It was a most unusual exploration, and one that I will never forget.

For the first few weeks, the betta explored the way you would expect any reasonably intelligent fish to go about the task. For the first few days he swam around to every nook and cranny of the tank, to make sure he had the lay of the land. Then for a few more days he experimented with staying put in different locations to see how he liked their feel.

Just like people, most fish will soon find a spot or a path they like best, and they'll stay in that spot or on that path for the rest of their lives.

But my betta was different. After his initial explorations he became restless. I'm no Fish Whisperer, but I could tell that he was restless. You would have thought so too. The betta started spending most of his time looking out of the tank, examining my bedroom. And he was clearly looking at specific things in the bedroom, not just "out there" in general. He would periodically swim around looking mildly agitated. He was acting like he wanted out.

I did everything I could to placate him. I experimented with different fish foods. I changed the water weekly and monitored it carefully to keep its temperature and pH within acceptable ranges. I added more lights. I added more plants. I rearranged the plants. In desperation, I even added a little castle.

Every time I tried something new, it would pique his interest for a little while. But in time, and faster each time than before, he would revert to his state of restlessness.

I'd never seen quite this behavior in other fish, so already he was demonstrating what seemed to be above average intelligence.

And then one day I found him engaged in an exploration that was altogether new.

He wasn't exploring the tank. He'd already investigated its topology for weeks. This time, he was exploring the nature of the tank. That's what caught my attention, and not just for that day, but for the rest of my life.

There was a twenty-inch vine in the tank that extended from the lower left back corner to the upper right front corner, along the diagonal of the main volume of the tank. The vine belonged to one of the many plants I'd put in there in the hopes of making it feel more like the Mekong river basin and less like a plexiglass tank in Seattle, Washington.

The betta had his nose on the vine. He was floating just above it, twitching his fins slightly to stay in place, and he was keeping his eyes as close to the vine as possible while keeping it in focus: about half a centimeter to a centimeter. And he was traversing the vine.

With the tiniest of motions, he was propelling himself along the vine towards the lower back right corner, keeping it under close scrutiny at all times. This excursion, from the halfway point to the end of the vine, took him perhaps three minutes. He was taking his time.

When he got to the end of the vine, he remained rooted in place while he inspected the 3-inch-radius spherical volume at the end of the vine, which was truncated in three dimensions by the walls and floor of the tank. He spent about three or four minutes doing this inspection, evidently making sure the vine really did terminate in the corner, and did not escape the tank.

After he had thoroughly scrutinized everything in the betta-sized vicinity of the vine's end, he turned back to the vine, nose pressed close, and began working his way along the vine in the other direction.

At this point I sat down to watch, because if he was doing what I thought he might be doing, then... I didn't know what to think. I wanted to see it for myself.

Over the next seven to ten minutes, he crept along the vine, never losing sight of it nor getting further than a centimeter from it, until he reached the upper-right front corner of the tank. He then proceeded to repeat his inspection of the volume at vine's end, ensuring himself that the vine terminated in the tank rather than protruding beyond the wall.

But what if he had missed something?

Sure enough, he turned and looked down the length of the vine for a time. And then he put his nose back on the vine and began again his long descent to the other end.

He did this for five days.

By the second day my amazement had turned to concern, and by the third day I felt utterly helpless. Here was an intelligent prisoner, my captive, exploring the mechanics of his prison with a thoroughness that only the imprisoned can afford, looking for an escape with deathly tenacity.

But while purchasing my betta had been easy enough, returning him to his real home would be unthinkably difficult, and probably unsuccessful even if I'd tried. Returning him to the fish store seemed like a dead end; he could easily wind up worse off than he was now.

So I concluded that there was nothing I could do. As he inspected the vine, I bit my nails, and timed passed in silence.

After the fifth day he gave up. And then he did something that I still don't understand, even though I've heard about this kind of thing before, and even though I personally saw him do it: he died of unhappiness.

It only took him a few days. He refused his food, he stopped moving, and to all external appearances he had become ill. But I knew better.

The lesson

Whenever I find myself struggling against the tide of massive system complexity, I think of my betta. He had a big heart, a small brain, and a small range of sensory input. I watched him use them all as methodically as any programmer to reason his way through to a soundness proof of the inescapability of his prison.

We like to think of ourselves as being pretty smart. Admit it. We do. But in the grand scheme of things we're intellectually little better off than that fish. We can easily find problems so complex that reasoning about them can take days or weeks of microscopic scrutiny, like my fish swimming along his vine.

And we can just as easily envision problems thousands or millions of times more complex: problems beyond the reasoning abilities of any person, any group of people, or even our entire species.

This has ramifications for the way we think about things today.

I believe I will have more to say about this soon. Right now I need to go mourn my fish, whose soul shone as brightly as that of anyone I've known.

The Universal Design Pattern

2008-10-20T03:06:00.000-07:00

This idea that there is generality in the specific is of far-reaching importance.
— Douglas Hofstadter, Gödel, Escher, Bach

Note: Today's entry is a technical article: it isn't funny. At least not intentionally.

Update, Oct 20th 2008: I've added an Updates section, where I'll try to track significant responses, at least for a week or so. There are three entries so far.

Introduction
Three Great Schools of Software Modeling
Brains and Thoughts
Who uses the Properties Pattern?
Properties Pattern high-level overview
Representations
- Keys
  - Quoting
  - Missing keys
- Data structures
Inheritance
Performance
Transient properties
- The deletion problem (remix)
Persistence
- Query strategies
- Backfills
Type systems
Toolkits
Problems
Further reading
New Updates
Final thoughts

Introduction

Today I thought I'd talk about a neat design pattern that doesn't seem to get much love: the Properties Pattern. In its fullest form it's also sometimes called the Prototype Pattern.

People use this pattern all over the place, and I'll give you a nice set of real-life examples in a little bit. It's a design pattern that's useful in every programming language, and as we'll see shortly, it's also pretty darn useful as a general-purpose persistence strategy.

But even though this pattern is near-universal, people don't talk about it very often. I think this is because while it's remarkably flexible and adaptable, the Properties Pattern has a reputation for not being "real" design or "real" modeling. In fact it's often viewed as a something of a shameful cheat, particularly by overly-zealous proponents of object-oriented design (in language domains) or relational design (in database domains.) These well-meaning folks tend to write off the Properties Pattern as "just name/value pairs" – a quick hack that will just get you into trouble down the road.

I hope to offer a different and richer perspective here. With luck, this article might even help begin the process making the Properties Pattern somewhat fashionable again. Time will tell.

Three Great Schools of Software Modeling

Before I tell you anything else about the Properties Pattern, let's review some of the most popular techniques we programmers have for modeling problems.

I should point out that none of these techniques is tied to "static typing" or "dynamic typing" per se. Each of these modeling techniques can be used with or without static checking. The modeling problem is orthogonal to static typing, so regardless of your feelings about static checking, you should recognize the intrinsic value in each of these techniques.

Class Modeling

You know all about this one. Class-based OO design is the 800-pound gorilla of domain modeling these days. Its appeal is that it's a natural match for the way we already model things in everyday life. It can take a little practice at first, but for most people class modeling quickly becomes second nature.

Although the industry loves OO design, it's not especially well liked as an academic topic. This is because OO design has no real mathematical foundation to support it — at least, not until someone comes along and creates a formal model for side effects. The concepts of OOP stem not from mathematics but from fuzzy intuition.

This in some sense explains its popularity, and it also explains why OOP has so many subtly different flavors in practice: whether (and how) to support multiple inheritance, static members, method overloading vs. rich signatures, and so on. Industry folks can never quite agree on what OOP is, but we love it all the same.

Relational Modeling

Relational database modeling is a bit harder and takes more practice, because its strength stems from its mathematical foundation. Relational modeling can be intuitive, depending on the problem domain, but most people would agree that it is not necessarily so: it takes some real skill to learn how to model arbitrary problems as relational schemas.

Object modeling and relational modeling produce very different designs, each with its strengths and weaknesses, and one of the trickiest problems we face in our industry has always been the object-relational mapping (ORM) problem. It's a big deal. Some people may have let you to believe that it's simple, or that it's automatically handled by frameworks such as Rails or Hibernate. Those who know better know just how hard ORM is in real-world production schemas and systems.

XML Modeling

XML provides yet another technique for modeling problems. Usually XML is used to model data, but it can also be used to model code. For instance, XML-based frameworks such as Apache Ant and XSLT offer computational facilities: loops or recursion, conditional expressions and setting variables.

In many domains, programmers will decide on an XML representation before they've thought much about the class model, because for those domains XML actually offers the most convenient way of thinking about the problem.

The kinds of data that work well with XML modeling tend to be poorly suited for relational modeling, and vice-versa, with the practical result that XML/relational mapping is almost as infamously thorny as O/R mapping.

And as for XML/OO mapping, most of us tend to treat it as a more or less solved problem. However, in practice there are several competing ways of doing XML/OO mapping. The W3C DOM and SAX enjoy the broadest use, but they are both sufficiently cumbersome that alternatives such as JDom and REXML (among others) have gained significant followings.

I mention this not to start a fight, but only to illustrate that XML is a third modeling technique in its own right. It has both natural resonances and surfaces of friction with both relational design and OO design, as one might expect.

Other schools

I'm not claiming that these three modeling schools are the only schools out there – far from it! Two other obvious candidates are Functional modeling (in the sense of Functional Programming, with roots in the lambda calculus) and Prolog-style logical modeling. Both are mature problem-modeling strategies, each with its pros and cons, and each having varying degrees of overlap with other strategies. And there are still other schools, perhaps dozens of them.

The important takeaway is that none of these modeling schools is "better" than its peers. Each one can model essentially any problem.

There are tradeoffs involved with each school, by definition — otherwise all but one would have disappeared by now.

Finding the sweet spot

Sometimes it makes sense to use multiple modeling techniques in the same problem space. You might do a mixed XML/relational data design, or a class-based OO design with Functional aspects, or embed a rules engine in a larger system.

Choosing the right technique comes down to convenience. For any given real-world problem, one or two modeling schools are likely to be the most convenient approaches. Exactly which one or two depends entirely on the particulars of the problem.

By convenient, I mean something different from what you might be thinking. To me, a convenient design is one that is convenient for the users of the design. And it should also be convenient to express, in the sense of minimalism: all else being equal, a smaller design beats a big one. One way of looking at this is that the design should be convenient for itself!

Unfortunately, most programmers (myself included) tend to use exactly the wrong definition of convenience: they choose a modeling technique that is convenient for themselves. If they only have experience in one or two schools, guess which techniques they'll jump to for every problem they face?

This problem rears its head throughout computing. There's always a "best" tool for any job, but if programmers don't know how to use it, they'll choose an inferior tool because they think their schedule doesn't permit a learning curve. In the long run they're hurting their schedules, but it's hard to see that when you're down in the trenches.

Modeling schools are just like programming languages, web frameworks, editing environments and many other tools: you won't know how to pick the right one unless you have a reasonably good understanding of all of them, and preferably some practice with each.

The important thing to remember is that all modeling schools are "first class" in the sense of being able to represent any problem, and no modeling school is ideal for every situation. Just because you are most comfortable solving a problem using a particular strategy does not mean that it is the ideal solution to the problem. The best programmers aim to master all available techniques, giving them a better chance at making the right choices.

Property Modeling

With this context in mind, I claim that the Properties Pattern is yet another kind of domain modeling, with its own unique strengths and tradeoffs, distinct from all the other modeling schools I've mentioned. It is their first-class peer, inasmuch as it is capable of modeling the same broad set of problem domains.

After we've finished talking about the Properties Pattern in exhausting detail, I think I'll have convinced you of the pattern's status as a major school of modeling. Hopefully you'll also start to have a feel for the kinds of problems it's well-suited to solve – sometimes more so than other schools, even your current favorite.

But before we dive into technical details, let's take a brief peek at a fascinating comparison of Property-based modeling to class-based OO design. It's a non-technical argument that I think has some real force behind it.

Brains and Thoughts

Douglas Hofstadter has spent a lifetime thinking about the way we think. He's written about it perhaps more than anyone else in the past century. Even if someone out there has beaten him in sheer quantity of words on the subject, nobody has come close to rivaling his style or his impact on programmers everywhere.

All of his books are wonderfully imaginative and are loads of fun to read, but if you're a programmer and you haven't yet read Gödel, Escher, Bach: An Eternal Golden Braid (usually known as "GEB"), then I envy you: you're in for a real treat. Get yourself a copy and settle in for one of the most interesting, maddening, awe-inspiring and just plain fun books ever written. The Pulitzer Prize it won doesn't nearly do it justice. It's one of the greatest and most unique works of imagination of all time.

Hofstadter made a compelling argument in GEB (thirty years ago!) that property-based modeling is fundamental to the way our brains work. In Chapter XI ("Brains and Thoughts"), there are three little sections titled Classes and Instances, The Prototype Principle, and The Splitting-off of Instances from Classes that together form the conceptual underpinnings of the Properties Pattern. In these little discussions Hofstadter explains how the Prototype Principle relates to classic class-based modeling.

I wish I could reproduce his discussion in full here — it's only three pages — but I'll have to just encourage you to go read it instead. His thesis is this:

The most specific event can serve as a general example of a class of events.

Hofstadter offers several supporting examples for this thesis, but I'll paraphrase one of my all-time favorites. It goes more or less as follows.

Imagine you're listening to announcers commenting on an NFL (American football) game. They're talking about a new rookie player that you don't know anything about. At this point, the rookie – let's say his name is L.T. – is just an instance of the class "football player" with no differentiation.

The announcers mention that L.T. is a running back: a bit like Emmitt Smith in that he has great speed and balance, and he's great at finding holes in the defense.

At this point, L.T. is basically an "instance" of (or a clone of) Emmitt Smith: he just inherited all of Emmitt's properties, at least the ones that you're familiar with.

Then the announcers add that L.T. is also great at catching the ball, so he's sometimes used as a wide receiver. Oh, and he wears a visor. And he runs like Walter Payton. And so on.

As the announcers add distinguishing attributes, L.T. the Rookie gradually takes shape as a particular entity that relies less and less on the parent class of "football player". He's become a very, very specific football player.

But here's the rub: even though he's a specific instance, you can now use him as a class! If Joe the Rookie comes along next season, the announcers might say: "Joe's a lot like L.T.", and just like that, Joe has inherited all of L.T.'s properties, each of which can be overridden to turn Joe into his own specific, unique instance of a football player.

This is called prototype-based modeling: Emmitt Smith was a prototype for L.T., and L.T. became a prototype for Joe, who in turn can serve as the prototype for someone else. Hofstadter says of The Prototype Principle:

"This idea that there is generality in the specific is of far-reaching importance."

Again, it's a three-page discussion that I've just skimmed here. You should go read it for yourself. Heck, you should read the whole book: it's one of the greatest books ever written, period, and every programmer ought to be familiar with it.

Hopefully my little recap of Hofstadter's argument has convinced you that my calling it the "Universal Design Pattern" might just possibly be more than a marketing trick to get you to read my blog, and that the rest of this article is worth a look.

The Properties Pattern is unfortunately big enough to deserve a whole book. Calling an entire school of modeling a "design pattern" is actually selling it short by a large margin.

Hence, if this article seems excruciatingly long, it's because I've tried to cram a whole book into a blog, as I often do. But look on the bright side! I've saved you a bunch of time this way. This will go much faster than reading a whole book.

Even so, don't feel bad if it takes a few sittings to get through it all. It's still a lot of information. I considered splitting it into 3 articles, but instead I just cut about half of the material out. (Jeff and Joel: seriously. I cut 50%.)

Who uses the Properties Pattern?

I assume I've already convinced you that this pattern is worth learning about, or you'd have left by now. I'm showing you these use cases not to draw you in, but to show you some of the very different ways the pattern can be used.

We could probably find hundreds of examples, but I'll focus on just a handful of real-world uses that I hope will illustrate just how widespread and flexible it is.

Before we start, there are two things to keep in mind. The first is that people have different names for this pattern, because even though it's quite commonplace, there hasn't been much literature on it. One paper calls it Do-It-Yourself Reflection. Another article calls it Adaptive Object Modeling. See Further reading for the few links I could dig up.

Whatever name you use, once you know how to look for it, you'll start seeing this pattern everywhere, wearing various disguises. Now that we have a common name for it, it should be easier to spot in the wild.

The second thing to keep in mind is that the Properties Pattern scales up: you can choose how much to use it in your system. It can be anything from simple property lists attached to a few of your classes to make them user-annotatable, up through a full-fledged prototype-based framework that serves as the foundation for modeling everything in your system.

So our examples will range from small ones to very big ones.

Eclipse

One nice small-scale example of the pattern is the Eclipse Java Development Tools (JDT): a set of classes that model the Java programming language itself, including the abstract syntax tree, the symbol graph and other metadata. This is used by the Eclipse backend to do all the neat magic it does with your Java code, by treating Java source code as a set of data structures.

You can view the javadoc for this class hierarchy at help.eclipse.org. Click on JDT Plug-in Developer Guide, then Programmer's Guide, then JDT Core, then org.eclipse.jdt.core.dom. This package defines strongly-typed classes and interfaces that Eclipse uses for modeling the Java programming language itself.

If you click through any class inheriting from ASTNode you'll see that it has a property list. ASTNode's javadoc comment says:

"Each AST node is capable of carrying an open-ended collection of client-defined properties. Newly created nodes have none. getProperty and setProperty are used to access these properties."

I like this example for several reasons. First, it's a very simple use of the Properties pattern. It doesn't muck around with prototypes, serialization, metaprogramming or many of the other things I'll talk about in a little bit. So it's a good introduction.

Second, it's placed smack in the middle of a very, very strongly-typed system, showing that the Properties pattern and conventional statically-typed classed-based modeling are by no means mutually exclusive, and can complement one another nicely.

And third, their property system itself is fairly strongly typed: they define a set of support classes such as StructuralPropertyDescriptor, SimplePropertyDescriptor, and ChildListPropertyDescriptor to help place some constraints on client property values. I'm not a huge fan of this approach myself, since I feel it makes their API fairly heavyweight. But it's a perfectly valid stylistic choice, and it's useful for you to know that you can implement the pattern this way if you so choose.

JavaScript

At the other end of the "how far to go with it" spectrum we have the JavaScript programming language, which places the Prototype Principle and Properties Pattern at the very core of the language.

People love to lump dynamic languages together, and they'll often write off JavaScript as some sort of inferior version of Perl, Python or Ruby. I was guilty of this myself for over a decade.

But JavaScript is substantively different from most other dynamic languages (even Lisp), because it has made the Properties Pattern its central modeling mechanism. It borrowed this heritage largely from a language called Self, and some other modern languages (notably Io and one other language that I'll talk about below) have also chosen prototypes and properties over traditional classes.

In JavaScript, every user-interactible object in the system inherits from Object, which has a built-in property list. Prototype inheritance (think back to our example of the Emmitt Smith instance having been the prototype for the L.T. instance) is a first-class language mechanism, and JavaScript offers several kinds of syntactic support for accessing properties and declaring property lists (as "object literals").

JavaScript is often accurately described as the world's most misunderstood programming language. Armed with our newfound knowledge, we can start to see JavaScript in a new light. To use JavaScript effectively, you need to gain experience with a whole new School of Modeling. If you simply try to use JavaScript as a substitute for (say) Java or Python, you'll encounter tremendous friction.

Since most of us have precious little actual experience with property-based modeling, this is exactly what happens, and it's no wonder JavaScript gets a bad rap.

Pushing it even further

In addition to how centrally you want to use the Properties pattern in your system, you can also decide how recursive to make it: do your properties have explicit meta-properties? Do you have metaprogramming hooks? How much built-in reflection do you offer?

JavaScript offers a few metaprogramming hooks. One such hook is the recently-introduced __noSuchMethod__, which lets you intercept a failed attempt to invoke a nonexistent function-valued property on an object.

Unfortunately JavaScript does not offer as many hooks as I'd like. For instance, there is no corresponding __noSuchField__ hook, which limits the overall flexibility somewhat. And there are no standard mechanisms for property-change event notification, nor any reasonable way to provide such a mechanism. So JavaScript gets it mostly right, but it stops short, possibly for performance reasons, of offering a fully-extensible metaprogramming system such as those offered by SmallTalk and to some extent, Ruby.

The pattern takes shape...

Before we move on to other uses of the Property Pattern, let's put JavaScript (and its central use of the pattern) into perspective, by comparing it to another successful language.

First: JavaScript is not my favorite language. I've done a lot of JavaScript programming over the past 2 years or so, both client-side and server-side, so I'm as familiar with it as I am with any other language I've used.

JavaScript in its current incarnation is not the best tool for many tasks. For instance, it's not great for building APIs, and it's not great for Unix scripting the way Perl and Ruby are. It has no library or package system, no namespaces, and is missing many other modern conveniences. If you're looking for a general-purpose language, JavaScript leaves you wanting.

But JavaScript is the best tool for many other tasks. As just one example, JavaScript is an outstanding language for writing unit tests — both for itself, and also for testing code in other languages. Being able to use the Properties Pattern to treat every object (and class) as a bag of properties makes the creation of mock objects a dream come true. The syntactic support for object literals makes it even better. You don't need any of the silly frameworks you see coming from Java, C++ or even Python.

And JavaScript is one of the two best scripting languages on the planet, in the most correct sense of the term "scripting language": namely, languages that were designed specifically to be embedded in larger host systems and then used to manipulate or "script" objects in the host system. This is what JavaScript was designed to do. It's reasonably small with some optional extensions, it has a reasonably tight informal specification, and it has a carefully crafted interface for surfacing host-system objects transparently in JavaScript.

In contrast, Perl, Python and Ruby are huge sprawls, all trying (like C++ and Java) to be the best language for every task. The only other mainstream language out there that competes with JavaScript for scripting arbitrary host systems is Lua, famous for being the scripting language of choice for the game industry.

And wouldn't you know it, Lua is also a language that uses the Properties Pattern as its central design. Its central Table structure is remarkably similar to JavaScript's built-in Object, and Lua also uses prototypes rather than classes.

So the world's two most successful scripting languages are prototype-based systems. Is this just a cosmic coincidence? Or is it possible that a suitably designed class-based language could have been just as successful?

It's hard to say. I've used Jython as an embedded scripting language for a long time, and it's worked pretty well. But I've personally come to believe that the Properties Pattern is actually better suited for extensibility than class-based modeling, and that prototype-based languages make better extension languages than class-based languages. That's effectively what's happening with embedded scripting: the end-users are growing and extending the host system.

In fact I was convinced of it before I even knew JavaScript. Let's take a look at another interesting "Who uses it?" example: Wyvern.

Wyvern

My multiplayer game Wyvern takes the Properties Pattern quite far as well, although in some different directions than what we've discussed so far. I designed Wyvern long before I'd heard of Self or Lua, and before I'd learned anything about JavaScript. In retrospect it's amazing how similar my design was to theirs.

Wyvern is implemented in Java, but the root GameObject class has a property list, much like JavaScript's Object base class. Wyvern has prototype inheritance, but since I'd never heard of prototypes before, I called them archetypes. In Wyvern, any game object can be the archetype for any other game object, and property lookup and inheritance work more or less identically to the way they work in JavaScript.

I arrived at this design after scratching my head for months (in late 1996) over how to build the ultimate extensible game. I wanted all the game content to be created by players, and I came up with dozens upon dozens of detailed use cases, in all of which I wanted players to be able to extend the game functionality in surprising new ways. In the end I arrived at a set of interleaved design patterns, including a rich command system, a rich hooks/advice system, and several other subsystems I'd love to document someday.

But the core data model was the Properties Pattern.

In some ways, Wyvern's implementation is more full-featured than JavaScript's. Wyvern offers more metaprogramming facilities, such as vetoable property change notifications, which gives in-game objects tremendous flexibility in responding to their environment. Wyvern also supports both transient and persistent properties, a scheme I'll discuss below.

On other ways, Wyvern just made different decisions. One big one is that Wyvern's property values are statically typed. The property names are always strings, just like in JavaScript, but the values can be various leaf types (ints, longs, booleans, strings, etc.), or functions (a trick that wasn't easy in Java), or even archetypes.

But despite the differences, Wyvern's core property-list infrastructure is a lot like that of JavaScript, Self and Lua. And it's been a design I've been fundamentally happy with for over ten years. It's met or exceeded all my original expectations for enabling end-user extensibility, particularly in its ability to let people extend the in-game behavior on the fly, without needing to reboot. This has proven extraordinarily powerful (and popular with the players.)

Where Wyvern clearly got the pattern wrong was in its lack of support for syntax. As soon as I decided to use the Properties pattern centrally in my game, I should have decided to use a programming language better suited for implementing the pattern: ideally, one that supports it from the ground up.

I eventually wound up using Python (actually, Jython) for a ton of my code, and it was far more succinct and flexible than anything I wrote in Java. But I was foolishly worried about performance, and as a result I wound up writing at least half the high-level game logic in Java and piling on hundreds of thousands of lines of getProperty and setProperty code. And now the system is hard to optimize; it would have been much easier if I'd had a cleaner separation of game-engine infrastructure from "scripty" game code.

Even if I'd done the whole game in Python, I'd still have had to implement a prototype inheritance framework to enable any object to be able to serve as the prototype for any other object.

I realize I haven't really explained why prototype inheritance works so well, except for my brief mention of mock objects for unit testing. But to keep this article tractable, I had to delete several pages of detailed examples, such as "Chieftain Monsters" that could be programmatically constructed by adding a few new properties to any existing monster

When I told you this pattern was big enough for a book, I meant a big book. Without the examples handy, all I can do is say that using JavaScript/Rhino (or Lua, once it became available on the JVM) might have made my life easier. Or heck, writing my own language might have been the best choice for a system that large and ambitious.

In any case, live and learn. It's a lot of code, but Wyvern is still a properties-based, prototype-based system, and it has amazing open-ended flexibility as a result.

We've been through the two big examples now (Wyvern and JavaScript). I'll close this "Who Uses It" section with just a few more key examples.

Lisp

Lisp features a small-scale example of the Properties Pattern: it has property lists for symbols. Symbols are first-class entities in Lisp. They're effectively the names in your current namespace, like Java's fully-qualified class names.

If Java classes all had property lists, it would still be a small-scale instantiation of the Properties pattern, but it would open up an awful lot of new design possibilities to Java programmers. Similarly, Lisp stops short of making everything have a property list, but to the extent it offers property lists they're exceptionally useful design tools.

Emacs Lisp actually makes heavy use of the Properties Pattern, inasmuch as essentially every one of its thousands of configuration settings is a property in the global namespace. It supports the notion of transient vs. persistent properties, and it offers a limited form of property inheritance via its buffer-local variables.

Unfortunately Emacs doesn't support any notion of prototypes, and in fact it doesn't have any object-orientation facilities at all. Sometimes I want to model things in Emacs using objects with flexible property lists, and at such times I find myself wishing I were using JavaScript. But even without prototypes, Emacs gains significant extensibility from its use of properties for data representation.

Keep in mind that there are, of course, big tradeoffs to make when you're deciding how much to use the Properties pattern; I'll discuss them in a bit. I'm not criticizing any of the systems or languages here for the choices they've made; all of them have been improved by their use of this pattern, regardless of how far they decided to take it.

XML revisited

Earlier I described XML as a first-class modeling school. Now that we have more context, it's possible to view XML as being an instantiation of the Properties Pattern, inasmuch as it uses the pattern as part of its fundamental structure.

XML's view of the pattern is two-dimensional: properties can take the form of either attributes or elements, each kind having different syntactic and semantic restrictions. Many folks have criticized XML for the unnecessary complexity of this redundant pair of property subsystems, but in the end it doesn't really matter much, since two ways to model properties is still better than zero ways.

So far we've seen various policies around static checking: Eclipse (strong/mandatory), JavaScript/Lua (very little), and Wyvern (moderate).

XML offers what I think is the ideal policy, which is that it lets you decide for yourself. During your initial domain modeling (the "prototyping" phase — a term now loaded with Even More Delicious Meaning), you can go with very weak typing, opting for nothing more than simple well-formedness checks. And for many problems, this is as much static checking as you'll ever need, so it's nice that you have this option.

As some of your models become more complex, you can choose to use a DTD for extra validation. And if you need a really heavy-duty constraint system, you can migrate up to a full-fledged XML Schema or Relax NG schema, depending on your needs.

XML has proven to be a very popular modeling tool for Java programmers in particular — more so than for the dynamic language communities. This is because Java offers essentially zero support for the Properties Pattern. When Java programmers need access to the pattern, the easiest approach is currently to use XML.

The Java/XML combination has proven reasonably powerful, despite the lack of syntactic integration and numerous other impedance mismatches. Using XML is still often preferable to modeling things with Java classes. Even Eclipse's AST property lists might have been better modeled using XML and a DOM: it would have been less work, and the interface would have been more familiar. And as for Apache Ant: JSON-style JavaScript objects for build files would have been exactly what they needed, but by the time they'd realized they needed a plug-in system, the damage was done.

As Mozilla Rhino becomes better documented, and as more Java programmers begin to appreciate the usefulness of JSON as a lightweight alternative to XML, JavaScript may begin to close the gap. Rhino provides Java with the Properties Pattern much more seamlessly than any XML solution. I've already mentioned that it's superior (even to XML) for unit tests and representing mock test data.

But it goes deeper than unit testing. Every sufficiently large Java program, anything beyond medium-sized, needs a scripting engine, whether the authors realize it or not. Programs often have to grow to the size of browsers or spreadsheets or word processors before the authors finally realize they need to offer scripting facilities, but in practice, even small programs can immediately benefit from scripting. And XML doesn't fit the bill. It's yet another example of programmers choosing a School of Modeling because they know it, rather than learning how to use the right tool for the job.

So it goes.

Bigtable

Last example: Google's Bigtable, which provides a massively scalable, high performance data store for many Google applications (some of which are described in the paper – click the link if you're curious.) This particular instantiation of the Properties Pattern is a multidimensional table structure, where the keys are simple strings, and the leaf values are opaque blobs.

Hardcore relational data modelers will sometimes claim that large systems will completely degenerate in the absence of strong schema constraints, and that such systems will also fail to perform adequately. Bigtable provides a nice counterexample to these claims.

That said, explicit schemas are useful for many domains, and I'll talk more about how they relate to the Properties Pattern in a bit.

This would probably be a good time to mention Amazon's Simple Storage Service, but I don't know anything about it. I've heard they use name-value pairs.

In any case, I hope these examples (Eclipse AST classes, JavaScript, Wyvern game objects, Lisp symbols, XML and HTML, and Bigtable) have convinced you that the Properties pattern is ubiquitous, powerful, and multifaceted, and that it should be part of any programmer or designer's lineup.

Let's look in more depth at how it's implemented, its trade-offs, and other aspects of this flexible design strategy.

Properties Pattern high-level overview

At a high level, every implementation of the Properties Pattern has the same core API. It's the core API for any collection that maps names to values:

get(name)
put(name, value)
has(name)
remove(name)

There are typically also ways to iterate over the properties, optionally with a filter of some sort.

So the simplest implementation of the Properties Pattern is a Map of some sort. The objects in your system are Maps, and their elements are Properties.

The next step in expressive power is to reserve a special property name to represent the (optional) parent link. You can call it "parent", or "class", or "prototype", or "mommy", or anything you like. If present, it points to another Map.

Now that you have a parent link, you can enhance the semantics of get, put, has and remove to follow the parent pointer if the specified property isn't in the object's list. This is largely straightforward, with a few catches that we'll discuss below. But you should be able to envision how you'd do it without too much thought.

At this point you have a full-fledged Prototype Pattern implementation. All it took was a parent link!

From here the pattern can expand in many directions, and we'll cover a few of the interesting ones in the remainder of this article.

Representations

There are two main low-level implementation considerations: how to represent property keys, and what data structure to use for storing the key/value pairs.

Keys

The Properties pattern almost always uses String keys. It's possible to use arbitrary objects, but the pattern becomes more useful with string keys because it trivially enables prototype inheritance. (It's tricky to "inherit" a property whose key is some opaque blob - we usually think of inheritance as including a set of named fields from the parent.)

JavaScript permits you to use arbitrary objects as keys, but what's really happening under the covers is that they're being cast to strings, and they lose their unique identity. This means JavaScript Object property lists cannot be used as be a general-purpose hashtable with arbitrary unique objects for keys.

Some systems permit both strings and numbers as keys. If your keys are positive integers, then your Map starts looking an awful lot like an Array. If you think about it, Arrays and Maps share the same underlying formalism (a surjection, not to put too fine a point on it), and in some languages, notably PHP, there isn't a user-visible difference between them.

JavaScript permits numeric keys, and allows you to specify them as either strings or numbers. If your object is of type Array, you can access the numeric keys via array-indexing syntax.

Quoting

JavaScript syntax is especially nice (compared to Ruby and Python) because it allows you to use unquoted keys. For instance, you can say

var person = {
  name: "Bob",
  age: 20,
  favorite_days: ['thursday', 'sunday']
}

and the symbols name, age and favorite_days are NOT treated as identifiers and resolved via the symbol table. They're treated exactly as if you'd written:

var person = {
  "name": "Bob",
  "age": 20,
  "favorite_days": ['thursday', 'sunday']
}

You also have to decide whether to require quoting values. It can go either way. For instance, XML requires attribute values to be quoted, but HTML does not (assuming the value has no whitespace in it).

Missing keys

You will need to decide how to represent "property not present". In the simplest case, if the key isn't in the list, the property is not there (but see Inheritance further on).

If a property is frequently removed and re-added, it may make sense to leave the key in the list with a null value. In some systems, you may need null to be a valid property value, in which case you'd need to use some other distinguished (and reserved) value for this micro-optimization to work.

Data structures

The simplest property-list implementation is a linked list. You can either have the alternating elements be the keys and values (Lisp does this), or you can have each element be a struct containing pointers to the key and value.

The linked list implementation is appropriate when:

you're just using the pattern to allow user annotations on object instances
you don't expect many such annotations on any given instance
you're not incorporating inheritance, serialization or meta-properties into your use of the pattern

Logically a property list is an unordered set, not a sequential list, but when the set size is small enough a linked list can yield the best performance. The performance of a linked list is O(N), so for long property lists the performance can deteriorate rapidly.

The next most common implementation choice is a hashtable, which yields amortized constant-time find/insert/remove for a given list, albeit at the cost of more memory overhead and a higher fixed per-access cost (the cost of the hash function.)

In most systems, a hashtable imposes too much overhead when objects are expected to have only a handful of properties, up to perhaps two or three dozen. A common solution is to use a hybrid model, in which the property list begins life as a simple array or linked list, and when it crosses some predefined threshold (perhaps 40 to 50 items), the properties are moved into a hashtable.

Note that we'll often refer to property sets as "property lists" (or "plists" for short), because they're so often implemented as lists. But it's fairly unusual for the order to matter. In the rare cases when it matters, there are usually two possibilities: the names need to be kept in insertion order, or they need to be sorted.

If you need constant-time access and want to maintain the insertion order, you can't do better than a LinkedHashMap, a truly wonderful data structure. The only way it could possibly be more wonderful is if there were a concurrent version. But alas.

If you need to impose a sort order on property names, you'll want to use an ordered-map implementation, typically an ordered binary tree such as a splay tree or red/black tree. A splay tree can be a good choice because of the low fixed overhead for insertion, lookup and deletion, but with the tradeoff that its theoretical worst-case performance is that of a linked list. A splay tree can be especially useful when properties are not always accessed uniformly: if a small subset M of an object's N properties are accessed most often, the amortized performance becomes O(log M), making it a bit like an LRU cache.

Note that you can get a poor-man's splay tree (at least, the LRU trick of bubbling recent entries to the front of the list) using a linked list by simply moving any queried element to the front of the list, a constant-time operation. It's surprising that more implementations don't take this simple step: an essentially free speedup over the lifetime of most property lists.

Inheritance

With prototype inheritance, each property list can have a parent list. When you look for a property on an object, first you check the object's "local" property list, and then you look in its parent list, on up the chain.

As I described in the Overview, the simplest approach for implementing inheritance is to set aside a name for the property pointing to the parent property list: "prototype", "parent", "class" and "archetype" are all common choices.

It's unusual (but possible) to have a multiple-inheritance strategy in the Properties pattern. In this case the parent link is a list rather than a single value, and it's up to you to decide the rules for traversing the parent chains during lookups.

The algorithm for inherited property lookup is simple: look in my list, and if the property isn't there, look in my parent's list. If I have no parent, return null. This can be accomplished recursively with less code, and but it's usually wiser to do it iteratively, unless your language supports tail-recursion elimination. Property lookups can be the most expensive bottleneck in a Properties Pattern system, so thinking about their performance is (for once) almost never premature.

The deletion problem

If you delete a property from an object, you usually want subsequent checks for the property to return "not found". In non-inheritance versions of the pattern, to delete a property you simply remove its key and value from the data structure.

In the presence of inheritance the problem gets trickier, because a missing key does not mean "not found" – it means "look in my parent to see if I've inherited this property."

To make the problem clearer, assume you have a prototype list called Cat with a property named "friendly-to-dogs", whose value defaults to the boolean true. Let's say you have a specific cat instance named Morris, whose prototype is Cat:

var Cat = {
  friendly_to_dogs: true
}

var Morris = {
  prototype: Cat
}

Let's say Morris has a nasty run-in with a dog, and now he hates all dogs, so we want to make a runtime update to his friendly-to-dogs property. Our first idea might be to delete the property, since a missing key or null value are often interpreted as false in a boolean context. (This is true even in class-based languages like C++ or Java, in which a hasFooBar function will return true if the internal fooBar field is non-null.)

However, Morris does not have a copy of "friendly-to-dogs" in his local list: he inherits it from Cat. So if your deleteProperty method does nothing but delete the property from the local list, he will continue to inherit "friendly-to-dogs", which will irk him (and you) endlessly until you figure out where the bug is.

You can't delete "friendly-to-dogs" from the Cat property list, or all of your cats will suddenly become dog-haters, and you'll have outright war on your hands. (Note that in some settings this is exactly what you want to do, illustrating the inherent universal trade-off between flexibility and safety.)

The solution for Morris is to have a special "NOT_PRESENT" property value that deleteProperty sets when you delete a property that would otherwise be inherited. This object should be a flyweight value so that you can check it with a pointer comparison.

So to account for deletion of inherited properties, we have to modify our property-lookup algorithm to look in the local list for (a) a missing key, (b) a null value, or (c) the NOT_PRESENT tag. If any of these apply, the property is considered not present on the object. [Note: the semantics of null values are up to the system designer. You don't have to make null values mean "not there." Either way is fine.]

Read/write asymmetry

One logical consequence of prototype inheritance as we've defined it is that reads and writes work differently. In particular, if you read an inherited property, it gets the value from an ancestor in the prototype chain. But if you write an inherited property, it sets the value in the object's local list, not in the ancestor.

To illustrate, let's add a "night-vision" property to our Cat prototype. Its value is expected to be an integer representing how well the cat can see in the dark. Let's say that the default value is 5, but our hero Morris has been eating his carrots, so we want to set his "night-vision" property value to 7.

The setProperty code does not need to check the parent chain: it simply adds the key/value pair {"night-vision", 7} to Morris's local property list. If we set the property on Cat, then all cats would have Morris's super-vision, which isn't what we want.

This asymmetry is normal. Back in our L.T. / Emmitt Smith example, when we were adding properties to L.T., we didn't want to modify Emmitt! It's just how the pattern works: you override inherited values by adding local properties, even when the override is a deletion.

Read-only plists

Many implementations of the pattern offer "frozen" property lists. Sometimes (e.g. for debugging) it's useful to flag an entire property list as read-only. Ruby supports this via the "freeze" method on the built-in root Object class. In any sufficiently large, robust implementation of the Properties pattern, you should include the option to freeze your property lists.

If you offer a "freeze" function, you should think about whether you want to offer a "thaw" as well. The decision depends on whether you want to offer programmers additional protection, or you just want to lock them up and throw away the key.

My personal view is that Java programmers tend to overuse the "freeze" function when they start with Ruby. For that matter, they tend to overuse "final" in Java. I mentioned before the trade-off between flexibility and safety. When you use the Properties Pattern, you're consciously choosing flexibility over safety, and in many domains this is the right choice. In fact, safety can be viewed as a kind of optimization: something that should ideally be layered on, functioning behind the scenes rather than being interleaved with the user APIs and flexible data model.

A nice (and simple to implement) compromise on safety and flexibility is to offer a ReadOnly property attribute, as JavaScript does. There are certain properties (such as the parent pointer) that are less likely to need to change as the system evolves, so it's probably OK to lock them down early on. Doing this on a property-by-property basis is much less draconian. Even better, you should consider making the ReadOnly property attribute non-inheritable, so that subtypes can choose their own policies without compromising the integrity of the supertypes.

We're more or less done with inheritance: it's not very complicated. There are a few other inheritance-related design issues that I'll cover in upcoming sections.

Performance

Performance is one of the biggest trade-offs of using the Properties Pattern. Many engineers are so concerned with performance (and its attendant paradoxes and fallacies) that they refuse to consider using the Properties pattern, regardless of the situation.

As it happens, the pattern's performance can be improved and mitigated in several clever ways. I won't cover all of them here, but I'll touch on some of the classics and one or two new approaches.

Interning strings

Make sure your string keys are interned. Most languages provide some facility for interning strings, since it's such a huge performance win. Interning means replacing strings with a canonical copy of the string: a single, immutable shared instance. Then the lookup algorithm can use pointer equality rather than string contents comparison to check keys, so the fixed overhead is much lower.

The only downside of interning is that it doesn't help much when you're constructing a property name on the fly, since you still need to hash the string to intern it.

That's not much of a downside, so as a rule, you should always intern your keys. A large percentage of property names in any system are accessed as string literals from the source code (or are read from a configuration file and can be interned all at once when the file is read), and interning works in these common cases.

Corollary: don't use case-insensitive keys. It's performance suicide. Case-insensitive string comparison is really slow, especially in a Unicode environment.

Perfect hashing

If you know all the properties in a given plist at compile-time (or at runtime early on in the life of the process), then you might consider using a "perfect hash function generator" to create an ideal hash function just for that list. It's almost certainly more work than it's worth unless your profiler shows that the list is eating a significant percentage of your cycles. But such generators (e.g. gperf) do exist, and are tailor-made for this situation.

Perfect hashing doesn't conflict with the extensible-system nature of the Properties pattern. You may have a particular set of prototype objects (such as your built-in monsters, weapons, armor and so on) that are well-defined and that do not typically change during the course of a system session. Using a perfect hash function generator on them can speed up lookups, and then if any of them is modified at runtime, you just fall back to your normal hashing scheme for that property list.

Copy-on-read caching

If you have lots of memory, and your leaf objects are inheriting from prototype objects that are unlikely to change at runtime, you might try copy-on-read caching. In its simplest form, whenever you read a property from the parent prototype chain, you copy its value down to the object's local list.

The main downside to this approach is that if the prototype object from which you copied the property ever changes, your leaf objects will have the now-incorrect old value for the property.

Let's call copy-on-read caching "plundering" for this discussion, for brevity. If Morris caches his prototype Cat's copy of the "favorite-food" property (value: "9Lives"), then Morris is the "plunderer" and Cat is the plundered object.

The most common workaround to the stale-cache problem is to keep a separate data structure mapping plundered objects to their plunderers. It should use weak references so as not to impede garbage collection. (If you're writing this in C++, then may God have mercy on your soul.) Whenever a plundered object changes, you need to go through the plunderers and remove their cached copy of the property, assuming it hasn't since then changed from the original inherited value.

That's a lot of stuff to keep track of, so plundering is a strategy best used only in the direst of desperation. But if performance is your key issue, and nothing else works, then plundering may help.

Refactoring to fields

Another performance speedup technique is to take your N most commonly used properties and turn them into instance variables. Note that this technique is only available in languages that differentiate between instance variables and properties, so it would work in Java but not in JavaScript.

This optimization may sound amazingly attractive, especially during your first round of performance optimizations. Be warned: this approach is fraught with pitfalls, so (as with nearly all performance optimizations) you should only use it if your profiler proves that the benefits will outweigh the costs.

The first cost is API incompatibility: suddenly instead of accessing all properties uniformly through a single getProperty/setProperty interface, you now have specific fields that have their own getters and setters, which could potentially have massive impact in your system (since, after all, these are the most commonly-accessed properties). And unless you're using Emacs, your refactoring editor probably isn't smart enough to do rename-method constrained on argument value.

You can mitigate the API problem by continuing to go through the get/setProperty interface, and have them check the argument to see if it's one of the hardwired fields. This will result in an increasingly large switch-statement (or equivalent), so you're trading API code maintenance for API simplicity. It also slows down the field access considerably, which offsets the performance gain from using a field.

The next cost is system complexity: you have twice as many code paths through every part of your system that deals with the Properties pattern. Does inheritance still work the same? What about serialization? Transient properties? Metaprogramming? What about your user interface for accessing and modifying property lists? You face a huge code-bloat problem when you split property access into two classes: plist properties and instance variables.

The next cost is accidentally getting it wrong: how do you know what the most-accessed properties are? You may do some runtime profiling and see that it's one set, but over time the characteristics of your system might change in such a way that it's an entirely different set. Realistically you will have to instrument your system to keep track, on a regular basis, of which properties are accessed at a rate beyond your tolerance threshold, so you can convert them to fields as well. This isn't a one-off optimization.

But all these costs pale in comparison to the big one, which is extensibility: instance variables are set in stone. It will be a vast amount of work to try to give them parity with your plist properties: change-list notifications, the ability to override them or remove them, and so on. It's likely that you will wind up sacrificing at least some flexibility for these fields.

So use this optimization with extreme caution.

Refrigerator

The last performance optimization I'll mention is more about conserving memory than CPU. If you're worried about the overhead of a per-object property-list field, even if it's usually null, then you can implement property lists using a separate, external data structure.

I don't know what it's normally called, so I'm calling it the Refrigerator, since you're basically putting yellow stickies all over it. The idea is that you don't need to pay for the overhead of property lists when very few of the objects in your system will ever have one. Instead of using a field in each class with a property list, you maintain a global hashtable whose keys are object instances, and whose values are property lists.

To fetch the property list of an object, you go look to see if it's on the Refrigerator. The property list itself can follow any of the implementation schemes I discussed in Representations above.

I first heard this idea from Damien Conway in roughly 2001 at a talk he gave. He said he was considering using it for Perl 6, and I thought it was pretty clever. I don't remember what he called it, and I don't know if he wound up using it, but consider this idea to be his gift to you. Thanks, Damien!

REDACTED

Brendan Eich came up with astoundingly clever performance optimization for the Properties Pattern, which he told me about back in January. I was ready to publish this article, but I told him I'd hold off until he blogged about his optimization. Every once in a while he'd ping me and tell me "any day now."

Brendan, it's October, dammit!

Rolling your own

Needless to say, I've only scratched the surface on performance optimization of the Properties pattern. You can get arbitrarily fancy. The point I'm trying to get across is that you shouldn't despair when you discover your system is unacceptably slow after designing it to use the Properties pattern. If this happens, don't panic and throw out your flexibility – go optimize! The game of optimization can be fun and rewarding in its own right.

Just don't do it before you need it!

Transient properties

While implementing Wyvern, I discovered that making changes to a persistent property list is a wonderful recipe for creating catastrophes.

Let's say some player casts a Resist Magic spell, which boosts her "resist-magic" integer property value by, oh, 30 (thirty percent). Then, while the spell is active, the auto-saver kicks in (writing her enhanced "resist-magic" property value out to the data store along with the rest of her properties), and then the game crashes.

Voilà – the player now has permanent 30% magic resistance!

It doesn't have to be a game crash, either. Any random bug or exception condition (a database hiccup, a network glitch, cosmic rays) can induce permanence in what was intended to be a transient change to the plist. And when you're writing a game designed to be modified at runtime by dozens of programmers simultaneously, you learn quickly to expect random bugs and exception conditions.

The solution I came up with was transient properties. Each object has (logically speaking) two property lists: one for persistent properties and one for transients. The only difference is that transient properties aren't written out when serializing/saving the player (or monster, or what-have-you.)

Wyvern's property-list system has typed values. I haven't talked about Properties Pattern type systems yet, but in a nutshell my property values can be ints, longs, doubles, strings, booleans, archetypes (which is basically any other game object), or "bean" (JavaBean) properties.

My early experimentation yielded the interesting rule that non-numeric transient properties override the persistent value, but numeric properties combine with (add to) the persistent value.

A simple example should suffice. If you have a persistent boolean property "hates-trolls" (and who doesn't, really?), and you accidentally ingest a Potion of Troll Love, then the potion should set a transient value of {"hates-trolls", false} on your character. It overrides the persistent value. There's no combining going on; it just replaces the original.

However, for our "resist-magic" int property, if you put on a ring of 30% magic resistance, it should (by default) add to your current value, which may be a combination of innate resistance and resistances conferred from other magic items and spells.

This numbers-are-additive principle applied pretty uniformly across my entire code base and property corpus, so it's built into the lookup rules for Wyvern's property lists. getIntProperty("foo") must get both the transient and persistent (possibly inherited) values for "foo" and add them before returning the result.

I experimented with different approaches for representing transient properties. Originally I used a kind of hungarian-notation, prefixing transient property names with an @-character ("@foo") and keeping them in the same hashtable as the persistent properties. One advantage of "@" was that it was invalid character in XML attribute names, so it was originally impossible for me to accidentally serialize a transient property.

Eventually I migrated to keeping them in separate (lazily created) tables. This made it easier to deal with interning names (not having to prepend "@" all the time) and generally made bookkeeping easier. I don't remember all the trade-offs involved with the decision anymore (it was about 7 years ago), so you'll have to retread that road yourself if you decide to offer transient properties in your system.

The deletion problem (remix)

Transient properties introduce their own version of deletion problem. You can't just remove the property from the transient list, since the lookup algorithm will just look for it in the persistent list. And you don't want to remove it from the persistent list; that defeats the purpose of using transients.

The solution is similar to what I did for deleting inherited properties: you insert a placeholder into the transient list saying "NOT_PRESENT", and as long as that placeholder is in the list, it's as if the object doesn't have the property.

Note that this implies the existence of two similar API calls: removeTransientProperty for deleting a transient property from the transient list, and transientlyRemoveProperty for temporarily hiding a property from the persistent list.

Persistence

Persisting property lists is a huge topic; I'll just touch on the basics.

For starters, XML and JSON (and for that matter, s-expressions) are all perfectly valid choices for serialization format. You can probably imagine how this works, so I won't beat it to death.

Text-based formats have big wins in readability, maintainability, and bootstrapping (you don't need to create special tools to read and write them).

For performance – both for network overhead and disk storage – you might want to consider designing a compressed binary format. One easy way to test whether this will be a win for you is to take the intermediate approach of gzipping your data to see how well it compresses, and whether it produces a discernable blip in performance.

Wyvern initially used a filesystem trie-like structure for storing its data, but as the number of distinct objects grew to several hundred thousand, I had to switch to a database.

You can use an RDBMS, but you're in for a world of hurt if you try to map the Properties pattern onto a relational schema. It's a tricky problem, and probably isn't one that you want to solve yourself.

I wound up using an RDBMS and just shoving the XML-serialized property list into a text/clob column, and denormalizing the twenty or thirty fields I needed for queries into their own columns. This gets me by, but isn't a happy solution.

What you really want is a hierarchical data store optimized for loose tree structures: in a word, an XML database. At the time I was designing Wyvern's persistence strategy (1998-ish), XML databases were pure vaporware, and even after a few years they were still fairly exploratory and unstable.

Today things are different, and there are many interesting options for XML databases, ranging from 100% free (e.g. Berkeley DBs) through 100% expensive (e.g. Oracle XML).

You might also look into Object databases, but I've never heard of anyone coming through that experience with anything but battle scars to show for it.

Query strategies

Querying goes hand-in-hand with persistence: once you have a bunch of objects in a data store, you'll want to ask questions about them. Producing a High Score List is a good example: you want to compute a function of some set of properties across all the players in your database.

If you're just using the filesystem, you're stuck with grep or its moral equivalent on Windows, which is likely to be painfully slow. So don't do that.

If you're using an RDBMS, and you've serialized your property lists into a single row-per-object clob or text column, then you can use (My)SQL's LIKE and RLIKE operators, or their equivalents, to do free-text searches.

However, your property lists are likely to be hierarchical (e.g. player inventory is a bag, which has its own properties and collection of objects it's holding), and free-text search doesn't understand hierarchy. So this approach is really just a faster version of grep.

Querying is the biggest reason for using an XML database, since it gives you XPath and XQuery as expressive languages that work on XML data about as well (give or take) as SQL works on relational data.

Because you have the advantage of working in "these days" (2008+) as opposed to "those days" (1998), you now have the interesting option of using JavaScript/JSON and JQuery. I don't know much about it, but what little I do know seems promising.

One final approach, which may not scale very well unless you can find a way to parallelize it, is to simply load all the objects into an instance of your server, and use programmatic access to walk the objects and construct your query results manually. Although it requires some infrastructure to make it work (and to make it not crash your system, once you have enough objects), it has the major benefit of giving you a full programming language, which can be useful if you're doing a complex query and your XPath/XQuery skills aren't up to par.

Backfills

Data integrity, a.k.a. Safety, is one of the two biggest trade-offs (the other being performance) you make when you choose to use the Properties pattern. In the absence of a schema, your system is open to property-list corruption through bugs and user error (e.g. typos).

Interestingly, in big companies I've worked at that have strong schema constraints, they still always seem to run into data-integrity problems, so it's not clear how much the schema is really helping here. A schema can certainly help with navigability and performance, but no schema can completely avert data corruption problems.

As soon as you notice you've got bad data, you need to do what many people in the industry term a "backfill": you have to run through all the existing data and fix the problem. Sometimes this is as simple as running a SQL update on a single column. And sometimes it involves writing a complex program that painstakingly computes the inverse of whatever bogus operation created the bad data in the first place.

And sometimes backfills require just winging it, since the lost data may be irrecoverable and you need to use heuristics to minimize the damage. No matter how you store your data and how careful you are about replicating it and backing it up, this kind of thing can happen at pretty much any scale.

The Properties pattern doesn't really introduce anything new to the backfill landscape; all the usual options apply. You'll just need to be mindful that user error (especially mis-typed property names) can make backfills a bit more commonplace, so you should plan to spend a fair amount of time developing a convenient backfill infrastructure for your system.

I should mention, embarrassing as it is, one other option, which I call "lazy backfill", and I've used it extensively. Sometimes I'll notice a data-corruption issue that needs fixing but doesn't really justify a day of my time to fix all at once. So I have a small subsystem in place for player logins and map loads: I iterate through the property lists looking for properties that I've flagged (hardwired in the code) as "bad data", and I call helper backfill functions on the fly to fix just that property list.

This is obviously a hack, and it also imposes some minor performance overhead (probably not detectable via profiling) on logins and map loads, but I'll be honest: it's served me well, and I've fixed at least 20% of my data-corruption problems this way.

Type systems

I've already touched on this a little here and there. Eclipse's AST property lists use an interesting type system that provides a reasonable amount of metadata for each property, although (I think) it stops short of allowing properties to have their own property lists.

JavaScript properties have a small, fixed amount of metadata. Each property has a set of flags. The flags include ReadOnly (can't modify the value), Permanent (can modify the value but can't delete the key), DontEnum (key doesn't show up in iterators but can be read directly), and others depending on the implementation.

Wyvern has its own Java-like flavor of typed properties, largely because I implemented the system in Java long before the advent of auto-boxing, and I needed a convenient way of dealing with primitive types vs. object types. If I were to do it all over again, I probably wouldn't go that route. I would want some sort of scheme for metaproperties (aka "property attributes") — perhaps in a separate per-object metaproperty-list. But I'd simplify the interface and get rid of all the primitive-typed versions of all my has/get/set inherited/persistent/transient property calls.

I won't go into any further detail about type systems, except to say that (a) you can use them to any degree you desire; there's nothing intrinsic to the Properties Pattern that precludes them, and (b) Duck Typing becomes fairly crucial to systems that are designed fully around the Properties Pattern, so if your language has any structural-typing support it'll help.

Toolkits

Wyvern has a Map Editor that allows you to create and edit objects. Since all the game objects are property lists that use the prototype inheritance pattern, the conventional JavaBean approach doesn't work, since the JavaBeans API (which is more or less designed for this problem, except with instance fields) uses Java reflection, and my properties don't have individual getters and setters.

Wyvern wound up with something very similar to a JavaBeans property editor, except it knows how to read, write and display my GameObject property lists.

It wasn't a huge amount of work, but it's something you should keep in mind as you decide whether to use the Properties pattern in your system. If you need a GUI for object edits, you'll probably need to do some custom UI work.

Problems

I've talked about the main problems imposed by the Properties pattern: performance, data integrity, and navigability/queryability. They're all trade-offs; you're sacrificing in these areas in order to achieve big wins in flexibility and open-ended future extensibility for users you may never meet. (You also get huge wins in unit-testability and prototyping speed, but I assume these benefits are obvious enough that I don't need to dwell on them.)

One other problem is reversability: it's hard to back out of the Properties pattern once you commit to it. Once people start adding properties, especially if they're using programmatic construction of key names, you'll have trouble on your hands if you want to try to refactor the whole system to use instance fields and getters/setters. So before you use this pattern, you should put your system through a prototyping phase (ironically enough) to determine whether it will work out as it scales.

New Updates

Oct 20th 2008: The comments on the article are outstanding. People have pointed out lots of further reading, as well as well-known systems (Atom, RDF) that make extensive use of the pattern at their core. Thanks, folks! This is great stuff.

Oct 20th 2008: Martin Fowler sent me a link to a 1997 paper he wrote on the topic. It's in the Further Reading section. Well worth a read. He mentions a few important considerations that I left out:

The (static) dependencies problem — your compiler can't generally help you with finding dependencies on properties, or even tell you what property names are used in your system. He suggests using a registry of permissible property names as one solution. I strongly considered that approach for Wyvern, but wound up relying on a mix of dynamic tracing and static analysis to get me the dependency graph, and it's been "accurate enough" for my needs. In particular, synthesizing property names on the fly happens about as (in)frequently as Reflection happens in Java. So for the most part the dependencies issue is as tractable as it is in Java: "tractable enough".

Substituting actions Fowler suggests that it's difficult to replace an existing property access with an action. This is only true in the Java implementation (hence, true for my game). In languages like Python, Ruby and JavaScript 1.5 that support property-access syntax for getters and setters this is a non-issue.

Overall, Martin's take on the pattern is "avoid it when possible", which is sound (if conservative) advice. My take is that everyone's doing it anyway, so we should formalize it. I've used it as the central data model for my 500,000-line multiplayer game for 10 years, and I assert that the benefits vastly outweigh the problems. I also witnessed the pattern's use in Amazon's Customer Service Tools database for some 5 years, and again, the benefits vastly outweighed the downsides.

You just have to know what you're getting into before you dive in, which is sort of the point of my article.

Oct 20th 2008: Fellow Googler Joe Beda mentioned that IE4 originally supported arbitrary attributes on HTML elements, which dramatically extended the flexibility for web developers. Today, no browsers support it, though John Resig claims HTML 5 will fix this. In the meantime, developers use fake css classes and hidden elements; it's a mess. I actually deleted a pretty large rant about this problem from the article. But yeah. It's a problem. When you don't provide the Properties Pattern to people, they find horrible workarounds, which is much worse than anything that can go wrong if you simply support it directly. (Joe mentioned that it posed serious technical problems with the cache, though, so I wouldn't assume it's trivial to add the support back in to browsers today.)

Final thoughts

I haven't covered the whole landscape for this pattern. There are concurrency issues, access-control issues (e.g. in Wyvern, some properties, such as email address, can only be read by very high-level Wizards), documentation issues, and a host of other considerations.

Let me summarize what I think are the key takeaways.

First: this is a critically important pattern. I call it the "Universal" design pattern because it is (by far) the best known solution to the problem of designing open-ended systems, which in turn translates to long-lived systems.

You might not think you're building that kind of system. But if you want your system to grow, and have lots of users, and spread like wildfire, then you are building exactly that kind of system. You just haven't realized it yet.

Second: even though people rarely talk much about this pattern, it's astoundingly widespread. It appears in strongly-typed systems like Eclipse, in programming and data-declarative languages, in end-user applications, in operating systems, and even in strongly typed network protocols, although I didn't talk about that use case today. (Nutshell: a team I know using CORBA got fed up and added an XML parameter to every CORBA API call, defeating the type system but permitting them to upgrade their interface without horking every existing client. Bravo!)

Third: it can perform well! Or at least, "well enough". The playing field for potential optimizations is nearly unbounded, and with enough effort you can reduce just about everything to constant time.

Finally, it's surprisingly versatile. You can use it on a very small scale to augment one teeny component of an existing system, or you can go the whole hog and use it for everything, or just about anything in between. You can start small and grow into it as you become more comfortable with the pattern.

The Properties Pattern is not "just" name/value pairs, although the name/value pair certainly lives at the heart of the pattern.

If you believe Hofstadter, the Properties Pattern (using the Prototype Principle) is an approach to modeling that complements class-based modeling: both are fundamental to the way our brains process symbolic information.

I suspect that if you've read this far, you'll start seeing the Properties Pattern everywhere you look. It's embedded in many other popular patterns and systems, from Dependency Injection to LDAP to DNS. As legacy systems struggle to evolve to be more user configurable and programmable, you'll see more and more of your favorite software systems (and, I hope, languages) incorporating this pattern to varying extents.

Again: I wasn't able to find much literature about this pattern. If you happen to know of books, papers or articles that expound on it in more detail, please link to them in the comments!

I hope you found this interesting and potentially useful. It was definitely more work than my typical posts, so if it doesn't go over well, I'm happy to go back to the comedy routine. We'll see.

The Bellic School of Management Training

2008-09-28T23:08:00.000-07:00

I haven't been blogging much this summer. Mostly it's because all my free time has been spent engaged in an important research project called "What Would Niko Bellic Do?" I've been enrolled in a high-quality Management Scenario Simulator with the unconventional name "Grand Theft Auto IV", probably some sort of inside joke, and I've been going through all its Developer Management training courses.

You know how these corporate training videos go, right? They set up some contrived scenario with actors you're supposed to identify with, and the actors have inane discussions about sexual harrassment or bribing government officials or stealing company equipment, and then you're asked to answer questions about whether it was OK for Bob to grab Sue's ass in that particular edge-case scenario.

Seriously, I just took one of these courses at work. You'd think they're a joke, but no, they're considered Important Employee Training.

Well, this Niko Bellic course isn't much different, just more fun. I finished the final management training session a couple weeks ago. And by truly amazing coincidence, right after I finished the final training mission, my blog made it into the Top 100 Blogs for Development Managers (Q3 2008).

I personally thought that was a little weird, seeing as I've never written explicitly about software development management. Unless of course you count that one time where I wrote about how you don't need them.

BUT, now that I've finished all the Niko Bellic Advanced Management Training courses, I'm officially a Certified Expert Dev Manager, capable of not only handling unforseen tricky management situations, but also teaching others about them! So now I think my blog being in that list is totally justified.

All the training in the simulator was top-notch. A lot of it was ground I'd covered before in my days as an actual dev manager. I already knew how to react, for example, when my mafia boss told me he'd been betrayed by my previous manager, and I had to take him out. I'd already gone through stuff like that at Amazon dozens of times, so I breezed through those missions.

But even though a lot of the training was pretty obvious stuff, which you come to expect from these corporate training courses, I was pretty impressed by the final Management Training Mission, in which I had to make an Important Management Decision about an employee in my care. I'm sure many of you have also completed this course.

In the missions leading up to the finale mission, the "Grand Theft Auto" training software gradually unfolds a background story in which you and this other employee used to work together but had a disagreement that never fully reached closure. You have some trouble arranging a meeting with the employee to discuss it. But as luck would have it, after doing some especially fine work for one of your internal customers, they manage to convince the employee to meet with you, by gagging and handcuffing him and stuffing him on a plane in St. Petersburg and dropping him off at the docks in Liberty City. I had to admit, the simulator really nailed the value of building strong relationships with internal customers.

You can imagine how hard it is to write an online corporate training seminar — I mean, you have to get all this material across but keep it entertaining so people actually remember the lessons. The fine folks at the consulting firm "Rockstar" have done an admirable job of presenting the content. They cover all of the HR-sensitive topics an aspiring manager needs to know about: conflicts of interest, employee drug abuse, bribery, sexual harassment, stealing company property, I mean the list goes on and on. It's amazingly thorough. These guys clearly have real-world software industry experience.

And they managed to make it fun! They did this by using certain metaphorical devices, purely for dramatic effect. For instance, in the final mission, you're given a choice: you can either let the employee walk away, allowing bygones to be bygones, or you can execute him by shooting him in the head as he begs for his life. This playful enactment is obviously a metaphor for deciding whether to take the "next step" in dealing with a problem employee: whether or not to put him on a Performance Improvement Plan, or if that's already failed, whether or not to terminate employment. By generalizing it to a back-alley execution scenario, they've given you a way to apply the lesson across many similar situations. It's actually quite brilliant, and I'm sure other firms offering management training courses will take note.

As you know, in many management situations there's no "right answer": it's a personal decision, and different managers will make different choices. Hence, the final mission, unlike all the other missions, has no "success" outcome: you just pick a course of action and you have to live with it, just like in real life. Regardless of whether you choose to forgive him for displaying a moment of weakness, showing that you understand he's human like the rest of us, or you pop a cap in his motherfucking partner-betraying ass, you goddamn mother fucker *blam* *blam* *blam* *blam* *BLAM* *click* *click* *click*, the other actors will empathize with you and feel your pain, since no tough decision comes without a few regrets.

As it happens, the simulation software is remarkably open-ended. When I did the mission, I decided to let the employee walk, but then I accidentally ran him over with an 18-wheeler as I was driving out of the parking lot, killing him instantly. Oopsie! I confess I may not have been paying strict attention to some of the simulator's more obscure traffic rules by the end of the training sessions. Fortunately they weren't grading me on little driving slip-ups.

The game chose to interpret my accidental crushing of the employee beneath the wheels of my semi as intentional termination. I like to think of it as modeling the "passive-aggressive" approach, in which you don't fire the employee personally; you have someone else do it, such as HR, or perhaps your boss. Some managers prefer the passive-aggressive approach.

To illustrate why it's popular, I'll use an analogy from the restaurant industry. Have you ever noticed that at restaurants, your waiter doesn't bring your food? Other waiters always bring out your food, during which time your waiter is nowhere to be seen. This is so that if you become infuriated because you specifically ordered tartar sauce on the side, and after a 45-minute wait the chef seems to have emptied the entire bottle of tartar sauce on your fish sandwich in some sort of twisted artistico-culinary attempt to make it look like he threw up on it, then you don't blame your waiter. Instead, you unwittingly direct your anger at the person who brought your food, who makes sympathetic noises ("Gosh, I'm so sorry - I can't believe they messed that up!") and runs away, never to be seen again. After it's eventually resolved (by still other people bringing replacements out), your waiter finally rematerializes and apologizes for the kitchen screwup.

Studies have shown that this yields higher tipping on average. That's why your waiter doesn't bring your food. Now you know.

The passive-aggressive employee-discipline approach is a bit unusual, but some managers feel it's better to have the employee mad at "the company", since then the manager will then get better manager reviews from the employee. I'll write more about this technique in my upcoming "How to Be an Evil Manager" handbook. Should be quite popular! Maybe it'll even bump me up the Top 100 list.

Obliviating

Now that I've finished those training courses, I have all these half-written blogs I need to finish up. Tons of cool stuff to talk about. Unfortunately they're all stalled in various stages because I spent the summer playing video games and doing a lot of bike rides. Sometimes you've just gotta do that, you know.

GTA IV was a really good game, so after I finished it (which required some time before I got tired of experimenting with 5- and 6-star warrant levels), I started looking around for an equally good game. GTA IV was basically the best game I'd played since Oblivion, which was two years ago. I don't play games that often (just a few each year), so I figured there must be something good out there.

I looked and looked, and by golly, the best game out there was still Oblivion. Not only that, but in the intervening 2 years it's sprouted all sorts of mods and expansions like the "Knights of the Nine" and the "Shivering Isles". Neat.

So after lots of searching (and not finding) a good follow-up to GTA IV, I stumbled on a used copy of the Game of the Year edition. Since nothing else out there today looked anywhere near as good, I started playing Oblivion again.

Remember how a few months ago I finally turned off my last Windows box? (I switched 100% to using Mac clients and Linux servers: at home, at work, and on the road.) Well, technically to restart Oblivion where I left off, I was going to have to turn my XBox 360 back on, which felt like cheating. But the used Game of the Year edition I found was for the PS/3.

Decisions, decisions...

Actually it wasn't much of a decision at all. I'm on a low-Microsoft Diet, meaning I avoid Windows whenever I possibly can. Given that I'd gone for 3 or 4 months or so without interacting with a single Windows installation, I really didn't want to turn on my XBox. Plus it's summer and I don't need a space heater yet.

So I went for it. Even though my previous character had finished the main quest and was waiting for her imperial dragon armor, I decided to start from scratch and play the whole thing over again on the PS/3, but this time continue on to the expansions after finishing the main quest.

I tell ya: if Oblivion had come out today rather than 2+ years ago, it would still win the Game of the Year. It's just the best game, period.

To be honest, I kind of miss Morrowind (the previous game in the series). I find myself missing it even when I'm playing Oblivion. I think it may have actually had more personality than Oblivion. But I'm into the Shivering Isles expansion now, and it's way better than Oblivion on personality; I'm enjoying it even more than the main game.

I still find it weird that they don't re-make classic games. Don't you? I mean, they re-make classic movies, so why not games? Sure, we've got game franchises like Zelda and Mario and Donkey Kong — Nintendo is awesome at building character brands. But even Nintendo doesn't re-make great old games; they just carry forward characters and ideas and themes. But that just makes me miss the old games!

For instance, Zelda 64: Ocarina of Time — my God, that was a fantastic game. One of the best ever. Ditto for Final Fantasy X. You could argue that was the best game ever, and I wouldn't try to pop a cap in your motherffffffffffff — sorry, sorry, management training getting the better of me. Let's just say I wouldn't try to argue with you. It was a uniquely great game.

But both of those games, great as they were, could stand a make-over: updated graphics, updated game physics and mechanics, updated for newer platforms, etc. They'd be playable and (presumably?) popular today, and it would surely be a lot easier than trying to design a new game from scratch, so studios could release these remakes in "dry spells" between flagship game releases for extra revenue.

It seems like it'd be better than releasing crap games that they know suck. Which is most of them.

I don't know. Maybe people just prefer new games, and that's that. Maybe I'm different. But nearly 40 years as a gamer has convinced me that truly great games only come along every few years, and the rest are all just filler. I'd rather go back and re-play a favorite game, and get that rush of nostalgia, than fill the void by playing crap.

Ultima IV — now there was a game. Once or twice a decade a truly extraordinary, groundbreaking game comes along, and I remember when Ultima IV was that game. I would love to play it again, except this time without having to swap the City Disk with the Wilderness Disk and listen to my Commodore 64 crunch away for minutes on end.

One of the really key elements to the overall atmosphere of a game is the music, and most great games were great — in large part, I think — due to their music. Zelda 64, Final Fantasy X, Kingdom Hearts, Ultima IV, they all had great music. Sometimes the composer was limited by the sound system - in Ultima IV's case, very limited. But I have all of Ultima IV's music on my iPhone, converted from MIDI files. Dead serious. I love those tunes. They remind me of the old Quest for the Avatar days, back in the Navy when my 3 roommates and I manned the console 24 hours a day in shifts, trying to find mandrake root.

Someone please make an updated Ultima IV! I want IV back, dammit. The sequels never even came close.

Not likely, though. Maybe in a generation or two. It's usually at least a few decades before they remake a classic movie, so maybe I'm just not waiting long enough.

If they remake a classic game, I think it's really important to have the same music. This is where remakes (if you can call 'em that — they're usually either sequels or unrelated new games with similar elements) tend to screw things up. The music should be updated by a competent composer, but it should be the original themes. The Mario franchise does a pretty good job of this, actually. Zelda, not so much. I haven't heard the "Gerudo Valley" theme in any Zelda game since Z64, which came out ten years ago, and along with the title theme it's one of the two best Zelda themes ever.

Remember Metroid? It had really cool music all around. I didn't play any of the Metroid franchise after that, until last year when I played Metroid: Prime on the Wii. The music sucked! Actually the game kinda sucked too. The controls were phenomenal, and made me want to play all games like that, but the game was boring. Sigh.

It's also kind of important that remakes Not Suck. I guess franchise sequels aren't really remakes, though, so apparently it's OK if they suck. That seems to be the standard. There are exceptions. Mario Galaxy's music was Top 5 of all time. But if you can't afford that Japanese guy, it's OK to reuse old music! I'm serious! I give you my explicit permission!

Anyway, I'll probably keep playing Oblivion until New Castle Wolfenstein comes out. I just don't see anything else all that compelling on the horizon. Just counting the weeks until Bethesday's next Elder Scrolls release, I guess. Morrowind to Oblivion was a 5-year wait, so I'm gonna have to find something to do for the next 3 years.

Excuses aside, I should really blog more often. I was pretty amazed at how much email my last entry received. To be perfectly on the up-and-up, I haven't read the comments. I'm sorry, but I've been too busy trying to find a Daedric Katana to enchant. Priorities first!

But I checked my GMail account on Friday and it was completely swamped with dozens and dozens of new emails, many more than usual. People seemed to feel it made us buddies, which meant they could email me just to say "hi", and ask if I wanted to "hook up." I don't know if that's good or not, but I tried to answer all their mail.

Oh wait, that was Niko's account in the Corporate Email Training simulator. My bad.

Anyway, today's blog entry was just procrastination. I'm supposed to be getting some work done, so I'll get back to it. Maybe this bandit will have a katana.

Programming's Dirtiest Little Secret

2008-09-10T16:59:00.000-07:00

"And as for this non-college bullshit I got two words for that: learn to fuckin' type"
— Mr. Pink

This is another one I've wanted to write forever. Man, I've tried a bunch of times. No ruck. Not Rucky. Once again I'm stuck feeling so strongly about something that I'm tripping over myself trying to get my point across.

So! Only one thing left to try: bust open a bottle of wine and see if that gets the ol' creative juices flowing all over my keyboard. Rather than top-down, which is boring, let's go bottoms-up.

Once upon a time...

...in, uh, let's see... it was about 1982. Yeah. A looooong time ago. This is practically a fairy tale.

Once upon a time in '82, there was this completely hypothetical fictitious made-up dorky 12-year-old kid named Yeev Staigey, who was enduring his sophomore year at Paradise High School in Paradise, California. Yeev had skipped 3rd, 7th and 8th grades and entered high school at age 11, in a heroic and largely successful effort to become socially inept for the rest of his life.

Boy, I could tell you all sorts of stories about little Yeev at that age. He was even lamer and more pathetic than you're probably already imagining.

However, our story today concerns Yeev's need to take a, um, an... elective of some sort. I'm not sure what they called it, but at Yeev's school, you couldn't just take math and science and languages and history and all that boring stuff. No! Yeev was being educated in the United States of America, so he had to take "electives", which were loosely defined as "Classes Taught by the Football Coach because Some Law Said that Football Coaches Had to Teach A Course Other Than Football."

These "electives" (which you could "elect" not to take, in which case they would "elect" not to graduate you) were the kinds of courses that put the "Red" in Red-Blooded American. These were courses like Wood Shop, Metal Shop, Auto Shop, and of course that perennial favorite, Just Chop Your Hand Off For Five Credits Shop.

At the time our story begins, our pathetic hero Yeev is peering through his giant scratched bifocal goggles at the electives list, trying to find one that doesn't involve grease and sparks and teachers screaming for a medic, can anyone here make a tourniquet, help help help oh god my pension, and all that manly American stuff you find in Shop class.

Yeev noticed that one of the electives, surely placed there by mistake, was Typing. Like, on a typewriter. Yeev thought this seemed, in the grand scheme of things, relatively harmless. The worst that could happen was maybe getting your fingers jammed in an electric typewriter just as lightning hit the building, causing you to jerk wildly in such a way that your pants accidentally fall down around your ankles and everyone laughs loudly at the Mervyn's white briefs your mom bought you. That would be mildly embarrassing, yes, but in a few years almost nobody would remember, except when they saw you.

Despite this potential pitfall, Typing definitely sounded more appealing than Tourniquet Shop.

Yeev checked, and sure enough, the school's football coach was actually teaching the class. For real. Seeing as this was going to be the closest Yeev would ever get to a football field during his educational career, Yeev decided to go for it.

Yeev didn't know it at the time, but they say coaches make the best teachers. You know. "Them." "They" say it. It's got some truth to it, actually. Coaches have to get a bunch of complicated information through to people with the attention spans of hungry billy goats. That takes mad skilz, as "they" also say.

Have you ever noticed how on NFL Prime Time, the ex-coach commentators and coached ex-player commentators always have big, beefy hands, and they wave them at you as they talk, riveting your attention on the speaker? It's because your reptilian brain is thinking "that dude is about to hit me." Coaches know how to get your attention. They know how to teach.

So Yeev was pretty fortunate in getting a coach. It wasn't all roses, mind you. He was unfortunate in the sense that he was living in 1982, he had little to no experience with computers, and the school was so backwards that by 2008 they still wouldn't have a fugging website, apparently. And back in 1982 they could only afford budget for half the typerwriters to be electric; the rest were the old, manual, horse-drawn kind of typewriter.

It would have been better if Yeev had been learning to type today. Today they have fast keyboards, and smart programs that can show you your exact progress, and so on.

I feel almost jealous of people who need to learn to type today. Don't you?

But in 1982, little bifocaled Yeev had no software training programs, so he had to learn from a football coach.

And all things considered, this was pretty rucky.

Let me tell you how it went down...

Learning Licks

Have you ever watched a professional concert musician practicing? I'm talking about those those world-class ones, the kinds of musicians that were trained in academies in China and Russia and have all the technique of Japanese robots combined with all the musical soul of, well, Japanese robots.

Well, they practice like this: Fast, Slow, Medium. Fast, Slow, Medium. Over and over.

That's how they practice.

It's kinda like Goldilocks. You remember her, don't you? That little girl in the fairy tale that got eaten by bears? Too hot, too cold, just right. That's how musicians practice.

In classical music, they call difficult hunks of music "passages". In electric guitar music, they call 'em "licks". But it's pretty much the same thing. You want to train your fingers to swipe through those notes like a Cheshire Cat licking its big smile. Llllllick!

Here's how you train your fingers.

You start with a passage. Anything at all. At first it'll just be a single note. Later it'll become a few notes, a phrase, a measure, a couple of measures. Anything you're having trouble with and you want to master.

First you play the lick as fast as possible. You don't care about making mistakes! The goal of this phase is to loosen your fingers up. You want them to know what raw speed feels like. The wind should be rushing through their fingery hair! Yuck!

Next you play it as slow as necessary. In this pass you should use proper technique. That basically means "as proper as you can", because state-of-the-art technique is (a) constantly evolving and (b) always somewhat personal. You pick any discipline, they've got schools of thought around technique. There's no right answer, because our bodies all work a little differently. You just have to pick the technique that you like best and try to do it right.

Eventually you can invent your own technique. Sometimes you're forced to: I'll tell you about that in a little bit. But at first you should learn someone else's tried-and-true technique, and after you've mastered it, then decide if you want to change it. Before you go charging off in your own crazy directions, you need to master your form.

Form is liberating. Believe it. It's what they say, and they say it for sound reasons.

Whatever technique you choose, in the slow pass you don't care about speed AT ALL. You care about accuracy. Perfect practice makes perfect, and all that. You want your fingers to know what it feels like to be correct. Doesn't matter if it takes you 30 seconds per note. Just get it right. If you make a mistake, start over from the beginning, slower.

Finally, you play it "at speed". If you're practicing a musical instrument, you play it at the target tempo. You want your fingers to feel musical. Musicians generally agree that you don't want to make mistakes in this phase, or you're just practicing your mistakes. But realistically, most musicians are probably willing to make a few minor sacrifices here in the third pass, as long as the music shines through beautifully.

Let's call it 5 sacrifices a minute. That's what you're targeting.

Fast, Slow, Medium. Over and over. That's what they do. And it works!

Learning to Type

Yeev's football coach was a very wise man. I don't know if he played a musical instrument, but he sure as heck used classical practice ideas.

Yeev dutifully attended class once a day. First he had to learn the basics of typing. There's not much to it, really. You hold your hands in a certain position on "home row". You keep your wrists off the keyboard. There's a diagram showing which fingers type which keys. You memorize it. You try each key a few times.

Think back to kindergarten, when they had you writing the alphabet. You'd fill a line with "A"s, and then a line with "B"s. Just like that.

Within a day or two, you have the keyboard layout memorized, and given enough time, you can type anything without looking at the keys. Just a day or two, and you're already touch-typing!

After the basics, unsurprisingly, Yeev's class played a lot of Typing Football. This was a game the coach had invented to help making learning how to type fun. It wasn't too hard, since the coach astutely realized that not everyone in the class would have the NFL rule book and playbook memorized. Typing football pretty much involved dividing the class in half and moving the ball down the field by typing better than the other half.

The drills Yeev did in 1982 could be done a lot better today using software. Heck, today they have software that lets you practice typing by shooting zombies. How cool is that?

If there's any secret to learning to type, it's persistence. Yeev's class kept at it. Every day, five days a week for 12 weeks, they typed. They didn't have homework, since it wasn't expected that they'd have typewriters. They just came in, played typing football, and did the fast/slow/medium drills.

Sure, sure, there were nuances. Sometimes they'd practice common letter groupings in their language of choice, which in Yeev's case was English. Groups like "tion", "the" and "ing" had to be practiced until they rolled out with an effortless flick. Sometimes they'd practice stuff with lots of punctuation, or numbers, or odd spacing.

That kind of detail is beyond the scope of our story. It's all handled by typing software these days. You'll see.

So how did it turn out? Well, by the end of the semester, Yeev was typing a healthy 60 words per minute. And he wasn't even the best in the class. It was approximately a 45-day investment of 50 minutes a day.

And it was fun!

Realistically, these days, with better software and better keyboards, learning to type is probably more like a 30-day investment of 30 minutes a day.

Today Yeev types about 120 wpm. He entered college still typing around 60-65 wpm, but he decided to practice up after IM'ing with a fellow student named Kelly who typed 120 wpm, using an old Unix program called "talk". Yeev could feel her impatience as they IM'ed. He mentioned it, and she said: "You should see me on a Dvorak keyboard."

Yowza. Yeev was just socially ept enough by then to bite his tongue really hard, and not type anything at all.

But enough about Yeev. The guy's made-up anyway.

Do you need to learn to type?

Well, uh... yeah. You know you do. That's the thing. Even as you make excuses, you know deep down that you need to learn. Typing is how we interact with the whole world today. It doesn't make sense to handicap yourself.

Perhaps you're one of those people who declares: "I'm not rate-limited! I spend all my time in design and almost none of it entering code!" I hear that all the time.

You're wrong, though. Programmers type all day long, even when they're designing. Especially when they're designing, in fact, because they need to have conversations with remote participants.

Here's the industry's dirty secret:

Programmers who don't touch-type fit a profile.

If you're a touch-typist, you know the profile I'm talking about. It's dirty. People don't talk about dirty secrets in polite company. Illtyperacy is the bastard incest child hiding in the industry's basement. I swear, people get really uncomfortable talking about it. We programmers act all enlightened on Reddit, but we can't face our own biggest socio-cultural dirty secret.

Well, see, here's how it is: I'm gonna air out the laundry, whether you like the smell or not.

What's the profile? The profile is this: non-touch-typists have to make sacrifices in order to sustain their productivity.

It's just simple arithmetic. If you spend more time hammering out code, then in order to keep up, you need to spend less time doing something else.

But when it comes to programming, there are only so many things you can sacrifice! You can cut down on your documentation. You can cut down on commenting your code. You can cut down on email conversations and participation in online discussions, preferring group discussions and hallway conversations.

And... well, that's about it.

So guess what non-touch-typists sacrifice? All of it, man. They sacrifice all of it.

Touch typists can spot an illtyperate programmer from a mile away. They don't even have to be in the same room.

For starters, non-typists are almost invisible. They don't leave a footprint in our online community.

When you talk to them 1-on-1, sure, they seem smart. They usually are smart. But non-typists only ever contribute a sentence or two to any online design discussion, or style-guide thread, or outright flamewar, so their online presence is limited.

Heck, it almost seems like they're standoffish, not interested in helping develop the engineering culture. Too good for everyone else!

That's the first part of the profile. They're distant. And that's where their claim that "most of their time is spent in design" completely falls apart, because design involves communicating with other people, and design involves a persistent record of the decision tree. If you're not typing as part of your design, then you're not doing design right.

Next dead-giveaway: non-typist code is... minimalist. They don't go the extra mile to comment things. If their typing skills are really bad, they may opt to comment the code in a second, optional pass. And the ones who essentially type with their elbows? They even sacrifice formatting, which is truly the most horrible sin a programmer can commit. Well, actually, no, scratch that. It's the second worst. The worst is misspelling an identifier, and then not fixing it because it's too much typing. But shotgun formatting is Right Up There.

You know. Shotgun formatting? Where you shove all your letters into a shotgun, point it at the screen, and BLAM! You've Got Code? I knew a dude who coded like that. It was horrible. It was even more horrifying to watch him, because he stared directly downward at his keyboard while he typed, and he'd type with exactly two fingers, whether he needed them both or not, and about once a minute he'd look up at the screen.

During these brief inspections of his output, one of two things would happen. The first possibility was that he'd reach for his mouse, because he had been typing into the wrong window for the past 60 seconds, often with comical results. If he didn't reach for his mouse, he'd reach for the backspace key, which he would press, on average, the same number of times he had pressed other keys.

That dude just may have been CPU bound rather than I/O-bound, though, so I guess I'll cut him some slack.

B-b-b-but REFACTORING *fap* *fap* *fap*

Yeah, yeah. Refaptoring tools make you feel studly. I hear ya. I've heard it plenty of times. The existence of refactoring tools makes typing practically obsolete! A thing of the past! You just press menu buttons all day and collect a paycheck!

I got it. Really. I hear ya.

Here's the deal: everyone is laughing at you. Or if they're your close friend, they're just pitying you. Because you suck. If you really think refactoring tools are a substitute for typing, it's like you're telling us that it's OK for you to saw your legs off because you have a car. We're not fucking buying it.

If you are a programmer, or an IT professional working with computers in any capacity, you need to learn to type! I don't know how to put it any more clearly than that. If you refuse to take the time, then you're... you're... an adjective interjection adjective noun, exclamation.

Yeah, I went ahead and redacted that last sentence a bit. It's better this way. I want us to remain friends. You just go ahead and madlib that sucker.

The Good News

Here's the good news, though. Seriously, there's good news. Like, now that you're finally gonna learn to type, I've got good news for you.

And I know you're gonna learn. You know how I know? I'll tell you: it's because you've read this far!

Seriously. The fact that you can actually read sets you apart.

You'd be absolutely astonishedly flabbergasted at how many programmers don't know how to READ. I'm dead serious. You can learn to speed read even faster than you can learn to type, and yet there are tons of programmers out there who are unable to even skim this blog. They try, but unlike speed readers, these folks don't actually pick up the content.

It's the industry's other dirty little secret.

So! Given that you've read this far, you now realize that it's high time you learned to type. You know you can do it. You even know it's not going to be that hard. You know you're just going to have to give up a couple dozen games of Wii Golf, and suddenly you'll be, like, twice as productive without having had to learn so much as a new data structure.

You know. That's why I know you're going to learn.

So I'll tell you the good news: it's frigging easy! Fast, slow, medium! Get some typing software and just learn. We're not talking about dieting here, or quitting smoking. It doesn't matter how old you are, how set in your ways you are: it's still easy. You just need to put in a few dozen hours.

Hell, if you're having trouble, just email me, and I'll give you a personalized pep talk. I can afford it. I type pretty fast. Plus your email will be really short.

In fact, here's a mini pep talk for ya: I didn't know how to touch-type numbers until I'd been out of college for 3 or 4 years. I finally got fed up with the fact that every time I had to type a number, I had to sit up, look down at the keys, and fumble through with a couple of random fingers.

So I finally spent 15 minutes a day for, like, 2 weeks. That's it. You don't have to type numbers very often, as it happens, so after a week or so, every time I needed to type a number I just slowed down and typed it right. After about 2 weeks, I was typing numbers.

That was 15 years ago! 15! 15! 15! I love being able to type that without looking! It's empowering, being able to type almost as fast as you can think. Why would you want it any other way?

C'mon. It's time to bite the bullet and learn.

Where to start?

Well, if it were me, I'd go online and look for free typing software. I'd search for, oh, an hour or two at most, spread over a week or so. I'd try everything out there. If nothing free seemed to be doing it for me, I'd get Mavis Beacon. It's, like, the brand name for typing software. I have no idea if it's good, but I imagine it's a hell of a lot better than a football coach teaching you on an electric typewriter.

I dunno, man. I just can't... I can't understand why professional programmers out there allow themselves to have a career without teaching themselves to type. It doesn't make any sense. It's like being, I dunno, an actor without knowing how to put your clothes on. It's showing up to the game unprepared. It's coming to a meeting without your slides. Going to class without your homework. Swimming in the Olympics wearing a pair of Eddie Bauer Adventurer Shorts.

Let's face it: it's lazy.

There's just no excuse for it. There are no excuses. I have a friend, John, who can only use one of his hands. He types 70 wpm. He invented his own technique for it. He's not making excuses; he's typing circles around people who are making excuses.

Shame on them!

If you have two hands, then 70 wpm, error-free, is easily within your reach. Or faster. It ain't as hard as you think. It's a LOT more useful and liberating than you think.

And since you're Rucky enough to be learning today, you might as well learn on a Dvorak layout. It's a free speed boost. Might as well give yourself a head start!

That's it. That's my article. Please — learn to type. This should be a non-issue, NOT one of the industry's dirty secrets that nobody talks about. Tell your boss you're going to take the time. Get your employer to pay for the software. Have them send you off to a course, if necessary, so you can't weasel out of it.

Do whatever it takes.

Then let me know how it goes. Believe it or not, I want to hear your success stories. Send me mail. It'll make my day!

Business Requirements are Bullshit

2008-08-12T00:57:00.000-07:00

Some CEO emailed me the other day. I don't remember who it was; people mail me all the time about their blah blah yawn product service thingy, and on the rare occasions I bother to read mail from strangers, I don't usually remember anything about the email, even if I respond to it. I can remember broad categories of questions I get, but everything else is just a blur. That's senility for ya.

But this dude, or possibly dudette, asked me a really good question. One that made an impression. And by "really good", I mean it was one of the flat-out wrong-est business questions you could possibly ask. It was like asking: "How can I get the most out of my overseas child laborer pool?"

There's no right answer to a question that's just wrong.

But his question, well, people ask it a lot. In fact there are whole books that give answers to this inherently-wrong question.

His question was:

"Hello, blah blah, I'm the CEO or C-something-O or whatever of a company that blah blah BLAH, and I read your blog about Agile from almost 2 years ago, which kinda resonated with me in a scary way, inasmuch as I realized perhaps belatedly that I was being duped, BUT I was sort of wondering:
How do you go about gathering business requirements, so you know what to deliver to the customer?

Signed, blah blah blaaaaaaaaaah."

(Note: not a verbatim transcript. But close enough.)

And my answer was: Business requirements are bullshit!

Well... actually it was: "gathering business requirements is bullshit", plus a bunch of accompanying explanation. But the shorter version sure is catchy, isn't it?

The rest of this little diatribe expands a bit on my reply to this guy. (I actually sent him an email with much of this material in it. When I do reply to strangers, I still try do a thorough job of it.)

Before we dig into the meat and potatoes of today's meal, let's talk about my credentials, and how it is that I am qualified to offer an opinion on this topic.

My Credentials

If you're a business person stumbling on my blog for the first time, welcome! Allow me to introduce myself: I don't matter. Not one teeny bit. You shouldn't care about my credentials! All that matters is the message, which is hopefully so self-evident that if you hadn't already realized it, you'll be kicking yourself.

It's OK, though. Kick away! In fact, give yourself an extra kick for me. Kicking yourself is a great way to remember a new lesson down in your bones. When you get actual bones involved, the painful throbbing for the next few days provides just enough of a reminder that you'll never forget again.

In any case, I make no claim to any credentials whatsoever, and this advice is all straight from my butt. Do what you like with it. The advice, that is.

With my credentials out of the way, let's see about this guy's question.

By the way, I'm specifically addressing this rant towards you only if you (or your team, or company) are building your own product or service offering, or at least defining it. If you're asking consultants to define the product for you, well, you're on your own. Good luck. And if you're a consultant, well, you don't get much choice about what to build, so you're also (mostly) on your own. Although I'd advise you to try to find work that fits the strategy I outline here, as you'll have more fun with it and do a better job overall.

Gathering Requirements 101

It's the first thing everyone wants to do! It's the first thing they teach you in Project Management Kindergarten: the very first thing you should do on a new project is look both ways before crossing the street to your building. Assuming you parked across the street. And the next thing is: start gathering business requirements!

Ah, the good old days of Project Management 101. Life seemed so easy back then.

The requirements-gathering process they teach you typically involves finding some "potential customers" for your product, and interviewing them in a nonscientific way to try to figure out what they want out of your proposed product. Or service. Or whatever. It doesn't really matter.

Potential customers can be hard to come by, especially since you're building something that nobody will ever, EVER want. Well, that's getting ahead of myself. Let's just agree that when you're setting out to Gather Business Requirements, your potential customers usually aren't already sitting in the room with you.

Sometimes it's a lot of effort to go find real customers and bring them in and get them to agree to be grilled for hours. Hence, requirements-gathering sometimes takes the form of "role play", where you get some poor saps in Accounts Receivable to pretend they're Potential Customers for your product, and you interview them instead.

You might be laughing about those poor Role Playing schmucks, but in reality it doesn't much matter who you interview, because by the time you get to this phase in the project, you've already screwed it up beyond all hope of repair.

That process of looking around for customers? It's a plot device that novelists like to call foreshadowing. If you don't have any readily available customers now, then they sure as hell won't be readily available when your product launches.

Allow me to illustrate with a Case Study.

An Illustrative Case Study

See? Credentials or no, I sure can talk the lingo, eh? I also like my incisive use of the popular business term "schmuck" in the previous section. It's a term that can apply to people in all phases of project catastrophes, not just Requirements Gathering.

For our Case Study, I will not do any research and the product will be entirely fictional. This is the approach used by most real companies.

Let's say that our Ideas Department has decided that we should get into Personal Nose Hair Grooming devices, because there's clearly a huge unmet need for nose hair grooming, as evidenced by Karl, our accountant, who has a thatch of nose hair that's almost long and thick enough to be a mustache, if only he would groom the thing a little. And we've seen lots of people like Karl. Clearly a nose-hair grooming kit would be the ideal addition to any man's personal grooming lineup, which typically consists of spitting into the sink and thinking he should get the mirror fixed someday.

So we need to gather some Business Requirements. Unfortunately nobody wants to talk directly to Karl, because, well, nose hair can sometimes be a mildly sensitive issue in some cultures. Plus nobody wants to make eye contact with him. So instead we hire some people to go out and do focus-group studies on the subset of people who are comfortable talking publicly about their nose-hair grooming problem, notwithstanding the fact that everyone in this tiny category is obviously too crazy to trust with important business questions.

The focus groups find a few nut-jobs who say: "Yeah, I'd love a nose-hair grooming appliance! Plucking your nose hair is painful, and trimming it involves jamming a small whirly Osterizer thing all the way up to your brain. Maybe if I just combed it into a mustache, nobody would notice!" And behold, we have the makings of some Business Requirements.

The product is a complete failure, of course. The project postmortem reveals that the user studies and focus groups failed to realize that only certain men, namely "men with significant others", ever even notice their nose hair, even when said hair becomes long enough to interfere with tying their shoelaces. The significant other must forcibly remove the nose hairs with garden shears at least twice before the man gets the hint and starts attending to the problem himself.

Not to mention the fact that a nose-hair mustache would be even more obvious and comical than a comb-over. Well, almost.

In the end, it doesn't matter how great a Nose Hair Groomer we build, because we failed at the business-requirements gathering phase: nobody actually wants this product. The people who might want it don't know they have a nose-hair issue, and the ones who do know just trim it.

The thing that might surprise you is that this fictitious (and yet eerily familiar) Case Study isn't just an illustration of how gathering business requirements can go afoul. We're not talking about a failure mode for Gathering Business Requirements (GBR). We're talking about something more fundamental: the GBR phase of a project is a leading indicator that the project will fail.

Put another way: GBR is a virtual guarantee that a project is building the wrong thing, so no amount of brilliant execution can save it.

From Business Requirements to Bad Idea

I learned a lot about the Fine Art of Building Shit that Nobody Wants back at Geoworks, when in 1993-1994 I was the Geoworks-side Engineering Project Lead for the HP OmniGo 100 and 110 palmtop organizer products. Both of them launched successfully, and nobody wanted either of them.

People spend a lot of time looking at why startups fail, and why projects fail. Requirements gathering is a different beast, though: it's a product failure. It happens during the project lifecycle, usually pretty early on, but it's the first step towards product failure, even if the project is a complete success.

Self-professed experts will tell you that requirements gathering is the most critical part of the project, because if you get it wrong, then all the rest of your work goes towards building the wrong thing. This is sooooort of true, in a skewed way, but it's not the complete picture.

The problem with this view is that requirements gathering basically never works. How many times have you seen a focus group gather requirements from customers, then the product team builds the product, and you show it to your customers and they sing: "Joy! This is exactly what we wanted! You understood me perfectly! I'll buy 500 of them immediately!" And the sun shines and the grass greens and birds chirp and end-credit music plays.

That never happens. What really happens is this: the focus group asks a bunch of questions; the customers have no frigging clue what they want, and they say contradictory things and change the subject all the time, and the focus group argues a lot about what the customers really meant. Then the product team says "we can't build this, not on our budget", and a negotiation process happens during which the product mutates in various unpleasant ways. Then, assuming the project doesn't fail, they show a demo to the original customers, who say: "This is utterly lame. Yuck!" Heck, even if you build exactly what the customer asked for, they'll say: "uh, yeah, I asked for that, but now that I see it, I clearly wanted something else."

So the experts give you all sorts of ways to try to get at the Right Thing, the thing a real customer would actually buy, without actually building it. You do mockups and prototypes and all sorts of bluffery that the potential customer can't actually use, and they have to imagine whether they'd like it. It's easy enough to measure usability this way, but virtually impossible to measure quality, since there are all sorts of intangibles that can't be expressed in the prototype. I mean, you can try — we sure tried on the OmniGo products — but in this phase nobody "imagines" that the thing will weigh too much, or be too slow, or will go through batteries like machine-gun rounds, or that its boot time will be 2 minutes, or any number of other things that ultimately affect its sales.

And even if you do a world-class job of prototyping and getting at the "real" desired feature set, it still goes through a series of iterations with the engineers and product team, who aren't the actual customers and have little interest in building what the customer really wants (because they're not being measured or evaluated on that — they're evaluated on delivering what everyone ultimately agrees to deliver). During each iteration they push back on things the customers asked for. As the proposal degrades, the customers like it less and less, but they want to be helpful, so eventually they say, "Yeah, I guess I could use that." (Which means: I wouldn't take one of these even if they were giving it away.)

Don't get me wrong: I'm not saying that usability teams can't do a good job. It's just that when the project implementation team and the target customer aren't exactly the same group of people, then there are inevitably negotiations and compromises that water an idea down about two levels of quality: great becomes mediocre, and ideas that start as "pretty good" come out "just plain bad."

So I'm tellin' ya: gathering requirements is the Wrong Way To Do It. At best, it results in mediocre offerings. To be wildly successful you need a completely different approach.

The investing analogy

Warren Buffett and Peter Lynch, both famous and successful investors, say pretty much the same thing about investing. Peter Lynch's mantra sums it up: "invest in what you know."

If you actually take the time to read Lynch's books (which I have), you'll see that this pithy mantra is a placeholder for something a bit more subtle: you should invest in what you know and like. You should invest in companies that make products or services that you are personally excited about buying or using right now.

When you invest with this strategy, you're taking advantage of your local knowledge, which tends to be more accurate than complicated quantitative packets put together by analysts. And your local knowledge is definitely more accurate than the reports produced by the companies, who want to paint themselves in the nicest light.

Warren took a lot of heat in the 1990s for not investing in the tech sector. But hey, he didn't feel comfortable with tech, so he didn't invest in it. One way to look at this is: "ha ha, what a dinosaur, he sure missed out, and now he's, uh, only the richest person in the world by a small margin." But another, more accurate way to look at it is this: he's the richest person in the world, you asshole. When he gives you investment advice, take it!

And Warren's advice is: don't invest in stuff you don't understand! Even if it seems like a sure thing.

That's the hard part. Sometimes it looks like a surefire winner for some large group of people that doesn't actually include you personally at this particular moment. But it's a really large group!

Let's say, for instance, that you hear that Subway (the sandwich franchise) is going to do an IPO. They've been privately held all these years and now they're going public. Should you invest? Well, let's see... the decision now isn't quite as cut-and-dried as it was in their rapid-expansion phase, so, um, let me see, with current economic conditions, I expect their sales to, uh... let me see if I can find their earnings statement and maybe some analyst reports...

No! No, No, NO!!! Bad investor! That's the kind of thinking that loses your money. The only question you should be asking yourself is: how many Subway sandwiches have I eaten in the past six months? If the number is nontrivial — say, at least six of them — and the rate of sandwiches per month that you eat is either constant or increasing, then you can think about looking at their financials. If you don't eat their sandwiches, then you'd better have a LOT of friends who eat them every day, or you're breaking the cardinal rule of Buffett and Lynch.

The investing analogy is an important one, because if you're a company or team planning to build something, then you're planning an investment. It's not exactly the same as buying stock, but it amounts to the same thing: you're betting your time, resources and money — all of which boil down to money (or equivalently time, depending on which one is in shorter supply.)

So when translated into project selection, Buffett's and Lynch's advice becomes: only build what you know. The longer, more accurate of the version of the investing rule — only invest in what you know and are excited about using yourself right now — has a simpler formulation for products and businesses. That formulation goes like this:

ONLY BUILD STUFF FOR YOURSELF
That's the Golden Rule of Building Stuff. If you're planning to build something for someone else, let someone else build it.

Building stuff for yourself

You can look at any phenomenally successful company, and it's pretty obvious that their success was founded on building on something they personally wanted. The extent that any company begins to deviate from this course is the extent to which their ship starts taking on water.

And the key leading indicator that they're getting ready to head off course? You guessed it: it's when they start talking about gathering business requirements.

Because, dude, face it: if it's something you want, then you already know what the requirements are. You don't need to "gather" them. You think about it all the time. You can list the requirements from memory. And usually it's pretty simple.

For instance, a few years ago I announced to some friends: "I sure wish someone would make a product you can spray on dog poop to make it, you know, just dissolve away." My friends laughed loudly and informed me that this was (apparently) the premise of some Adam Sandler movie I hadn't seen.

Well, OK, sure, but... I mean, they kinda missed the point. I still want the product. Its requirements list is pretty simple. Here's the business requirements list:

a) It should dissolve dog poop.

Gosh. Can that really be the entire list? You bet! Sure, there are lots of implicit requirements: it shouldn't cost a fortune, it should be environmentally friendly, it shouldn't kill kittens, etc. But those kinds of requirements are true for all products and services.

If I knew how to make this product, then Adam Sandler movie or no, I'd probably try to make it. The target market is larger than just pet owners; anyone living near a park would probably own a bottle or two. I would use the stuff like it's going out of style. I'd attach it to my shoes so that every time I took a step it would spray the area in front of me, like a walking garden hose.

Building a product for yourself is intrinsically easier, since you don't have to gather requirements; you already know what you want. And you also know, for any given compromise anyone suggests, whether it will ruin the product. If someone says, "I have a product that dissolves dog poop, but it takes 18 hours", then you'll know you've entered into "prolly not worth it" territory. You don't have to go ask the focus group. You just know.

The Mistake of Imagination

Despite its obvious advantages, following the rule of building stuff for yourself is actually really hard to carry out in practice. Why? Oddly enough, it's ego.

For one thing, people like to think they're unique and special, and that their tastes aren't necessarily widely shared by others. This is what drives fashion: the need to differentiate yourself from "the crowd", by identifying with some smaller, cooler crowd.

The reality is that for any given dimension of your personality, there are oodles of people just like you. If you want something, other people want it too. You define a market: a bunch of them, in fact.

You just have to be smart about which of your needs you want to fulfill, since if it's building yachts, well, it's not exactly going to be mass-market, unless you find a way to build a mass-market yacht. Which would be pretty cool, incidentally. I'll buy one if you make it.

It's also really easy to fool yourself into thinking that this is a product or service you would use, because, hey, you have a great imagination. When you lose your car keys, you can picture them in all sorts of places: the kitchen drawer, the coffee table, on top of the fridge, and when you picture them there, it's just as vivid as a memory. So you wind up looking for them everywhere! Your powerful imagination is pretty much your worst enemy when it comes to deciding whether you'd like something enough to use it yourself.

We all made the Mistake of Imagination on the OmniGo projects. "I could see myself using this product," we'd tell ourselves, "if, that is, I were the kind of person who used this product, which I could sort of envision." You'll tell yourself almost anything to justify the work you're doing. Giving up in mid-project is a big loss of face for an individual, harder for whole teams, virtually impossible for most companies. The OmniGo had four companies involved, making it the hardest possible project to back out of, even though by the halfway point virtually everyone involved knew the product would kind of suck.

What we really wanted, while we were building the OmniGo, was summed up by our brilliant product manager Jeff Peterson. We were having beers one day, and he said, "You know, this thing is just getting way too complicated! It doesn't need a 12C calculator emulator! It doesn't need a spreadsheet! It doesn't need a database application! I mean, come on! All it needs is a notepad, a simple calendar, an alarm clock, and maybe a pocket calculator, and it should fit in your front shirt pocket, and it should be a phone."

It was 1994 and he was describing the iPhone. And you know what? He was right! That was what we wanted. But HP was driving the spec, and they weren't building it for themselves. They were building the product specifically for this imaginary group of high-power on-the-go consumer accountants who use 12C calculators and want a whole desktop suite crammed onto their 1994-era mobile device. And that's just who bought the thing: pretty much nobody. (They sold a few thousand units, which in mass-market terms is "pretty much nobody.")

Trimming the Requirements

Who was it who said that you're done writing not when there's nothing left to add, but when there's nothing left to take away? Was it St. Exupéry? I promised not to do any research for this rant, but I think it might have been him.

Ideally the product you're building for yourself should be simple to describe, so that other people can quickly evaluate whether they, too, want this thing. It's often called the "elevator pitch", because you should be able to describe the product in the time between when the cable snaps and the elevator hits the ground. "Dissolves dog poooooop!!! <crash>" It used to just be the time for an elevator ride, but those investors keep raising the bar.

You can almost always make a product better by trimming the requirements list. We're talking brutal triage: throwing out stuff that's really painful to lose, such as the ability to change the battery.

If you're lucky, you should be building a product that so excites everyone involved that everyone has an opinion, and you wind up spending most of your time in triage.

When you're trimming the business requirements, then you're exhibiting healthy project behavior. This contrasts directly with gathering requirements, which has both the connotation that you're clueless about the product and the connotation that you're inflating the requirements list in direct conflict with schedule, usability and fashion. Trimming: good. Gathering: bad.

Trimming the requirements list is a leading indicator that you're a smart company who's about to launch something major. An ideal requirements list looks something like this:

a) It should dissolve dog poop.

As a great real-life example, consider the the Flip camcorder, which kinda came out of nowhere and "stole" 13% of the camcorder market (although I'd bet good money that it actually created new market share). Does it dissolve dog poop? Well, no, but it's still pretty cool:

1) it costs $150 or less. (A lot less, actually.)
2) it has no cables or wires. Just one flip-up USB connector.
3) it has one big red button: RECORD, plus a teeny one for playback.
4) it doesn't take cartridges or cassettes or discs or cards or anything
5) it doesn't have any controls or settings or anything
6) it stores one hour of video and has roughly one hour of battery life
7) it's about the size of a cell phone
8) it records videos that work well with YouTube
9) it comes in pretty colors

I mean, DAMN, those guys knew what they were doing. We always used to joke about a product so simple that it only had one button, which we pressed for you before it left the factory. That's how simple a product needs to be in order to make the mass market. One button, pretty colors. They nailed it. Talk about a missed startup opportunity. (Flip guys: if your equity plan is still reasonable and you want someone to make your desktop software not suck, and yes, it really sucks, please give me a call.)

You don't need an original idea to be successful. You really don't. You just need to make something that people want. Even if someone else appears to be making something popular, it's usually possible to improve on the idea and grab market share. And it's painfully counterintuitive at times, but the best improvements often come from simplifying.

The easiest way to build a product that kicks ass is to start with someone else's great idea (camcorders, for instance), and take stuff away.

In any event, originality is overrated. Coming up with something completely original isn't just hard to do: it's also hard to sell, because investors (and possibly customers) will need to be educated on what this new thing is and why people would want it. And when it comes to buying stuff, nobody likes to be educated. If the product isn't immediately obvious, investors and customers will pass it up.

It's easy to come up with new product ideas if you start with the understanding that everything sucks. There are no completely solved problems. Just because someone appears to be dominating a market with an "ideal" offering doesn't mean you can't take market share from them by building a better one. Everything can stand improvement. Just think about what you'd change if you were doing it for yourself, and everything should start falling into place.

If nothing else, building things for yourself is more fun, so you're successful regardless of what happens. But it also has great product-survival characteristics, because people can't bluff you into making something lame.

Sometimes you just can't win

By way of don't-sue-me disclaimer, I should point out that building something for yourself doesn't guarantee success. Even if you build a product that most of your target market really really wants, and you hit the right price point and release date and everything, your product can still fail catastrophically.

The example that leaps to mind is that company in the 1990s (can anyone remind me of the name? I've forgotten) that built a mountain-bike seat extender. I ride mountain bikes, yes, on actual mountains, so this product made immediate sense to me. I really wanted one. Sounds like a winner, right?

The basic physics problem this company was solving is that a lower seat position gives you better balance, and a higher seat position gives you more power. It's a trade-off. You generally want more power when you're grinding uphill, and you want more balance when you're speeding downhill. But during a race you would have to give up precious seconds to adjust your seat between every uphill and downill transition; you'd get your butt kicked. Even as a casual rider, adjusting your seat height all the time is annoying enough that most people just don't do it, resulting in some sacrifice of balance, power, or both.

So this brilliant, innovative company came out with a well-made product that lets you adjust your seat height "in flight", as it were, without slowing down, and without adding much (if any) weight to the bike. I don't remember exactly how it worked, but it was a reasonable implementation.

Interestingly, it didn't matter how it worked. It could have been actual magic: a product that read your mind and raised or lowered your seat by exactly the right amount, at exactly the right speed (you don't want it to rabbit-punch you in the nuts, for example — remind me sometime to tell you about how I found that out), as frequently or infrequently as necessary to strike the perfect trade-off of balance and power for any slope, at any time.

And it still would have failed, even if 90% of the mountain biker population, like me, secretly coveted the product. It could have cost fifty cents, been available everywhere, and been installable by a four-year-old with one hand in under two seconds. It could have come pre-installed on all bike models, via a brilliant channel deal with all the main manufacturers (or retailers), and bikers would have ripped the thing off the bike faster than their kickstand (which is usually the first thing to go.)

So, uh, what happened, exactly? Wasn't I just ranting that building a product for yourself, one that you know is the Right Thing for some well-defined audience consisting of people just like you in some dimension — wasn't I ranting that this would always work?

Well, no. It's just the best way to pick projects. But they can still fail for surprising reasons.

In this case, the fundamental marketing force that the company failed to take into account was fashion. People don't often think of mountain biking (or programming, for that matter) as a fashion industry, but failing to understand the role of fashion is a recipe for random disasters.

What happened was this: while they were hyping the product — by demoing it at trade shows, letting bike magazines check it out, and generally working their way towards getting retailer shelf space — the pro bikers took one look at the thing and said: "Hey, looks pretty cool. Maybe I'll get one for my girlfriend. Or my fugging grandma. How much is it?"

At which point, everyone (me included) who had been ranting about standing in line to buy one when they came out, we, ah, we were very quick to point out that we were also excited to buy them for our grandmothers, whom we loved just as much as the pros love their grandmothers, thank you very much. In fact, our grandmothers are too macho to use this thing. Maybe we'll put one on our kid's stroller! So there!

And that's how a great product, one that probably would have changed biking nearly as fundamentally as the derailleur, was doomed from birth because the trendsetters of Mountain Biking Fashion pronounced it a Product for Sissies, and that was that.

Heck, even derailleurs are falling out of fashion in some circles. Go figure. Someone took the time-tested "solved problem" of bicycles, removed something, and wound up creating a new market.

Fashion is generally hard to predict, but it usually means "sacrificing comfort or convenience for the sake of style". Take another look at the iPod: it has almost no features. It doesn't have an "off" button. Heck, you can't even change the battery. Not exactly convenient in many ways. But it sure has style!

Fashion's not the only way your otherwise perfect product can fail, but it's definitely one to keep an eye on.

Less is more... more or less

OK, fine, I haven't exactly been following my own advice about minimalism. But I have been writing my blog the way I like it, haven't I? So even if nobody reads it, I'm still having fun. If you're going to create something that has a nonzero chance of failure — and believe you me, it's nonzero — then you might as well have fun doing it, right?

Anyway, there you have it: the slightly expanded version of the email I sent that CEO guy. Gathering business requirements is hokum. Hooey. Horseshit. Call it what you want, but it's a sign of organizational (or individual) cluelessness. If you don't already know exactly what to build, then you're in the wrong business. At the very least, you should hire someone who does know. Don't gather business requirements: hire domain experts.

If you can't think of anything in your company's "space" that you personally would use, then you should think seriously about (a) changing your company's direction, or (b) finding another company. This is true no matter what level you're at. You should be working on something you love, or failing that, at least working on something that you know really well.

"But... but..." I hear you saying. I hear you! You sort of get what I'm saying, but you have all these reservations and objections and questions and stuff.

Well, that's what the comments section is for. I'm sure you can think of some other explanation for why Warren Buffett is the richest person in the world. Let's hear it!

Done, and Gets Things Smart

2008-06-16T18:44:00.000-07:00

Disclaimer: I do not speak for Google! These are my own views and opinions, and are not endorsed in any way by my employer, nor anyone else, for that matter.

Everyone knows and quotes Joel's old chestnut, "Smart, and Gets Things Done." It was a blog, then a book, and now it's an aphorism.

People quote Joel's Proverb all the time because it gives us all such a nice snuggly feeling. Why? Because to be in this industry at all, even a bottom-feeder, you have to be smart. You were probably the top kid in your elementary school class. People probably picked on you. You were the geek back when geeks weren't popular. But now "smart" is fashionable.

That's right! All those loser kneebiter jocks in high school who played varsity and got all the girls and sported their big, winning, shark-white smiles as they barely jee-parlor-fran-saysed their way through the classes you coasted through: where are they now? A bunch of sorry-ass bank managers and failed insurance salesmen and suit-wearing stiffs at big law firms working for billion-dollar conglomerates where their daddies got them VP jobs where they just have to show up and putt against the other VPs on little astroturf greens in the hallways!

That's right, los... er, well, some of them are doing OK, I guess. "But they're not as rich as Bill Gates!" That's the other big tautology-cum-aphorism we geeks like to pull out when we're feeling insecure. Notwithstanding that Bill had a rich daddy and made his money through exactly the kind of corporate shenanigans that Big Meat is using on us today, with those selfsame jocks. We like to think the more important point is that Bill, Jobs, Bezos and the other tech billionaires (all fierce, business-savvy people) were geeks, and are thus evidence of the revenge of the nerds. And hey, tech did make a lot of money in the 80s and 90s. So "smart" is sort of in fashion now.

Unfortunately, "smart" is a generic enough concept that pretty much everyone in the world thinks they're smart. This is due to the Nobel-prize-winning Dunning-Kruger Effect, which says, in effect, that people don't know when they're not smart. (Note: people have pointed out that it was an Ig-Nobel. Me == Dumbass. Fortunately, "me == dumbass" was more or less the point of the article, so I'll let it stand.)

So looking for Smart is a bit problematic, since we aren't smart enough to distinguish it from B.S. The best we can do is find people who we think are smart because they're a bit like us.

Er, what about Gets Stuff Done, then? Well, the other qualification you need to be in this industry is that you have to have produced something. Anything at all. As soon as you write your first working program, you've gotten something done. And at that point you probably think of yourself as being a pretty hot market commodity, again because of the Dunning-Kruger Effect.

So, like, what kind of people is this Smart, and Gets Things Done adage actually hiring?

I've said it before: I thought I was a top-5% (or so) programmer for the first fifteen years I was coding. (1987-2002). Every time I learned something new, I thought "Gosh, I was really dumb for not knowing that, but now I'm definitely a superstar." It took me fifteen frigging years before I realized that there might in fact still be "one or two things" left to learn, at which point I started looking outward and finally discovered how absolutely bambi-esquely thumperly incompetently clueless I really am. Fifteen years!

But hell, let's be honest here: I still think I'm smart. We all do. Sure, I've managed to figure out, at least partly, just how un-smart I am, but I've got the whole "I'm smart" thing so hardwired from when I was a kid surrounded by "dolts" who couldn't memorize things as fast as I could, or predict the teacher's punch line way in advance, or whatever questionable heuristic I was using for measuring my own smartness, that it's hard not to think that way. And of course the "dolts" were way better than me at just about everything else. And they think they're pretty smart too! Definitely above average, anyway.

Squaaaaawk

So we all think we're smart for different reasons. Mine was memorization. Smart, eh? In reality I was just a giant, uncomprehending parrot. I got my first big nasty surprise when I was in the Navy Nuclear Power School program in Orlando, Florida, and I was setting historical records for the highest scores on their exams. The courses and exams had been carefully designed over some 30 years to maximize and then test "literal retention" of the material. They gave you all the material in outline form, and made you write it in your notebook, and your test answers were graded on edit-distance from the original notes. (I'm not making this up or exaggerating in the slightest.) They had set up the ultimate parrot game, and I happily accepted. I memorized the entire notebooks word-for-word, and aced their tests.

They treated me like some sort of movie star — that is, until the Radar final lab exam in electronics school, in which we had to troubleshoot an actual working (well, technically, not-working) radar system. I failed spectacularly: I'd arguably set another historical record, because I had no idea what to do. I just stood there hemming and hawing and pooing myself for three hours. I hadn't understood a single thing I'd memorized. Hey man, I was just playing their game! But I lost. I mean, I still made it through just fine, but I lost the celebrity privileges in a big way.

Having a good memory is a serious impediment to understanding. It lets you cheat your way through life. I've never learned to read sheet music to anywhere near the level I can play (for both guitar and piano.) I have large-ish repertoires and, at least for guitar, good technique from lots of lessons, but since I could memorize the sheet music in one sitting, I never learned how to read it faster than about a measure a minute. (It's not a photographic memory - I have to work a little to commit it to memory. But it was a lot less work than learning to read the music.) And as a result, my repertoire is only a thousandth what it could be if I knew how to read.

My memory (and, you know, overall laziness) has made me musically illiterate.

But when you combine the Dunning-Kruger effect (which affects me just as much as it does you) with having one or two things I've been good at in the past, it's all too easy to fall into the trap of thinking of myself as "smart", even if I know better now. All you have to do, to be "smart", is have a little competency at something, anything at all, just enough to be dangerous, and then the Dunning-Kruger Effect makes you think you're God's gift to that field, discipline, or what have you.

This is why everyone loves Smart, and Gets Things Done, which Joel always writes in boldface. We love it because right from the start it has the yummy baked-in assumption that you are smart, and that you get things done. And it also tacitly assumes that you know how to identify other people with the same qualities!

But... you don't.

It sucks, but the Dunning-Kruger Effect is frighteningly universal. It's a devil's pitchfork two horrible, boldfaced prongs. First:

Incompetent people grossly overestimate their own competence

We already talked about that, right? You're a good programmer! Heck, you're a great programmer! You're "smart", so anything you don't know you can go look up if you need it! Right?

Welcome to incompetence.

The second prong is a bit longer, and has a barbed poison tip:

Incompetent people fail to recognize genuine skill in others

Both prongs are intrinsically funny when you're watching them in action in someone else, and they're incomprehensible when anyone tries to poke you with them. Not necessarily offensive, mind you: you might get offended if someone tries to imply that you're not as competent as you feel you are, but it's more likely (per the D-K effect) that you'll simply not believe them. Not even close. You're smart! They're just wrong! Gosh!

The second prong, that of the ability to recognize true competence, has major ramifications when we conduct interviews. That's what Joel was writing about in Smart and Gets Things Done, you know: conducting technical interviews.

How do you hire someone who's smarter than you? How do you tell if someone's smarter than you?

This is a problem I've thought about, over nearly twenty years of interviewing, and it appears that the answer is: you can't. You just have to get lucky.

It's easy for a candidate to sound smart. Anyone can use big jargon-y words and talk the talk like nobody's business, but I'm living, breathing proof that articulacy doesn't connote any other form of intelligence. Heck, the Markov-chain synopses of my blogs that people post in quasi-jest tend to look like I wrote them.

All too often I find myself on interview loops where the candidate knows a seemingly astounding amount about coding, computer science, software engineering, and general industry topics, but we find out at the last minute that they can't code Hello, World in any language.

This is, of course, one of the failure-patterns that Joel's Get Things Done clause is designed to catch. But interviews are conducted under pretty artificial conditions, and as a result they wind up being most effective at hiring people who are good at interviewing. This is a special breed of parrot, in a way. Interviewing isn't a particularly good predictor of performance, any more than your rank in a coding competition is a predictor of real-world performance. In fact, somewhat depressingly, there's almost no correlation whatsoever.

If interviews suck, then why do we still do them this way?

Lots of reasons. One is just history: everyone else does it that way. Companies tend to hire pretty similar HR departments, and HR tends to guide companies towards doing it the way everyone else does it. Same goes for technical management, which is all too often HR-driven as the company grows.

Another is that interviewing is already a fairly time-intrusive function for the interviewers, and it tends to be miserable for the interviewees. Trying to make the process somehow more rigorous or accurate just exacerbates these side effects.

Another is the "rite of passage" phenomenon, wherein engineers feel that if they had to go through the gauntlet, then everyone else should, too.

So for the most part, everyone does the same non-working variety of interviews, and hopes for the best.

As far as identifying good people goes, the best solution I've ever seen was at Geoworks, where you were required to do a six-month internship before you could get hired full-time. This seems to be the norm in non-tech departments at most tech companies. They often substitute "contractor" for "intern", but it works out roughly the same. Geoworks is the only company I've seen stick to their guns and make it mandatory for engineers.

However, I'm convinced that it only worked because Geoworks seeded the engineering staff with great people. The founding engineers set up a truly beautiful software engineering environment, with lots of focus on tools, mentoring, continuing education, "anarchy projects" to let off steam and encourage innovation, and a host of other goodnesses.

I've been a contractor at companies that had no good engineers at all, literally none whatsoever. A mandatory six-month internship at such companies would only serve to lower their average bar, since anybody competent would leave after the six months was up. This doesn't contradict the D/K Effect. It's easy to spot lackluster, soulless engineering organizations, and doing so doesn't imply that you're especially smart.

The "Extended Interview"

Anyway, I've often wondered where the Geoworks founders found such great engineers. The short answer is: "Berkeley", but I'm really looking for something deeper than that; namely: how did they know?

Along similar lines, I've long felt that Amazon's success was due in no small part to Bezos having seeded his tech staff with great engineers. World-class great. I don't know where or how he found them, since, again, how do you hire someone who's smarter than you? He's a brilliant guy, but his original choices (ex-Lucid folks, by and large) seem a stroke of blind luck that's hard to attribute to mere genius. I'll probably never know how it happened. Wish I did!

They weren't too big on software engineering, though, or more likely they all felt that time-to-market trumped everything else (and were correct, at least in their case), so Amazon is successful but lacks a high-quality software engineering culture. It's gotten much better over the years, of course, but it's a far cry from Geoworks. It's largely up to individual teams at Amazon to decide how much engineering to sprinkle into their coding. There's no central group (or distributed peer group) who can tell any team how to build their software. This is effectively mandated from the top, in fact.

But time-to-market is a pretty powerful business force. Maybe that's the missing link? Lucid was founded by Dick Gabriel, the "worse is better" guy, and Amazon took the "worse is better" idea (internally) to untold new extremes.

Dunno! But it sure worked for them.

Google has a process similar to the Geoworks 6-month internship idea. Geoworks's internship was a form of "extended interview", since it was obvious even in the 1980s that the classic interview format isn't a very good predictor of performance. At Google, you don't have to do an internship. However, unlike at most other companies, you're not "slotted" into a real position on the tech ladder until you've been on the job at least six months. During that time you need to prove that you can function at the level you were hired for, and if it's wrong in either direction your level is adjusted at slotting time.

The "extended interview" (in any form) is the only solution I've ever seen to the horrible dilemma, How do you hire someone smarter than you? Or even the simpler problem, How do you identify someone who's actually Smart, and Gets Things Done? Interviews alone just don't cut it.

Let me say it more directly, for those of you who haven't taken this personally yet: you can't do what Joel is asking you to do. You're not qualified. The Smart and Gets Things Done approach to interviewing will only get you copies of yourself, and the work of Dunning and Kruger implies that if you hire someone better than you are, then it's entirely accidental.

Don't get me wrong: you should still try! Don't throw the bath and baby away. Smart and Gets Things Done is a good weeder function to filter out some of the common "epic fail" types.

But realize that this approach has a cost: it will also filter out some people who are just as good as you, if not better, or even way better, along dimensions that are entirely invisible to you due to Dunning-Kruger forces mostly beyond your control.

So there's this related interviewing/hiring heuristic that I think may better approximate the kinds of people you really want to hire: Done, and Gets Things Smart.

I'll take Joel's cue and write it in bold so you know it's important. It's not condescending or anything. Really. Let's all recite it together, to make it catchy and stuff. Maybe we should have a little ditty for it, so it sticks in our heads annoyingly, forever. I think the Happy Birthday song will do nicely:

Hap-py Birth-day To Yooooooou,
Done-and Geh-ets Things Smaaaart
Hmmm hmm HMMMMM Hmmmm Hmmmm whoeeeeeever
Done and geh-eeeeeets thiiiiiings Smaaaaaaaart *clap* *clap*

There! You'll remember it every time anyone has a birthday.

Done, and Gets Things Smart

All gentle faux-condescension aside (or as the classroom jocks would have read, "fox condesomething"), Joel's Smart, and Gets Things Done heuristic seems really obvious to everyone. It has this magical "we're all smart in this together" appeal. But sadly, for the reasons I've outlined, almost everyone interprets it to mean "Carbon Copy, of Myself". Great guidance gone astray!

The Done, and Gets Things Smart approach is also a way of finding great people, but it recognizes that the Dunning-Kruger Effect requires some countermeasures. It's modeled on the early successes I've witnessed at Geoworks, Amazon, and Google, all of whom had one thing in common: they hired brilliant seed engineers. This boldface is really addictive when you get started on it!

They all managed to continue hiring great people afterwards, but the seed engineers were the most important. I'm hoping that this is intuitively obvious in much the same way that wanting smart people who get things done is intuitively obvious.

I think you really, really want great seed engineers, and that this is a different class of engineer from the "pretty darn good" engineer we typically hire using Joel's oft-misinterpreted advice.

Seed engineers. It's key. You can apply the "I want ideal seed engineers" rule recursively to an organization, a department, a project, a team, and even your officemates. We're not just talking about startups.

Let me ask you a brutally honest question: since you began interviewing, how many of the engineers you've voted thumbs-up on (i.e. "hire!"), are engineers you'd personally hire to work with you in your first startup company? Let's say this is a hypothetical company you're going to found someday when you have just a little more financial freedom and a great idea.

I posit that most of you, willing to admit it or not, have a lower bar for your current company than you would for your own personal startup company.

The people you'd want to be in your startup are not of the Smart and Gets Things Done variety.

For your startup (or, applying the recursion, for your new project at your current company), you don't want someone who's "smart". You're not looking for "eager to learn", "picks things up quickly", "proven track record of ramping up fast".

No! Screw that. You want someone who's superhumanly godlike. Someone who can teach you a bunch of stuff. Someone you admire and wish you could emulate, not someone who you think will admire and emulate you.

You want someone who, when you give them a project to research, will come in on Monday and say: "I'm Done, and by the way I improved the existing infrastructure while I was at it."

Someone who always seems to be finishing stuff so fast it makes your head spin. That's what my Done clause means. It means they're frigging done all the time.

I met my first Done, and Gets Things Smart engineers back at Geoworks. This was looong before I had any sort of a clue that I suck as an engineer, but these folks (Andrew Wilson and Chris Thomas, if you really must know) were weird. They never seemed to be working that hard, but they were not only 10x as productive as their peers, they also managed technical feats that were quite frankly too scary for anyone else. They could (as just one trait) dive in and learn new languages and make fixes to tools that the rest of us assumed were, I dunno, stuff you'd normally pay a vendor to fix. They dove into the hairiest depths of every code base they encountered and didn't just add features and make fixes; they waved some sort of magic wand and improved the system while they were in there: they would Get Things Smart. Make the systems smarter, that is. Sort of like getting your act together, but they'd do it for your code.

I've met many more such engineers along the way. They're out there. They're better than you. They were better twenty years ago than I am today or ever will be. Maybe it's natural ability. Maybe it's luck in education or upbringing. Maybe they have a secret recipe for improving rapidly and learning utter fearlessness. I don't know. But I've met 'em, and they aren't "smart". They're abso-flutely fugging brilliant.

You can't interview these people. For starters, they're not interested; these are the people that companies hold on to as long as humanly or companyly possible. The kinds of people that companies file lawsuits over when they're recruited away.

You can only find Done, and Gets Things Smart people in two ways, and one of them I still don't understand very well.

The first way is to get real lucky and have one as a coworker or classmate. You work with them for a few years and come to realize they're just cut from a finer cloth than you and your other unwashed cohorts. You may be friends with some of them, which helps with the recruiting a little, but not necessarily. The important thing is that you recognize them, even if you don't know what makes them tick.

This is the one great hope we programmers have for fighting the Dunning-Kruger Effect, the one hope we have for getting something better than the average "just like me" Solid Plugger Joe Nobody you pick up with the Smart, and Gets Things Done approach. Our only "out" is that working side-by-side with someone will show us clearly when they vastly outclass us.

Your devious little mind will come up with all sorts of rationalizations for why they're so damn good, so you can continue to think of yourself as Smart, and Gets Things Done material. You may conclude that they're just a genetic anomaly, and it's no fair even trying to compare yourself to someone who obviously has an unfair gift from the heavens. Or you may tell yourself that they're just a domain expert in various domains that you don't "need" right now. Or you may simply choose not to think about it too much. Good Old Compartmentalization to the rescue!

But working with them directly will show you when they're better. It's the only way. You'll gradually realize that your math deficiencies aren't just something that you might need to beef up on if you ever "need to"; you'll see that virtually every problem space has a mathematical modeling component that you were blissfully unaware of until Done, and Gets Things Smart gal points it out to you and says, "There's an infinitely smarter approach, which by the way I implemented over the weekend." You stare slack-jawed for a little while, and then she says: "Here's a ball. Why don't you go bounce it?"

These people aren't just pure gold; they're golden-egg-laying geese. They are the ones you want to bring with you to your own startup. Not the Smart, and Gets Things Done, Just As Soon As I Read Up On The Subject, On The Company's Dime riff-raff like you and me. No. They're your seed engineers: the ones who will make or break your company with both their initial technical output and the engineering-culture decisions they put into place — decisions that will largely determine how the company works for the next twenty years.

I've been debating whether to say this, since it'll smack vaguely of obsequiousness, but I've realized that one of the Google seed engineers (exactly one) is almost singlehandedly responsible for the amazing quality of Google's engineering culture. And I mean both in the sense of having established it, and also in the sense of keeping the wheel spinning. I won't name the person, and for the record he almost certainly loathes me, for reasons that are my own damn fault. But I'd hire him in a heartbeat: more evidence, I think, that the Done, and Gets Things Smart folks aren't necessarily your friends. They're just people you're lucky enough to have worked with.

At first it's entirely non-obvious who's responsible for Google's culture of engineering discipline: the design docs, audited code reviews, early design reviews, readability reviews, resisting introduction of new languages, unit testing and code coverage, profiling and performance testing, etc. You know. The whole gamut of processes and tools that quality engineering organizations use to ensure that code is open, readable, documented, and generally non-shoddy work.

But if you keep an eye on the emails that go out to Google's engineering staff, over time a pattern emerges: there's one superheroic dude who's keeping us all in line.

How do you interview for such a person? You can't! Everyone will tell you they're God's Gift to engineering quality. Everyone knows how to give it impressive lip service. Heck, there are lots of people who take it way too far and try to gridlock the organization in their over-enthusiasm, when what you really want is a balanced and pragmatic approach. I'd argue that it's virtually impossible to detect these "soft skills" in a classic interview setting, except to the extent that you're hiring your own clone, which according to our thesis here, is NOT what you want. You want Done, and Gets Things Smart: done faster than you, and made your system smarter!

I'm guessing that Google's founders worked with this person in school, enough to recognize his valuable talents. Hence they used Identification Approach #1: get lucky in who you work with.

Incidentally, they hired plenty of other brilliant seed engineers who were equally responsible for Google's great technical infrastructure. I'm just using this one guy as an illustrative example. But you really want the Done, and Gets Things Smart people on every team. If you could mix in one Done, and Gets Things Smart person with every five to ten Smart, and Gets Things Done people, then you'd be in good shape, since the latter, being "smart", can hopefully learn a lot from the former.

But when you're starting a company, or an organization, or a big project, the need for Done, and Gets Things Smart seed engineers is desperate. It's dire, in the sense that if you don't get the right seed people in place, you're dooming your organization to mediocrity, if you manage to succeed at all.

And it's direst when you're in a startup, because you can't pillage people from elsewhere in your organization who you know are good. And because Done, and Gets Things Smart people are worth their weight in refined plutonium, they're probably reasonably happy in their current position.

So let's assume you're looking at the vast ocean of programmers, all of whom are self-professed superstars who've gotten lots of "stuff" done, and you want to identify not the superstars, but the super-heroes.

How do you do it? Well, Brian Dougherty of Geoworks did it somehow. Jeff Bezos did it somehow. Larry and Sergey did it somehow. I'm willing to bet good money that every successful tech company out there had some freakishly good seed engineers. But a lot of company heads who make these decisions aren't necessarily industry programmers, and they still manage to find some world-class people in all the noise. So there must be a second way to identify Done, and Gets Things Smart.

I think Identification Approach #2, and this is the one I don't understand very well, is that you "ask around". You know. You manually perform the graph build-and-traversal done by the Facebook "Smartest Friend" plug-in, where you ask everyone to name the best engineer they know, and continue doing that until it converges.

I think this might work, assuming you have lots of connections initially (lots of roots for your graph), so you don't get stuck in some slummy local minimum.

I've seen companies go to university professors and ask them who their brightest students are; that's a variant of this approach, although it usually only turns up future Done, and Gets Things Smart engineers. Every once in a very rare while you'll get a recent college grad in this category, but I think more often they tend to be experienced enough to make Gandalf feel young again.

Because technical brilliance, seemingly superhuman productivity, and near-militaristic adherence to software discipline aren't enough. They also need leadership skills. They don't have to be great leaders; in fact in a pinch, just being bossy might work for a while, as long as they're bossing people in the right directions. But they need to have the ability to guide the organization (or new team, or whatever) in uniformly excellent directions, which requires some leadership, even if it's bad or amateurish leadership.

As much as I suspect Approach #2 may work, I think Approach #1 is probably more reliable. Take a closer look at your coworkers who are doing things that you "could learn if you ever need it". Read up on the old Dunning-Kruger Effect. I recommend it with irony dialed at 11, since personally I have yet to read more than the Wikipedia article and a few other articles here and there. I'll read it if I "need to". Psh.

Done, and Gets Things Smart

Not superstars: superheroes! People who are freakishly good at what they do. People who finish things so fast that they seem to have paranormal assistance. People who can take in any new system or design for all intents instantaneously, with no "ramp-up", and who can immediately bring insights to bear that are quite simply beyond your rustic abilities.

Those are the folks you want. I'm not going to tell you: "Don't settle for less." Far from it. You still want to hire the Smart, and Gets Things Done folks. But those folks have a long way to grow, and they probably have absolutely no idea just how far it is. So you want some Done, and Gets Things Smart people to guide them.

And now, to play us out...

Dooooone and Ge-ets Things Smart,
Done and Ge-ets Things Smart,
Done and Geeee-eeeets Thiiings Smaaaa-aaaart,
Done and Ge-ets Things Smart!