Egomania Itself

Q. Is it your testimony that it's perfectly okay for GNC to put products on its shelves, to sell pills to people if those pills have no benefit? A. We make the assumption that they make the decision based on some perceived benefit. — GNC board chairman Jerry D. Horn, in sworn deposition

Whew! What a crazy ride the last 2 weeks have been: I was Joel'd, Bray'd, Slashdotted, featured in Agile Mafia Monthly, and completely buried in an avalanche of email from Alert Readers.

The highlight of it all was probably the invitation I received to speak at an Agile conference in Vancouver, B.C. They told me, and this is a direct quote, "we promise not to cut your balls." They then went on to reassure me that just to be sure, they'd get me a security guard. Yikes!

No doubt the Agilawyers have been scrutinizing my Tyler Durden clause, seeing if they can find a loophole and take me up on it early. Ha! I won't be taken so easily. "A fool and his balls are soon parted," as the old saying goes. More or less. So I declined politely, filed for a legal name change, got on the list for facial reconstructive surgery, and moved to Oklahoma. Can't be too careful when your you-know-whats are at stake.

The second runner-up was definitely this email, which I've reproduced in full:

From:  Agilists Carnival

If you've received this email, you've been referenced in the
October 5th Carnival of the Agilists.  Thank you for your
insight and commentary on how to make Agile software
development as good as it can be.

If you like, please consider linking to the carnival from
your blog, to help spread the word.  And, as always, if you
would like to help out with the Carnival, either as a
rotating host, or as a one-time guest host, please feel free
to contact me at: <email withheld> and let me know.

Thanks again!
    J.

Hee hee. Looks like someone didn't get the memo.

Anyway, because my last blog entry was soooo read and skimmed and linked and contested and grokked and misunderstood and praised and denounced and everything, I figured I'd break tradition, buckle down, and actually write a follow-up.

Incidentally, I can't even begin to tell you how happy it makes me to break all the stupid-ass grammatical rules my Agile Grammar Teachers forced on me in grade school. They represent a brittle, method{olog}ical personality type that's still alive and thriving today, busy certifying scrum masters around the world, painting their cats, all that good stuff. [We'll get to the cats in a little while, in case you were wondering.]

OK, enough preambling already. On to the main show! I promise you many mean jokes in tonight's entertainment, an insight or two, and if you're a good audience, I've got a bona-fide magic trick for you at the end. Hang in there!

There's More Than One Good Kind

Before we get into the interesting stuff, I thought I'd clear up one major misconception that kept coming up.

Some people apparently thought that by using Google's development practices as a counterexample, I was implying that Google's approach is the only alternative to (a) Agile, (b) Waterfall, or (c) Nyarlathotep.

Not true! I was merely using Google's process to illustrate that alternatives do exist and they do work. I offered this evidence largely to combat the Agile claim — and I'm paraphrasing loosely here — that "if you don't forward this Agile Manifesto to 5 friends within 47 seconds, you will die instantly and then be thrown off a high building into a pile of manure."

Seriously, though: you don't need to look at Google to discuss life beyond Agile. If you just look around, you'll find plenty of "success stories" from non-Agile teams. Looking for "success stories" is, of course, stupid; it's exactly the kind of selective reinforcement that can lead to (e.g.) chronic gambling disease. If you want real supporting evidence, you should construct the scientific kind, using experiments. Yes, I'm afraid that means looking at the failures too. Ouch! So bad for marketing. Whenever you hear Agile people asking around for "success stories", remind them politely that only looking at the positives is pseudoscience.

If you walk around and do a survey for yourself, you'll find two dimensions producing four categories: Agile vs. non-Agile, failure vs. success. You'll be able to find projects that fit into each category. Don't focus on the "success stories" — look at the whole picture, and correlate your findings with those of friends from other companies. Most of you will find that if you did an honest assessment, Agile's not very widespread, and it's not as success-prone as they'd have you believe. Not by a long shot.

As one concrete example, one commenter observed that Blizzard doesn't ship on a schedule; they ship "when the app is done". Well, raise your hand if you haven't heard of World of Warcraft. Hands? Anyone? Didn't think so. WoW broke into the crowded and locked-in gaming space (people like to stay on their current MMORPG), and stomped the living crap out of huge competitors like Sony and Microsoft. Overnight. Boom. And Blizzard is in a business that's as about different from Google's as you can get — especially their shrink-wrap games, which have also historically been groundbreaking, award-winning, runaway bestsellers.

But rather than enumerating all the non-Agile teams and companies as special cases, let's get straight to the rule: Most great software developers around the world don't use Agile. They just work hard, they stay lightweight, and they ship great stuff. Most developers still have no idea what Agile even is. Think of that!

Agile is a niche, a market minority, almost an aberration, really. It just happens to have a lot of marketing, because it opened up a hitherto entirely untapped new market for snake oil in the tech sector. Consultants are making money hand over fist by extending their contracts with credulous clients who are told they don't have the process "quite right" yet, but they're almost there!

Agile wouldn't be a big deal, except the Agile camp is really loud. Annoyingly so. Loud enough to start interfering with regular developers' work. Which is why I had to speak up. Agile was the Mystery Topic that gave me Blogger's Block for nearly 2 months. At some point, though, I just couldn't take it anymore. There were critics here and there, sure, but none of them were as loud as the Agile folks.

So I yelled as loud as I could: loud enough to make it onto Slashdot, even! Which is exactly what I wanted. I only had one high-level takeaway, one marketing message I wanted to get out to developers everywhere: It's OK to say No to Agile. That's it. Nothing more complicated than that. But the Church of Agile was getting so powerful that it was becoming increasingly acceptable to criticize people in the workplace for not being Agile. More on that in a bit.

As it happens, getting my broadcast message out wasn't too difficult, on the whole, because snake oil's kinda funny. It's an easy humor target.

You've probably noticed by now that the Agile folks are utterly lacking in humor. After my last blog, the whole Agile community started maneuvering and positioning and strategizing — and I'm talking about the smarter ones. The others just screamed "I lost thirty-five pounds on that diet!", demonstrating that they might possibly have missed my point about experiments vs. statistically meaningless anecdotes. Can't blame 'em; it was a lot of information to process: a whole book squeezed into a chapter-sized blog. But across the board, there hasn't been a whole lot of self-deprecating humor on the Agile side.

Well, any time you find a community whose overall Humor Level is rated at "Homeland Security", it's a pretty good indicator that you don't ever want to have to deal with them. Ever.

In any case, the whole discussion about whether Google's approach is viable for tech companies in other domains is a red herring. Most companies don't use Agile Methodologies, or if they do, it's only a handful of teams, maybe 10% or fewer, I'd guess. At least it's true at the companies I know lots of people from - Sun, Microsoft, Yahoo, Amazon, Google, Blizzard, and other places like them: industry leaders who write kick-ass software. They do it almost entirely without Agile. It's not just Google. It's everyone.

Agile folks are so damn loud, I mean I swear they ought to get some sort of disturbing-the-peace ticket, that they'd have you believe that even if they're not a majority, they're some sort of moral majority.

I think you know better by now.

Static Typing and the Need for Reassuring Metadata

A few people caught a rather subtle point about static typing I made towards the end of my good/bad agile rant. I didn't dwell on it, but all the follow-up cries of "well what should we use, then?" have got me wondering whether it's actually more central than I suspected at the time.

I'm going to share some apparently unrelated short anecdotes. They mesh, though. Bear with me.

I had a high school English teacher who, on the last day of the semester, much to everyone's surprise, claimed she should tell just by looking at each of us whether we're a slob or a neat-freak. She also claimed (and I think I agree with this) that the dividing line between slob and neat-freak, the sure-fire indicator, is whether you make your bed each morning. Then she pointed at each each of us in turn and announced "slob" or "neat-freak". Everyone agreed she had us pegged. (And most of us were slobs. Making beds is for chumps.)

As I've done for a great many other programming languages, I've bashed on Perl's technical weaknesses at length in the past. To my continued amazement, the Perl folks are the only ones who never get upset. They just say "Haha, yeah, boy, you're right, it sure is ugly. Heh. Yeah, so, um, anyway, I'm going to get back to work now..." It's awesome. I've gained so much respect for them. It's almost enough to make me go back to programming in Perl.

Well, almost.

The Perl folks have this Perl Haiku competition each year. It's a nifty idea, and it's pretty amazing that you can write useful seventeen-syllable programs.

I tried it with Java once, and produced a valid Java haiku:

ArrayList<int> myListOfInt = new ArrayList<int>();

which, spoken aloud, reads:

ArrayList of Int
my list of int, equals new
ArrayList of Int

Maybe it's just me, but I find it pretty farging depressing that a simple declaration like that can be haiku-sized in Java.

The noted Far Side cartoonist and computer scientist Dr. Gary Larson published the definitive paper on Java-style static type systems in October 1987. I've taken the questionable liberty of reproducing his paper here, in full, sans permission, in the hope that it'll fall under "fair use" if enough of you buy a Far Side book. If you don't, well, just remember what happened to Miranda Pinsley!

Here's Dr. Larson's renowned treatise on non-inferring static type systems:

It's truly one of the most insightful Computer Science papers ever published. Yet what Dr. Larson may not have realized at the time was that he was also predicting Agile Methodologies!

If there's one thing I've learned over the years about type systems, it's that you have your slob type systems and your neat-freak type systems, and it comes down to personal preference. The neat freaks (Java, C#, C++, Pascal) know damn well that the slobs (Perl, Python, Ruby, JavaScript) are just as productive. Maybe even more so. Nobody really knows for sure. One thing is clear, though: everyone's getting stuff done, regardless of their language choice. The whole debate isn't actually about productivity at all, even though most people still think it is. It's about whether you can stand to live in a house where the bed isn't made.

Did I mention making beds is for chumps? Well, that's just my take on it. My wife makes our bed almost every morning. I find it annoying and useless, especially since we have to tear the thing apart again every night. But she has to have the place spotless or she gets really uncomfortable. So, you know, I have to pick up my socks and all that stuff.

There will always be slobs, and there will always be neat freaks. There will always be programmers who have to paint "THE CAT" on their cat and "THE DOG" on their dog, metaphorically speaking, or they'll be as stressed and uncomfortable as my wife is when I've left a bunch of dishes in the sink and clothes on the floor. And there will always be programmers who need a methodology: even if you can convince them that their current one isn't helping, they can't just drop it. They need to switch to something else.

And that's perfectly OK with me, as long as they understand that it's not some proven silver bullet — it's just a style thing.

Unfortunately, Agile has been propagating like a virus via a slightly buggy survival mechanism built into all our brains, every single one of us. You have to take desperate measures to combat a viral epidemic, so I'm going to pull out the really big guns, and tell you about my dad's chili.

Making Chili

Many die-hard Agilytes have angrily retorted that Agile has been working for them for years, that they've been successful with it, and so on. They seem to think I'm claiming that Agile "doesn't work". A few even thought I was claiming that Agile projects are unsuccessful 90% of the time!

Well, make no mistake: Agile works (or at least it can work) just fine. The problem is much more subtle than that; most of you appear to have understood it no problem, but many Agile folks missed it entirely. So allow me to try being un-subtle, and see if that gets the point across to them any better.

When I was a teenager, my dad and my brother Mike decided to make homemade chili. I'd never seen it made before, and I watched with keen interest as they added beef, beans, some veggies and spices, and other ingredients. Dad would taste it, add some more ingredients, wait a bit, taste it again. My dad has some pretty good recipes. So you can imagine my puzzlement when he opened the cupboard, pulled out 2 cans of Hormel chili, opened them and dumped them in.

I waited a respectful moment or two before asking him why he was adding canned chili to his chili. They both said it tasted terrible, but, as my dad now-famously observed: "You can start with dog shit, and if you add enough chili, you get chili."

Similarly, if you start with an Agile Methodology, and you add enough hard work, you get a bunch of work done. Go figure.

But that's a tautology; you can substitute anything you like for "Agile Methodology" and it's still true. It's probably not difficult to find people who believe that Feng Shui has brought them success in their projects for years. Or throwing pennies in fountains. Heck, there are probably some people who practice witchcraft to help their projects out, and a great many of those projects — probably the majority — wind up being successful.

If you do a rain dance for enough days in a row, it will eventually work. Guaranteed.

So I'm not saying Agile doesn't work. It does work! But it's plain, unadulterated superstition.

And that, dear audience members, brings us to our last big topic today. Let's start prepping for that magic trick!

The Break-Dancing Chicken

Oh yeah. First I have to tell you about the chicken. We've done the cats, we've done the chili, we're on to the chicken. Almost ready for the final act, which is really gonna piss a lot of people off, I can tell you that right now.

In 1986, my college Psychology 101 professor told us a little story about her Abnormal Psych class from the previous year. The students each got a lab rat, and they had to train their rats to run through mazes by rewarding the rats for exhibiting properly ratty maze behavior. But there was one girl who was terrified of rats, so they gave her a chicken, and she trained it to play the piano by pecking notes with its beak. Each time the chicken played the right notes, she'd give it a pellet (or whatever the hell chickens eat). Wrong notes, no pellet. She'd teach it one new note at a time, and eventually it would know the whole song.

At one point, after she'd coaxed the chicken to learn maybe half its song, the chicken made a violent side-to-side motion just before playing exactly the right sequence of notes, including the new note she was trying to teach it. It had played the right notes, so she had to give it a pellet. The chicken had definitely figured out that doing what she wanted would get it yummy pellets. After that last pellet, it decided that it had been rewarded for both the note and the motion, and from then on, it made crazy twisting motions continuously as it played. She had to keep rewarding it when it played the right notes, and she had no good way of informing it that the twisting was unnecessary (not without starting over, and there wasn't time). So the chicken went on happily believing that thrashing violently was helping its project succeed.

They eventually went on tour to the local colleges with it, billing it as "The Break-Dancing, Piano-Playing Chicken".

Our prof told us wide-eyed freshmen that this phenomenon is called superstition, and it refers to the exact same kind of superstition you think of when someone mentions black cats, walking under ladders, breaking mirrors, opening umbrellas indoors, and so on. 'Superstition' is any belief not based on scientific experiment or pure deductive reasoning. (More or less.)

Well, geez, that seems pretty broad. We don't bother to go verify most of the stuff we know (or think we know) — does that mean we're all really superstitious?

It turns out we are. See, our brains are powerful pattern-matching machines, and in order to survive and to learn about the world we have to be able to build up millions of inferences about our experiences, and then match against that "pattern database" in real time. In order to process all that information fast enough, our minds have a tendency to assume that if two events are coincidental, they're also correlated.

That last sentence was probably the most important in this whole rant, so I'll restate it, in case you were dozing off at my boring lecture. Don't do it! We have a magic trick coming soon. You won't want to miss it!

Restated: when you see two events happening that appear to have a cause-and-effect relationship, then your brain naturally and automatically looks for reinforcement. Also known as "success stories". The chicken, completely by accident, got the right notes at roughly the same time it twisted. Then it received a pellet, and from then on it believed increasingly that the twisting was one of the things that "caused" pellets to appear. The belief was only reinforced from then on.

That's it. That's how superstition happens. And wouldn't you know it, most of the time it's right. That's why it works so well as a pattern-formation strategy. You touch the stove, you burn your finger. Do it again, sure enough, you burn it again. Cause and effect. But just because you've drawn the correct inference doesn't mean it's not superstition! You never really know if it's a "fact" until science has promoted it. Getting things promoted from superstition to accepted fact is the fundamental endeavor of all science.

Ironically, the chicken story is an Urban Legend. I have no idea whether my prof was telling the truth when she said it was "her" Abnormal Psych student who trained the chicken. Maybe she said it was another prof, and I just remembered it as being her class. I suppose I could go dig up the details (with a LOT of effort; this was 20 years ago) and possibly find out whether it's true or not. Until then, it's an Urban Legend.

That doesn't actually mean it's not true. "Urban Legend" doesn't refer to the message content; it refers to the transmission protocol. As it happens, many urban legends are true, if you believe Snopes.com at any rate. What makes them urban legends is that they're passed from person to person without keeping any of the original details correct about who, where, when. So even if they're true, the way you heard about them makes them superstition. Until they've been verified (if necessary, by repeatable experiment), that's all they are.

So superstition isn't necessarily a bad thing! We couldn't possibly go and verify every single fact we've heard that we think is probably true. We'd never make any progress (as individuals or as a civilization). We have to take most things we know for granted. Most of what we know at any given time is superstition. It's normal.

And of course many of our superstitions are transmitted to us from other people, through virus-like propagation. One propagation type is the urban legend. Another one, sadly, is marketing. One way or another, ideas make their way to us, and our tendency is to believe anything that seems plausible.

As I was thinking last week about what I'd write about in this essay, I Googled for "technological superstition" and came across this lovely little ACM article by Jef Raskin from 2003. In it, he talks a little about the psychological foundations of superstition, and goes on to poke fun at a popular and widespread superstition, namely that expensive audio cables sound better than cheap ones. It's definitely worth a read, and it has a little jewel at the end that I'm sure will make you gasp with appreciation for the author's Nostradamus-like predictive powers.

It's all fun and games until someone loses a testicle

Superstition is harmless, right?

Well, sort of. Mostly. Usually. In fact, the point of the previous section was the superstition is not only inevitable (for all of us), it's actually a critical brain function. Most of what we believe is superstitious, and if we really want to, we can go track down any particular belief and do some reference-checking on the sources (e.g. is the source Wikipedia or the National Enquirer? Both are sometimes right and sometimes wrong, but the confidence levels differ dramatically) or even repeat the experiments ourselves.

Superstition is strongest when people desperately want to influence or control events around them, but it's too hard for them, so they turn to the magic of wishful thinking: Feng Shui, pennies in fountains, shooting stars, Agile Methodologies.

Nothing wrong with that.

The fundamental cultural problem with superstition arises when people don't know how to differentiate between reliable and unreliable data-verification sources, so they treat non-science as science. If they don't recognize or admit that they're being superstitious, then they'll feel no qualms at all about trying to propagate their beliefs to YOU.

In fact, if they can do it successfully, if everyone believes something, then it's no longer superstition. (This is why philosophers wonder whether even scientifically accepted facts are still just superstition. Quantum mechanics has revived that line of inquiry in the last century.) That's why they want you to believe what they believe: it reduces the nagging doubt that they might be wrong. It's why superstitious communities spring up and clump together and get all insular — they self-reinforce within the group.

This, folks, is what's been gradually going wrong with Agile over the past decade. It's a complicated problem. And it has NOTHING TO DO with whether it "works" for you or not.

The problem is that if it kept spreading, it was eventually going to be a majority opinion, at which point the non-believers would start losing their jobs because they weren't playing along with the Establishment. Cowboys, indeed. Now you know why Agile folks call them Cowboys. They want to brand us all as rogues, as non-team-players. Scary, eh?

I don't want to have to wear an armband just because I don't share their techno-religious beliefs. I'm already wearing enough of them as it is. The Church of C++, for instance, is a physical and moral majority, and you're a lesser programmer if you don't use it. Get used to it. You live in an unenlightened age.

As far as I'm concerned, Agile Methodologies are fine for you to use, but if you try to push them on a coworker, it should be treated as an HR violation, just as if you were discriminating against them because they don't believe in Feng Shui.

If it's a manager pushing Agile on a subordinate, it should be treated with extra severity. Reprimand, goes on the employee record, mandatory training, and 3 strikes you're out.

You have to fight fire with fire. I want it to be their career problem before it becomes your career problem.

I joke about it a lot, and I'll continue to do so, but this is actually pretty serious stuff.

But let's go ahead and close out on a lighter note. I promised you a pony, right? No, wait, it was a magic trick. That's right. Sigh. All right, then. You asked for it, you'll get it!

Anagrams and Hidden Meanings

Here's one laaaaaaaaast superstition for you: anagrams! Ooooh, they're one of the most popular of all. You rearrange the letters of a word or phrase to see if you can find hidden meanings. That's all there is to it!

Unless you're truly superstitious — and I mean the Old World kind, where you still believe in crazy stuff like leprechauns and numerology and a steroid-free Olympics — then you know that anagrams don't really have hidden meanings in them; it's just random coincidence.

I mean, let's face it: if you look hard enough, you can always find some phrase describing your intended target that also has a really unfortunate anagram. It's just blind (usually bad) luck. I think Spiro Agnew knows this better than anyone in history.

And yet... sometimes it's soooo hard to believe it's really random. When I was in my mid-twenties, I ran my full legal name (Stephen Francis Yegge) through an anagram solver to see what came up. Ouch! I mean, here I was, struggling with my weight, and all the meaningful anagrams of my name seemed to be about being fat. The worst one was "piggery, hence fatness". Gosh. Was I cursed? It sure seemed that way.

Meanwhile, my little brother Dave was all pissed off. "Why are all mine about being sick? This is lame!" I remember it like it was yesterday. "David Francis Yegge" (my parents were so creative) came back with anagrams like "gas acid fever dying" and "dying grief cave sad", and Dave was NOT happy about it. Hidden meanings my ass, he announced, and he stopped playing with the program. I don't think he ever used one again for the rest of his life.

Are "hidden messages" like these a coincidence? Yeah. Yes. They are. From any rational, sane perspective, they're just sheer bad luck. Problem is, our brains sometimes just don't want to be rational.

I'll close with one last example, and it's a dirty trick — one that would be utterly beneath a more principled blogger than myself. Because after all my ranting about superstition, I'm about to use superstition for my own nefarious purposes.

Let's say, hypothetically speaking of course, you were to rearrange the letters of — oh, how about Agile Manifesto, just for pure curiosity's sake. An innocent exercise, nothing more. ("Hey, it can't hurt! It works for me! I've been rearranging letters since 1990!" Dorks.)

Well, chances are pretty good that you'll find lots of meaningless and ho-hum anagrams, and possibly one or two moderately interesting ones.

But if you're hoping for a really clear message, especially with bigger words in it — you know, something that would be unmistakable as a hidden message — you know you're likely to be disappointed.

So let's try it. Can't hurt.

How many 2-word anagrams of "Agile Manifesto" do you think there are?

Guess what? There's exactly one:   Egomania Itself

Yup. I thought the exact same thing you just did. :-)

Sure, you can argue about probability and statistics and the fundamental nature of randomness, and you can cry foul play! no fair! poor sportsmanship! low blow! until you're blue in the face, and you'd probably be right.

But in the end, I mean, come on, you have to admit, it's a pretty spooky coincidence...

(cue haunted mansion bat music)

...or IS it?

(cue happy end-of-show music)

Tune in next time! I think I've pretty much finished up with Agile for now; I've stuffed garlic in its mouth and pounded a wooden stake through its heart and it's making melodramatic retreating gestures, shaking its fist at me and promising revenge in the inevitable sequel.

So now I can go back to writing about boring stuff that won't make me infamous. Like, you know, Google.

Hope you enjoyed the show!