Tuesday, December 19, 2006

Parabola

The airport security line wound its way like an amusement-park ride through Terminal C. The line was moving caterpillar-like, unhurried.

Some of the people in the line fidgeted and stared every direction, measuring the caterpillar's progress by the big clock on the wall. These were the passengers whose flights were leaving soon. You could read each passenger's face and in an instant know how desperate he or she was. The desperate ones plainly wished the big caterpillar would hurry up.

But most passengers, like the caterpillar they formed, shuffled along with that particular type of stoic boredom you only find in airport security lines and waiting rooms at the Department of Motor Vehicles. Most of them simply switched off, and did their best to leave their bodies behind while the minutes ticked by.

A little sign by the velvet-roped entrance told arriving passengers the wait was 45 minutes.

At this moment the line was in perfect working order, and nobody was more contented by this than T.S., the big security guard at the caterpillar's head. T.S. had worked in this airport for twenty-two years, doing exactly the same job, and one of the great satisfactions of his life was that each year things became more orderly. There were rules and laws and airport regulations governing every conceivable aspect of the operation of his security-line.

Whenever an inconceivable thing happened, which is to say that some sufficiently clever or desperate passenger found a hole in the rules and made things temporarily disorderly, they'd have a new rule for just that situation by the very next day. After more than twenty years of such additions, the regulations manual was thick and heavy with special cases.

T.S. had memorized every rule in the thick blue security-line manual. He knew all the rules because he had helped make most of them, and his job was to enforce them.

Years ago the rule book was just a tiny thing. It was the kind of rule book you'd write if you had grown up watching black-and-white movies in which people were mostly good, honest, and wholesome. The early rules were just common sense. But over the years, after people had tried to get on the plane naked, or had shown up drop-dead drunk and vomited on other parts of the caterpillar, or had tried to sneak their fireworks onto the plane for the big Fourth of July celebration, the rule book had become massive and daunting.

T.S. needed a special rule for each possible way to break the orderly workings of the line, because otherwise people would argue endlessly. You had to show them that it was written down, a real rule, before they'd listen.

On most days the big rule book was sitting on a black shelf upstairs by the coffee machine in the Terminal C security office. T.S. didn't need it. He was a walking, talking version of that book, and passengers knew this without needing to be told. T.S. looked like a rule book. Plus the rules were still mostly common sense, so the passengers all knew the important bits. You stood in line until it was your turn to show your pass and ID, and then the guard would let you go take your shoes off and put your things on the conveyor belt. There wasn't much to it, if you behaved yourself.

Most passengers breathed a deep sigh of relief once T.S. waved them through the gate. They might think it was due to the sudden relief from the boredom of being a caterpillar segment, but it was more often because they had been holding their breath while T.S. was examining their paperwork. He looked a little scary. After you got past T.S., you didn't have that warm feeling of being one of the good guys. You felt like you had narrowly escaped being in trouble, and that you were really hiding something, but you had escaped detection this time. Somehow T.S. managed to make almost everyone feel guilty.

T.S. surveyed the line about every seven seconds, the way you might glance habitually in your mirrors while driving. He could read the line in an instant, could spot potential troublemakers before they even knew they were planning to make trouble. T.S.'s job was to examine the boarding pass and identification for each passenger, but he also had broad responsibility for keeping the line moving and orderly.

Nobody got past T.S. without the right credentials. No exceptions.

A dark-haired woman arrived at the back of the line, breathless, towing a young boy who might have been her son. The woman was wearing a dark blue dress covered in nice white dots, with small ruffles on the shoulders. She had no luggage: just a black handbag in one hand and the boy in the other. There were hundreds of people in the line, and others walking around the terminal, but T.S. began watching the dark-haired woman from the moment she rounded the corner. He knew she was going to be disorderly. It was on her face, in her wide eyes. The only question remaining was how long it would take him to quiet her down and get her to stand in the line like everyone else.

The woman and boy were making their way up the shortcut lane, the one reserved for airline personnel. T.S. had seen this play out a thousand times, and not one time in a hundred did the passenger actually have the right paperwork for being in that lane. But troublemakers always wanted to use the shortcut lane. They always wanted to talk to him. They didn't understand how the system worked. Troublemakers always thought they were special cases.

The woman and the boy made their way up to T.S. and waited for him to finish with the current head of the caterpillar, a family of four apparently headed for a ski trip. T.S. ignored the two newcomers in that special way only a career administrator can muster, by suddenly taking a keen interest in his own work. He smiled kindly at the father of the family, looked again at their four passports and boarding passes.

The father of the family smiled back nervously and held his two children very still. His wife smiled as well, wide-eyed and alert. They knew they had been made lucky by the sudden appearance of the dark-haired woman. The big security guard with the initials T.S. on his name tag was going to take a little longer with the family, but it was only a gesture. The guard had to show everyone in line that people who abuse the shortcut lane have to come away with a clear net loss of time. This would necessitate a more leisurely inspection of the ski-trip family, who knew they had just been granted a temporary immunity from T.S.'s suspicion. T.S. would treat them almost politely, as long as they played along. This was one of the rules that wasn't written down, but everybody knew how it worked.

The other passengers in line began to watch closely, with interest. There was clearly going to be a showdown. That big security guard hadn't smiled once in the last 45 minutes, and he was only doing it now to show just how thoroughly he was ignoring the woman and child. The crowd wanted the dark-haired woman to lose the showdown and slink to the back of the line, because it was clear she didn't have one of those magical Shortcut Lane passes that nobody ever gets. She wasn't acting confident enough to have one of those. She looked nervous, almost sad, like she knew she'd never make it through.

T.S. addressed the father for the first time. "Ski trip?"

The big guard asked his question curtly. It was a ritual. He was permitted one question for every few passengers. This was another Unwritten Rule. Sometimes security guards had to ask questions, because you could find bad guys that way. It happened in the movies all the time: a first-time bad guy always spilled his guts if a cop asked him a stern question. First-time bad guys are always nervous in the movies. So even though T.S. was technically only supposed to check credentials and either wave people through or send them back to the ticket counter, he could get away with asking spot-check questions now and again, as long the conveyor belts behind him never starved for bits of caterpillar. And he was using up one of his free bad-guy questions on the family, to make a point.

The father knew the game and played it gladly. "Yep. Aspen. We can't wait." Everyone watching knew that he was really saying: "Please shoot that dark-haired woman instead of me. She's one of those Shortcut-Lane bad guys." Everyone knew T.S. would nod and smile and let the family through then, so that's just what he did.

T.S. was feeling momentarily magnanimous. "Have yourselves a good time, y'hear?" The family gratefully took their papers and squeezed through the gate and into the x-ray area beyond.

Now it was time to deal with this troublemaking woman. The caterpillar stopped shuffling as everyone strained to watch.

"Can I help you, miss?" There was no reason not to be polite. Not yet. T.S. prided himself in his fairness to troublemakers.

The dark-haired woman was clutching a handwritten note. She presented it to T.S. "Yes, sir. I have this note from the manager at the ticket counter. She said to show it to you, and said we should go through the shortcut line."

T.S. didn't bother taking the note. Handwritten notes never, ever constituted valid credentials. He didn't even look at it. He made a dismissive turn back to the new head of the line, and replied: "I'm sorry miss, but without a lane pass, you need to stand in the regular line. Like everyone else," he added pointedly, nodding at the good people forming the long caterpillar. They were all good people now. T.S. was feeling generous today. Restoring order always made him feel grand.

The boy with the woman started to look very worried. "Mom, are we going to make it? Are we going to get to see Dad?"

The woman pulled the boy closer to her, and she gave the second speech she had rehearsed on the way to the guard. "Sir, the ticket counter is all out of lane passes, but the agent at the counter and her boss said that you could call them if you had any questions. My son and I need to get on this plane, and it's leaving in less than ten minutes. They wrote this note for me. They said it's a temporary lane pass that I should give to you."

T.S. was unmoved, and turned his back to the woman and her son, taking the next passenger's boarding pass. He would keep talking to her, for a little while anyway, but it should be clear to her that the conversation was over and that she would be leaving very soon. The people in line were on his side. The ticket counter agents might try to help a late passenger, but they could only do so much. The woman was just unlucky today. They were out of lane passes, so she was going to have to wait in line. The showdown was over.

The dark-haired woman played her last card. "Please, Sir, listen to me. My husband has been in an accident. The Red Cross notified us this morning. My son and I need to see him, we need to see him. They say my husband is in critical condition, and that he might..." she broke off and bit her lip, looking down at her son.

She considered for a moment. T.S. nodded the next person through and turned back to her.

"We need to see him very soon, and the airline counter contacted the Red Cross and got me set up on this flight. It's leaving in 10 minutes. They gave me this note to give to you. Please take it. We have our boarding passes and identification. If you don't let me through, I may not get to see my husband... today," she added quickly, hugging the boy.

The passengers looked from T.S. to the woman. Her revelation was very serious news. Most of the waiting passengers were willing to accept her story at face value. You don't make up a story like that. Not when you're a woman with no luggage and a small boy, not when you've offered to call the ticket counter over. This woman was clearly an exception to the rules.

That was another unwritten rule: all rules can be bent. Everyone in line knew it. This was one of those times. The caterpillar leaned forward, towards the guard, urging him on. Without speaking, they let him know it was time for him to act. He had lost the showdown after all, and they all still had their planes to catch.

T.S. looked the dark-haired woman up and down. She was clearly in a great hurry and had not taken any great pains with her cosmetics today. But she was in fact quite beautiful, with bright eyes, long dark lashes, and long curls. T.S. guessed she might have been of Spanish or Italian descent. He had never been to Spain or Italy, and could not have told you which one was bigger or closer to the United States.

The dark-haired woman was neither Spanish nor Italian. She had been born in Bangalore, India, and she had come to the United States and, against all odds and expectations, had fallen in love with a Pakistani man, and they were married with one son, the boy at her side. Both she and her husband were computer programmers. They were brilliant, and they were fiercely devoted to each other.

T.S. could not have named three differences between India and Italy. They were as alike to him as Austria and Australia. They weren't in his airport and they weren't in his rule book. His mind raced, as evidenced by a slight slacking in his jaw.

T.S. knew he was in a bind, because he sensed that the woman might actually have gained some credibility with the people in line. Her story didn't matter to him in the least bit, and he didn't stop to consider whether he believed her. His concern was the caterpillar: his audience. T.S. viewed himself as more than a gatekeeper and a peacekeeper. He was the big show, the main event for the segmented bug creeping its way through the velvet ropes. They were a captive audience, but a real audience nonetheless. T.S. believed that the line looked up to him for protection, and feared him a little as well, on account of their all being potential rule-breakers.

So what should he do about this woman? He didn't have any way to call the ticket counter, and there was no protocol for it in any case. Calling the ticket counter would hold up the line. The overriding rule was that each segment of the caterpillar needed to conduct its business in a few seconds or be turned away. His audience ought to know that. It's another Unwritten Rule. But they suddenly seemed sympathetic to this lane-hopper woman with her scribbled note, which was probably a forgery. It all should have gone smoothly, but the woman with the dark hair and dark dress was unexpectedly articulate, and she had somehow swayed the onlookers in just a few short breaths.

He opted for faux sympathy. "Ma'am, I do sympathize with your situation, but I'm not in a position to call the ticket counter." He took the next passenger's boarding pass, to show that he didn't sympathize so much that he'd stop the line for her, and continued: "Now, you have two options."

Giving them Options was generous. It also let T.S. show off his profound knowledge of the Rules. When disorderly passengers were given Options, they usually understood that he was giving them a way to back down without losing too much face. It was OK for them to be ignorant of the rules; only a professional like T.S. could be expected to know them all. Presenting Options permitted a troublemaker to thank T.S. for the information - loudly, for the benefit of the caterpillar crowd, so they'd know no face had been lost. Then the troublemaker could pretend to go off and take care of things, before slinking to the back of the line. It was win-win.

"Your first option is to bring me back a genuine Lane Pass. I'm afraid I can't accept any handwritten notes. One of the other airlines' ticket agents might have one that your agent can borrow." He was warming up to the explanation. His generosity would win the crowd back, the woman would leave with her dignity intact, and he would be King for the next 45 minutes. If only there were some way to video his performance for the benefit of people not yet in line.

"Your second option is to stand in this line. If nobody holds us up," he paused a moment, giving his implication time to sink into the caterpillar, "the line is moving along pretty quick, and I expect you've got a good chance of making your flight and seeing your..." He struggled, having forgotten the details of her story. "Seeing your friend."

The caterpillar recoiled in horror, and T.S. knew he'd made a terrible mistake. He remembered and quickly added: "...your husband, I mean." But the damage was done. All semblance of sympathy was lost. He had alienated his audience utterly. It was a good thing he wasn't on video; this would all be forgotten in 45 more minutes.

The woman looked desperately at the people in line. "Is anyone here a programmer?" she asked. Time was short.

A heavy woman standing in the second fold of the caterpillar answered the implied question: "Have you thought about maybe using the debugger? Sometimes they have a back port you can connect to. He might have a bypass flag."

The dark-haired woman considered for a moment, then turned to T.S., who had delivered his Options speech and was now ignoring her. She pulled out a long metal rod with some red and green LEDs built into it. It had a thick cable attached, leading back into her handbag. "Sir, is there any chance you might be able to bend over for a second and, um, let me put this debugger in your... back, ah, port?"

T.S. had an idea what she meant, and his face broadened into a wide, wolfish grin. "Lady, you try a stunt like that, I might actually get to use this here nightstick for the first time in seven years." He patted the black baton at his hip.

A bronze-colored man with a huge dark mustache near the front of the second bend chimed in: "I don't think you can attach the debugger to these systems in production. They've talked about it, but there are some performance and stability issues."

The dark-haired woman put her rod away. She was now exhausted and on the verge of tears. T.S. continued to wave people through, and a teenager who had just rounded the last bend suggested: "Have you thought about just rebooting? You could change his bypass logic and recompile."

The woman behind the teenager, a portly woman with short spiky blonde hair, immediately shrieked: "How about I reboot YOU, you rotten skunk, and see if YOU come back up? I'm almost through this line and I'm not about to have to come all the way through that parking lot again, go through that ticket line again, and stand in this damn line again! No rebooting, and that's final!" The teenager hung his head, abashed.

The bronze man touched his mustache and offered tentatively: "He's written in Java. Have you thought about using reflection? You might be able to find a way through that way."

The dark-haired woman nodded and sighed. "Thanks. I'm Anushri, by the way. I really appreciate your patience." She said this to him and the others in line.

The bronze-colored man replied: "I'm Delmore. Good luck to you. We'll wait." The rest of the caterpillar nodded their agreement. This woman, Anushri, she had to get on that flight. It really was life or death. It wasn't a laughing matter. There had to be a way to make an exception for her.

Her son was now very worried and looked like he would cry. "Mom, are we going to make the flight?"

Anushri stooped down and hugged her son, and said to him: "We are going to try, Bibi. I'm sure your father will be so happy to see us! But I must hurry and convince this guard to let us through, or we will miss this flight and we will have to find another."

She stood abruptly and faced T.S., who was shuffling his lost audience through the gate as quickly as he felt he could manage safely. Her voice took on a tone of command. "What are your methods?"

T.S. snapped to attention and turned to her with eyes unfocused. He recited mechanically: "checkCredentials, denyAccess, getVersionString, grantAccess, setSecurityLineState, travelHome, travelWork."

A young teen in a white t-shirt near the back of the line shouted: "man, that's the lamest crap-ass API I ever heard! Who the hell designed this loser?"

Another teenager yelled: "I think he forgot reeeeadNuuuuudieMaaaag, hahahaha!" The crowd tittered. Anushri ignored them and remained intently focused on T.S.

As she considered her next query, one of the guards beyond the gate, in the x-raying area, noticed the commotion and walked over. "Hey Type, you doin' OK? Everything in order over here?"

T.S.'s eyes refocused. "Yup. She ain't gettin' through. She can try her fancy reflection tricks, but rules are rules, and she doesn't have the bypass." T.S. turned back to examining boarding passes and waving people through. The second guard nodded knowingly and returned to the x-ray machines.

Anushri spoke suddenly, with the same metallic command note in her voice as before: "Arguments to checkCredentials!"

This time T.S. was unfazed, and continued to check identification as the words barked mechanically from him: "PassengerProxy, CredentialFactory."

The crowd was surprised. The blonde woman muttered: "Credential factory? How the hell does anyone get through this line? Why doesn't the API just take a Passenger?" There were murmurs of agreement as everyone waited to see what Anushri would try next.

Anushri thought for a moment. Chasing the chain of wrapper objects would take her too long. The flight gates would close in maybe five minutes, maybe less. She thought of her husband. He was such a great programmer, and so conscientious with his testing... wait! That was it!

Her voice, suddenly like gears grinding, commanded: "What is your Unit Test API?" If he had a unit test, then there was probably a way to substitute special bypass logic.

T.S. smiled this time: "Ain't got one -- not on a production box, anyway." His big smile was meant to be disarming; he was still trying to woo the crowd. But looking at him was now positively chilling. The next person through the gate snatched back his boarding pass as if it were about to go into a paper shredder.

Anushri was crushed. She held her son tight to her. She'd been brave, but the tears finally came. She wept for her husband in the E.R., for her son who might never see his father alive again. "You bastard!" She railed against T.S., hopeless and defeated. "Why couldn't they have written you in JavaScript!"

Delmore, who was nearing the front of the line, took pity at last. He turned to the crowd and said to them: "Folks, this woman has to get on her plane. She has to, and you all know it, and it's almost too late. Mr. System here isn't going to let her through, and we've all seen firsthand how pathetically inflexible his interface is."

He paused and looked them all in the eye. The crowd was defiant by now, and they returned his gaze. "There's only one way through this jerk, and it's through this line. And that means we can't be in it."

Delmore stepped backwards, over the velvet rope. Two tall crew-cut men next to him did the same. Soon the whole caterpillar was dismantling itself as the crowd clambered away, clearing a path.

T.S. waved the last caterpillar-head through, and the line was now gone, his audience all staring at him expectantly. The big guard couldn't think of much to say, so he called out: "Next!"

Nobody moved.

Delmore beckoned to Anushri. She and her boy walked up to T.S., who smiled triumphantly. He had won. Nobody had broken any rules. The dark-haired beauty and her son had their boarding passes and IDs; all was in order. He waved them through. "Next!"

Anushri and the boy raced to the x-ray machine. It looked as if they were going to make their flight after all. The caterpillar re-formed as people silently found their places again.

Forty-five minutes later, not a soul in the terminal remembered the events of the previous hour, save for the guard T.S., and he was already forgetting it all rapidly. After all, no rules had been broken. He had performed his duties to the letter. The line was orderly, and the terminal was secure.

There was no need for loopholes, no need for exceptions. The system was working precisely as it was designed to work. The caterpillar flowed lazily, like a river. T.S.'s mind settled into his work.

Some time later, T.S. noticed a young girl standing at the head of the shortcut lane. He'd seen her head bobbing around at the end of the security line a while ago. That was one of the sure-fire signs of a troublemaker: they always started by bobbing around, looking to see if there was some way through other than the Line.

The girl began shyly: "Sir, they're all out of Lane Passes, but my plane is leaving in 15 minutes..."

Sunday, December 17, 2006

I take it all back! Send me your money!

Author's Note, written (shortly) after this entry — I'll spare you the suspense: today's post is crap! It's partly just a "hello, I'm still alive" ping, and partly an effort to close out some of the topics I opened recently, so I can move on to new things. And it does poorly at both. But I've been insanely busy at work for the past 2 months, and I wanted to get one more post in before the end of the year, so there's been no good time for it. We just recovered from a 3-day power outage in Kirkland after a nasty windstorm, so before diving back into work, I figured I'd throw these thoughts out, jumbled as they are. There's nothing new here, though.

After reading the first couple of comments, I've decided that my next few blogs will almost certainly be about Emacs hacking, since I've done a ton of that on the side lately. It's fun to talk about, and yields lots of prettily-colored code. So after the first of the year, look for some Emacs stuff from me.




Rumor has it that I've been eaten by Godzilla. There's a fat-joke in there somewhere, my gut tells me. Ahem. There's also apparently a rumor that I was abducted by the Agile Mafia, as if they were somehow competent enough to get past their standing morning meetings and do something about me. And there's evidently speculation that Google got mad at me for exposing how cool they were, and they locked me in a room filled with doughnuts and are forcing me to eat them all.

That last one is -almost- true, except the part about the lock on the door. They know all they need is the doughnuts.

Anyway, I figure I'd better write something before Godzilla really does eat me. Lord knows all this Google food is turning me into a target even Gojira-sama might notice.

I don't know what I'll end up writing about today, but let's start with my blog anniversary, shall we?

One-Year Blog-O'-Versary

Almost exactly a year ago (give or take a couple days), my buddy and fellow Googler Sean O'Connor was walking over to invite me to play foosball on our crappy old table, as we often did, when he mentioned in passing: "hey man, I read your blog".

"Sorry, which blog is that?" I hadn't blogged anything to my internal Google blog in a while, but I assumed that's what he meant.

"The one about Emacs".

That was just weird. I hadn't written anything about Emacs since back at the 'zon. What was weirder was that he was the third person that week to mention a blog entry of mine, which had never happened in my six-month tenure at Google, and each person had mentioned a different old Amazon entry.

A little investigation turned up that there was this spiffy new-ish user-posted news site called Reddit, a PG-startup site no less, and a bunch of my old Amazon internal blogs had been showing up there that week. Which was odd, because I had put them online fully six months earlier and then forgotten all about them. Why were they popping up now, all at once?

For lack of a better term, I'll call this phenomenon Essay Molasses. If you know what it's really called, please enlighten me. I wrote about it in my You Should Write Blogs essay back at Amazon, describing an essay my good friend Jacob Gabrielson had written (still Amazon proprietary info, alas, though it wouldn't hurt them to publish it now.) Almost nobody read it initially, to my considerable dismay, but after six or eight months its key ideas had mysteriously spread to virtually everyone with decision-making power in the company.

Essay Molasses is a time-delay on a meme that would otherwise spread rapidly, but because the idea is encoded in an essay from a relatively unknown author, most people aren't going to read it until it seems everyone else has read it. Like anything else, the growth rate is effectively exponential, so by the time you hear about it, it seems to have appeared overnight. But its initial readership is measured in days (or weeks) per person, rather than persons per day. By the time you hear about it, it may have been around for quite some time.

Based on my two data points — Jacob's essay and my Amazon blogs — I could conjecture that the time delay is about six to eight months. But I'm sure it's a function of context, and content. How long was it before Hardy noticed Ramanujan, or Einstein paid attention to Bose? (Not that anything I or Jacob have written is nearly so original nor valuable, of course, but I figured I'd pick some examples of Essay Molasses that you've actually heard about.)

Although I don't have anything resembling conclusive evidence, I think Essay Molasses is closely tied to the survival of the idea. It's a gestation period, and if you take away the essay that wraps the idea, the embryo won't survive. People accuse me of being long-winded, but it's not often that breaking short wind, amusing as it can be, has any real long-term effect.

I could use a good marketing name for this longer-is-better phenomenon too. The synopsis is that I think taking the time to write about something thoroughly gives it a greater (if slower) impact. Look at Gladwell's "The Tipping Point," or Surowiecki's "The Wisdom of Crowds." Either of their theses could have been succinctly expressed in a simple essay or paper, but would they have had the same global impact? I think not.

With respect to blogging and essays, researching your idea a bit is helpful, sure, but even just thinking about it carefully can often provide that nourishing yolk that gets your idea past the chicken-and-egg adoption phase and into the mainstream, where people can whine about how much they dislike you on Reddit. Ah, sweet, sweet success.

And taking too long to research it can effectively kill it. I've been blocked many times by the desire to turn an idea into a more formal thesis, when all I really needed to effect change was to get the idea out there. People are smart, and they'll figure out the remaining details (and omissions or errors on your part) quickly enough once you've got your basic idea across. So call it laziness, call it long-windedness, call it sloppy pseudo-journalism, but I'm writing the way I do because I suspect it yields the best ratio of effort to impact. Feel free to be your own judge!

So, after that six-month pregnant pause last year, my Amazon blogs burst out last December like a sudden, fierce, smelly wind, blowing everyone's hair back and causing mere candles to flame like war-torches. And I've emitted occasional new outbursts ever since.

The problem is that once you've said one or two things that are tolerably credible, everything else you write gets immediate attention, with no molasses anywhere in sight, and if you say something stupid it will be all over the place in no time. So it can be a little scary writing anything at all. It's no wonder people are so slow to publish research papers; they have to cover their arses tenfold, and hem and haw with the approriate disclamatory rituals to minimize the risk of being found wrong on some point, to their supposed perpetual embarrassment.

Of course Ramanujan was wrong about a lot of his theorems, and it doesn't mar his genius in the slightest. But most people would rather be right about something trivial, or simply not be heard at all, than be wrong occasionally in the quest to get their ideas out to others.

OK, I've said way too much on Essay Molasses. Presumably the lessons here are: (1) say what's on your mind, and don't worry overly much about being right or wrong or you'll delay the flow of ideas; (2) don't expect anyone to read your work until sufficient Essay Molasses has flowed slowly under the bridge, and (3) take some time to tell the whole tale. Even if people don't care for the destination, they might possibly enjoy the journey.

Agile Postscript

I've been thinking of writing some sort of Grand Finale to what would then become my Agile Trilogy, but none of my ruminations have really jelled into any, ah... any jelly worth... er, ruminating. *Cough* So rather than trying to make them pretty, I'll just dump my ideas on you in the buff. They're flitting around in my head like, oh, a bit like healing fairies from any Zelda game, if you really must know where I'd rather be right now, and I'm going to snatch them in mid-flight and put them in little jars for you to gawk at.

You can really tell when the wine gets ahead of the writing. I'm telling you.

First, I was going to write an earnest little memo from Karl Marx, written to Josef Stalin and Mao Tse Tung, along these lines:

Memo: Communist Imposition
To: Josef Stalin, Mao Zedong
From: Karl Marx

Dear esteemed General Secretary Stalin and PRC Chairman Mao,

I have lately been surfing the net and have heard alarming rumors to
the effect that certain regimes have been imposing their Communist
processes on the proletariat. I assure you gentlemen that this
behavior is one hundred percent at odds with the core principles of my
Communist Manifesto.

To be sure, introducing Communism does sometimes requires a bit of
counterintuitive bloodshed in overthrowing the bourgeoisie. However,
after the revolution, self-determination should be the rule: citizens
should stop and reflect at regular intervals on how to become more
effective, and adjust their behavior accordingly. These rumors of
enforced adoption of personality cults, violent suppression of
non-Agi, er, I mean non-Communist Party ideas, and the blatant
exploitation of the working class — these are all distortions of
the ideas I set forth in my Manifesto.

Well I say: tsk, tsk. All an educated man can hope for is that
continued patient education will eventually help people understand
what Communism is really about. And I prefer to explain rather than
to kill dissenters.

Yours always,

Karl Marx


Wouldn't that have been cute? Nah, you're right. Stupid. Plus, I kinda have a soft spot for Fowler on account of his Refactoring book (you know, the one none of you IntelliJ users actually read), so I'd have reservations about poking fun at him in public, even if he'd said something really dumb.

Next, I was thinking of responding individually to each of the 10 trillion comments I got on my last 2 blog entries, like so:

Commenter: Entertaining rant, but you just don't get it! Agile works for me, so it Must Be True.

Me: I take it all back! Give me your money! Oh, God, what have I done? Is it too late for me to get away with becoming a Feng Shui consultant? I had no idea how much money you can make by exploiting the credulous. Seriously. Had I only known... ohhhh, the pain... But given that I've pretty effectively excluded Agile Consulting from my agenda, how far afield must I reach before my balls are safe?

Commenter: Great post! It's the best!! If you've got a moment, please drop by http://www.i-beat-captcha.com and find out all about my spam thingy. Made it myself!!

Me: Mmmmmm.... Spaaaaammmmmm....

But I'm too lazy to respond to all of them personally. So, you know, just send all your disposable income to an Agile Consultant near you. Any one of them will do just fine. They all say the same thing: "You don't quiiiiiite get it yet. Cha-ching!"

NEXT, I was going to reflect on how I don't like ceremonies. Yeah. Seems unrelated, I know. But Agile, being a church and all, is big into ceremony and ritual. You'd think ceremony would kinda get in the way of actually getting real work done, but they'll tell you otherwise. So they get this aging bald guy to cut a big ribbon in front of the new mall... damn, I'm sorry. My bad, I've got the wrong ritual. They get an aging agile consultant to steep your team in all the latest stand-up rituals, including the all-important Passing the Hat ritual.

I'm not big into ceremony. I've known this for many, many years, since I was a wee youngun on my way to First Communion, and my intense dislike was deeply reinforced during my long tour in the heavily ceremony-laden U.S. Navy. I don't like dressing up. I don't even particularly care for the Two Big Ones: weddings and funerals. I can deeply appreciate receptions and wakes, since they're much less rehearsed, and peoples' true feelings tend to display. But ceremonies, where we all dress up and hold hands and mouth the expected mouthings, they're just not for me.

Maybe they're for you. If so, we differ, and I can respect that. Most people like their ceremonies; they think they're one of the things that separates from animals. Not true, of course; I've seen hippos and crocodiles performing the most amazing and heart-rending death-rituals on Discovery Channel. But ceremonies make people feel like, you know, life's all important and meaningful, and that big green mall-ribbon somehow imparts depth to an otherwise shallow and trivial historical event.

Not for me. And I see a clear connection between Agile, with its stand-up meetings and its status reports and its hokey project management artifacts, and the desperate need for ceremony. To me, ceremony is necessarily phony. Real ceremonies are unrehearsed, impromptu gatherings in which the participants stand momentarily in awe of the amazing events in which they are (almost always unexpectedly) partaking. Such occurrences happen once and only once per event. Spare me the Mall Grand Opening Guy.

FINALLY, in my third big Agile Installment, I was planning to demonstrate to all interested parties that Agile is simply about recycling tired ideas, nothing more. The Agile folks are entirely concerned with being able to predictably replicate shit that other people have already delivered: it's just the Church of Me-Too.

This last point seems pretty obvious, if you stop to think at all about it for a moment. Do we have Agile Modern Art, in which teams gather to predict how many days it will take to produce the next modern masterpiece? Do we have Agile Great American Novelists getting together in little self-help meetings to talk about how Operations Research can help them write the next Da Vinci Code? (You can just smell the jealousy on them, too. How did such a crap novel make it so big? A hundred thousand prospective Agile Authors have been asking themselves that since it farted itself into the public consciousness. Oh you can bet your sweet b-hind that I'm jealous. At least I admit it!)

The answer to all these rhetorical questions, is, of course: "eight?" (If you're a Simpsons fan, anyway. "Do I know what a rhetorical question is?" --Homer, in quite possibly the Best Line Ever.)

You can't predict the delivery or quality of great art, unless maybe you consider Thomas Kinkade to be Great Art. You can't predict the next Mark Twain or the next Douglas Hofstadter, and no amount of stand-up-meetingism is going to produce one for you. Index cards surely will not. But prediction is what half the Agile folks claim to stand for. Ah, prediction. How we look to our psychics, our soothsayers, our bookies. Now, yesterday, tomorrow, forever. We all yearn for prediction. But when you fall for that crap, you've fallen, like a star from the sky.

I predict that I won't last another glass of wine at this rate, so perhaps I should begin meandering towards a wrap-up.

It seems pretty clear to me, on reflection, that Agile is really all about being able to predict and manage the delivery of stuff that others have delivered. The same old tired websites, the same old inventory-management systems, the same old database-driven doohickeys that the consultants in question have seen a hundred times before.

That's fine and dandy. Wonderful, in fact. As long as someone else does it. You? Fine. Go for it. But I'm going to close this blog with a little belief of mine, one that you're most welcome and encouraged to disagree with, although it won't dissuade me in the least. Let's get to it, shall we?

The Silver Keyboard

In his Mythical Man-Month essay, Fred Brooks Jr. argues that our attitude towards software project management derives from our folklore, in which people cried out for a silver bullet to slay werewolves. But so far there's been no silver bullet: no single, magical, overnight 10x productivity gain.

So when you meet a werewolf: some project or task you're saddled with that threatens your livelihood, what can you do? Lacking a silver bullet, all you have is your own two hands. You grab your silver keyboard and start bashing away. That is to say, you roll up your sleeves and you do the work.

If your boss is a jerk, fire your boss. That much, I hope, is obvious. When I first wrote about software development at Google, most people wrote me and said "you pegged it." But a few people mailed me privately and said: "which Google do you work for? Because it's not like that for me at all!" The answer is: Google is what you make of it, for yourself. If your boss is a vampire, fire him or her! Google can't (and shouldn't) police every last nook and cranny of its own development organization. You need to make sure your group is Googley. And you know exactly what that means, so if it's not that way now, fix it!

And if you're not at Google, which many of you brilliant engineers out there are not, you still know what Googley means! It's the opposite of "sucky". If your organization or team or little cubicle cul-de-sac is un-Googley, then fix it! Googliness needn't be constrained to Google, however prevalent it might be there today.

But how? I mean, do you have that kind of power? You're just a peon like me, right? How can you fire your boss? Or for that matter, if your boss totally rocks in an unrecognized way, how do you promote your boss?

The answer is that you need to be confident. Agile consultants are on the lookout for weakness. If they espy it in you, they'll pounce! And you may not be strong enough to withstand their Agiley claims. I wouldn't have been, fifteen or twenty years ago.

But there's an out for you — for all of us: if you're a superstar, then your management chain needs you. They want the best for you, and if for some reason they don't, or they just don't understand how valuable you are, then they're doomed. Which means "screwed," in modern terminology.

So how do you make yourself a superstar? Never stop learning. I've heard people say they think this position is a crock, that it's ludicrous, that you couldn't possibly spend your whole career learning new things.

But I think differently. I think every program you write should be the hardest you've ever written. And that's what I blog about, mostly. Improving yourself. Because most employers, the ones who matter in the long run, they'll be able to see how great you've become, and they'll alter the very course of their business plan, if necessary, to make the best use of you. Does that sound incredible? Well, I've seen it play out both ways over my modest 20 years in our industry, and that's how I see it.

And that's my little belief. You're welcome to regard it or disregard it as you see fit. But I call my own shots. And even now, after I've found an employer who surrounds me with almost terrifyingly brilliant people, people far greater than I am in virtually any technical dimension you care to name, I can still tell, and tell them, when they've gone astray, and they listen. (Fortunately they rarely stray, and I'm still learning a ton from them every day. The benefit of working for Google, at this point, is still mostly mine.)

And it's a wrap!

My next few blogs, I think, will be about how to make yourself an engineer who's above the Agile Treadmill, who can call your own shots, do your own startup, whatever you see fit.

But I haven't been blogging for a reason. Well, actually a few. One is that I still feel a pressing need to earn my keep at Google, and I've been working like a demon possessed on a new framework of my own devising that I think will more than adequately pay my way for the past year here. Another is that I have an obligation to get my computer game, Wyvern, back online: an obligation to some 50,000 players (out of 250,000 who've tried the game) that I dursn't easily turn my back on. Plus I have some pretty cool stuff in store there, for you Wyverneers who may be listening. Once I get this Google project reasonably secured, that is.

And I do take my own advice: I read voraciously, and I spend as much time bettering myself as I can possibly muster. It's paid off so far, so I see no reason it shouldn't continue to do so.

But these past four hours are all I can afford towards blogging this quarter, so it'll have to do! Hopefully I've quelled any concerns (or hopes) that I've been eaten by Godzilla. And I'll see you all in the New Year, blogging afresh about fresh new topics, God willing.

Happy holidays, and Happy New Year!

p.s. We have 2 Tornado foosball tables in the Google Kirkland Office now. Amazing what just one short year can witness...

Saturday, October 07, 2006

Egomania Itself


  Q. Is it your testimony that it's perfectly okay for GNC to put products on its shelves, to sell pills to people if those pills have no benefit?
A. We make the assumption that they make the decision based on some perceived benefit.
GNC board chairman Jerry D. Horn, in sworn deposition


Whew! What a crazy ride the last 2 weeks have been: I was Joel'd, Bray'd, Slashdotted, featured in Agile Mafia Monthly, and completely buried in an avalanche of email from Alert Readers.

The highlight of it all was probably the invitation I received to speak at an Agile conference in Vancouver, B.C. They told me, and this is a direct quote, "we promise not to cut your balls." They then went on to reassure me that just to be sure, they'd get me a security guard. Yikes!

No doubt the Agilawyers have been scrutinizing my Tyler Durden clause, seeing if they can find a loophole and take me up on it early. Ha! I won't be taken so easily. "A fool and his balls are soon parted," as the old saying goes. More or less. So I declined politely, filed for a legal name change, got on the list for facial reconstructive surgery, and moved to Oklahoma. Can't be too careful when your you-know-whats are at stake.

The second runner-up was definitely this email, which I've reproduced in full:

From:  Agilists Carnival

If you've received this email, you've been referenced in the
October 5th Carnival of the Agilists. Thank you for your
insight and commentary on how to make Agile software
development as good as it can be.

If you like, please consider linking to the carnival from
your blog, to help spread the word. And, as always, if you
would like to help out with the Carnival, either as a
rotating host, or as a one-time guest host, please feel free
to contact me at: <email withheld> and let me know.

Thanks again!
J.

Hee hee. Looks like someone didn't get the memo.

Anyway, because my last blog entry was soooo read and skimmed and linked and contested and grokked and misunderstood and praised and denounced and everything, I figured I'd break tradition, buckle down, and actually write a follow-up.

Incidentally, I can't even begin to tell you how happy it makes me to break all the stupid-ass grammatical rules my Agile Grammar Teachers forced on me in grade school. They represent a brittle, method{olog}ical personality type that's still alive and thriving today, busy certifying scrum masters around the world, painting their cats, all that good stuff. [We'll get to the cats in a little while, in case you were wondering.]

OK, enough preambling already. On to the main show! I promise you many mean jokes in tonight's entertainment, an insight or two, and if you're a good audience, I've got a bona-fide magic trick for you at the end. Hang in there!

There's More Than One Good Kind

Before we get into the interesting stuff, I thought I'd clear up one major misconception that kept coming up.

Some people apparently thought that by using Google's development practices as a counterexample, I was implying that Google's approach is the only alternative to (a) Agile, (b) Waterfall, or (c) Nyarlathotep.

Not true! I was merely using Google's process to illustrate that alternatives do exist and they do work. I offered this evidence largely to combat the Agile claim — and I'm paraphrasing loosely here — that "if you don't forward this Agile Manifesto to 5 friends within 47 seconds, you will die instantly and then be thrown off a high building into a pile of manure."

Seriously, though: you don't need to look at Google to discuss life beyond Agile. If you just look around, you'll find plenty of "success stories" from non-Agile teams. Looking for "success stories" is, of course, stupid; it's exactly the kind of selective reinforcement that can lead to (e.g.) chronic gambling disease. If you want real supporting evidence, you should construct the scientific kind, using experiments. Yes, I'm afraid that means looking at the failures too. Ouch! So bad for marketing. Whenever you hear Agile people asking around for "success stories", remind them politely that only looking at the positives is pseudoscience.

If you walk around and do a survey for yourself, you'll find two dimensions producing four categories: Agile vs. non-Agile, failure vs. success. You'll be able to find projects that fit into each category. Don't focus on the "success stories" — look at the whole picture, and correlate your findings with those of friends from other companies. Most of you will find that if you did an honest assessment, Agile's not very widespread, and it's not as success-prone as they'd have you believe. Not by a long shot.

As one concrete example, one commenter observed that Blizzard doesn't ship on a schedule; they ship "when the app is done". Well, raise your hand if you haven't heard of World of Warcraft. Hands? Anyone? Didn't think so. WoW broke into the crowded and locked-in gaming space (people like to stay on their current MMORPG), and stomped the living crap out of huge competitors like Sony and Microsoft. Overnight. Boom. And Blizzard is in a business that's as about different from Google's as you can get — especially their shrink-wrap games, which have also historically been groundbreaking, award-winning, runaway bestsellers.

But rather than enumerating all the non-Agile teams and companies as special cases, let's get straight to the rule: Most great software developers around the world don't use Agile. They just work hard, they stay lightweight, and they ship great stuff. Most developers still have no idea what Agile even is. Think of that!

Agile is a niche, a market minority, almost an aberration, really. It just happens to have a lot of marketing, because it opened up a hitherto entirely untapped new market for snake oil in the tech sector. Consultants are making money hand over fist by extending their contracts with credulous clients who are told they don't have the process "quite right" yet, but they're almost there!

Agile wouldn't be a big deal, except the Agile camp is really loud. Annoyingly so. Loud enough to start interfering with regular developers' work. Which is why I had to speak up. Agile was the Mystery Topic that gave me Blogger's Block for nearly 2 months. At some point, though, I just couldn't take it anymore. There were critics here and there, sure, but none of them were as loud as the Agile folks.

So I yelled as loud as I could: loud enough to make it onto Slashdot, even! Which is exactly what I wanted. I only had one high-level takeaway, one marketing message I wanted to get out to developers everywhere: It's OK to say No to Agile. That's it. Nothing more complicated than that. But the Church of Agile was getting so powerful that it was becoming increasingly acceptable to criticize people in the workplace for not being Agile. More on that in a bit.

As it happens, getting my broadcast message out wasn't too difficult, on the whole, because snake oil's kinda funny. It's an easy humor target.

You've probably noticed by now that the Agile folks are utterly lacking in humor. After my last blog, the whole Agile community started maneuvering and positioning and strategizing — and I'm talking about the smarter ones. The others just screamed "I lost thirty-five pounds on that diet!", demonstrating that they might possibly have missed my point about experiments vs. statistically meaningless anecdotes. Can't blame 'em; it was a lot of information to process: a whole book squeezed into a chapter-sized blog. But across the board, there hasn't been a whole lot of self-deprecating humor on the Agile side.

Well, any time you find a community whose overall Humor Level is rated at "Homeland Security", it's a pretty good indicator that you don't ever want to have to deal with them. Ever.

In any case, the whole discussion about whether Google's approach is viable for tech companies in other domains is a red herring. Most companies don't use Agile Methodologies, or if they do, it's only a handful of teams, maybe 10% or fewer, I'd guess. At least it's true at the companies I know lots of people from - Sun, Microsoft, Yahoo, Amazon, Google, Blizzard, and other places like them: industry leaders who write kick-ass software. They do it almost entirely without Agile. It's not just Google. It's everyone.

Agile folks are so damn loud, I mean I swear they ought to get some sort of disturbing-the-peace ticket, that they'd have you believe that even if they're not a majority, they're some sort of moral majority.

I think you know better by now.

Static Typing and the Need for Reassuring Metadata

A few people caught a rather subtle point about static typing I made towards the end of my good/bad agile rant. I didn't dwell on it, but all the follow-up cries of "well what should we use, then?" have got me wondering whether it's actually more central than I suspected at the time.

I'm going to share some apparently unrelated short anecdotes. They mesh, though. Bear with me.

I had a high school English teacher who, on the last day of the semester, much to everyone's surprise, claimed she should tell just by looking at each of us whether we're a slob or a neat-freak. She also claimed (and I think I agree with this) that the dividing line between slob and neat-freak, the sure-fire indicator, is whether you make your bed each morning. Then she pointed at each each of us in turn and announced "slob" or "neat-freak". Everyone agreed she had us pegged. (And most of us were slobs. Making beds is for chumps.)

As I've done for a great many other programming languages, I've bashed on Perl's technical weaknesses at length in the past. To my continued amazement, the Perl folks are the only ones who never get upset. They just say "Haha, yeah, boy, you're right, it sure is ugly. Heh. Yeah, so, um, anyway, I'm going to get back to work now..." It's awesome. I've gained so much respect for them. It's almost enough to make me go back to programming in Perl.

Well, almost.

The Perl folks have this Perl Haiku competition each year. It's a nifty idea, and it's pretty amazing that you can write useful seventeen-syllable programs.

I tried it with Java once, and produced a valid Java haiku:
ArrayList<int> myListOfInt = new ArrayList<int>();

which, spoken aloud, reads:
ArrayList of Int
my list of int, equals new
ArrayList of Int

Maybe it's just me, but I find it pretty farging depressing that a simple declaration like that can be haiku-sized in Java.

The noted Far Side cartoonist and computer scientist Dr. Gary Larson published the definitive paper on Java-style static type systems in October 1987. I've taken the questionable liberty of reproducing his paper here, in full, sans permission, in the hope that it'll fall under "fair use" if enough of you buy a Far Side book. If you don't, well, just remember what happened to Miranda Pinsley!

Here's Dr. Larson's renowned treatise on non-inferring static type systems:
Man has painted large labels on cat, dog, house, tree, door, himself.  'That ought to clear up a few things around here!'

It's truly one of the most insightful Computer Science papers ever published. Yet what Dr. Larson may not have realized at the time was that he was also predicting Agile Methodologies!

If there's one thing I've learned over the years about type systems, it's that you have your slob type systems and your neat-freak type systems, and it comes down to personal preference. The neat freaks (Java, C#, C++, Pascal) know damn well that the slobs (Perl, Python, Ruby, JavaScript) are just as productive. Maybe even more so. Nobody really knows for sure. One thing is clear, though: everyone's getting stuff done, regardless of their language choice. The whole debate isn't actually about productivity at all, even though most people still think it is. It's about whether you can stand to live in a house where the bed isn't made.

Did I mention making beds is for chumps? Well, that's just my take on it. My wife makes our bed almost every morning. I find it annoying and useless, especially since we have to tear the thing apart again every night. But she has to have the place spotless or she gets really uncomfortable. So, you know, I have to pick up my socks and all that stuff.

There will always be slobs, and there will always be neat freaks. There will always be programmers who have to paint "THE CAT" on their cat and "THE DOG" on their dog, metaphorically speaking, or they'll be as stressed and uncomfortable as my wife is when I've left a bunch of dishes in the sink and clothes on the floor. And there will always be programmers who need a methodology: even if you can convince them that their current one isn't helping, they can't just drop it. They need to switch to something else.

And that's perfectly OK with me, as long as they understand that it's not some proven silver bullet — it's just a style thing.

Unfortunately, Agile has been propagating like a virus via a slightly buggy survival mechanism built into all our brains, every single one of us. You have to take desperate measures to combat a viral epidemic, so I'm going to pull out the really big guns, and tell you about my dad's chili.

Making Chili

Many die-hard Agilytes have angrily retorted that Agile has been working for them for years, that they've been successful with it, and so on. They seem to think I'm claiming that Agile "doesn't work". A few even thought I was claiming that Agile projects are unsuccessful 90% of the time!

Well, make no mistake: Agile works (or at least it can work) just fine. The problem is much more subtle than that; most of you appear to have understood it no problem, but many Agile folks missed it entirely. So allow me to try being un-subtle, and see if that gets the point across to them any better.

When I was a teenager, my dad and my brother Mike decided to make homemade chili. I'd never seen it made before, and I watched with keen interest as they added beef, beans, some veggies and spices, and other ingredients. Dad would taste it, add some more ingredients, wait a bit, taste it again. My dad has some pretty good recipes. So you can imagine my puzzlement when he opened the cupboard, pulled out 2 cans of Hormel chili, opened them and dumped them in.

I waited a respectful moment or two before asking him why he was adding canned chili to his chili. They both said it tasted terrible, but, as my dad now-famously observed: "You can start with dog shit, and if you add enough chili, you get chili."

Similarly, if you start with an Agile Methodology, and you add enough hard work, you get a bunch of work done. Go figure.

But that's a tautology; you can substitute anything you like for "Agile Methodology" and it's still true. It's probably not difficult to find people who believe that Feng Shui has brought them success in their projects for years. Or throwing pennies in fountains. Heck, there are probably some people who practice witchcraft to help their projects out, and a great many of those projects — probably the majority — wind up being successful.

If you do a rain dance for enough days in a row, it will eventually work. Guaranteed.

So I'm not saying Agile doesn't work. It does work! But it's plain, unadulterated superstition.

And that, dear audience members, brings us to our last big topic today. Let's start prepping for that magic trick!

The Break-Dancing Chicken

Oh yeah. First I have to tell you about the chicken. We've done the cats, we've done the chili, we're on to the chicken. Almost ready for the final act, which is really gonna piss a lot of people off, I can tell you that right now.

In 1986, my college Psychology 101 professor told us a little story about her Abnormal Psych class from the previous year. The students each got a lab rat, and they had to train their rats to run through mazes by rewarding the rats for exhibiting properly ratty maze behavior. But there was one girl who was terrified of rats, so they gave her a chicken, and she trained it to play the piano by pecking notes with its beak. Each time the chicken played the right notes, she'd give it a pellet (or whatever the hell chickens eat). Wrong notes, no pellet. She'd teach it one new note at a time, and eventually it would know the whole song.

At one point, after she'd coaxed the chicken to learn maybe half its song, the chicken made a violent side-to-side motion just before playing exactly the right sequence of notes, including the new note she was trying to teach it. It had played the right notes, so she had to give it a pellet. The chicken had definitely figured out that doing what she wanted would get it yummy pellets. After that last pellet, it decided that it had been rewarded for both the note and the motion, and from then on, it made crazy twisting motions continuously as it played. She had to keep rewarding it when it played the right notes, and she had no good way of informing it that the twisting was unnecessary (not without starting over, and there wasn't time). So the chicken went on happily believing that thrashing violently was helping its project succeed.

They eventually went on tour to the local colleges with it, billing it as "The Break-Dancing, Piano-Playing Chicken".

Our prof told us wide-eyed freshmen that this phenomenon is called superstition, and it refers to the exact same kind of superstition you think of when someone mentions black cats, walking under ladders, breaking mirrors, opening umbrellas indoors, and so on. 'Superstition' is any belief not based on scientific experiment or pure deductive reasoning. (More or less.)

Well, geez, that seems pretty broad. We don't bother to go verify most of the stuff we know (or think we know) — does that mean we're all really superstitious?

It turns out we are. See, our brains are powerful pattern-matching machines, and in order to survive and to learn about the world we have to be able to build up millions of inferences about our experiences, and then match against that "pattern database" in real time. In order to process all that information fast enough, our minds have a tendency to assume that if two events are coincidental, they're also correlated.

That last sentence was probably the most important in this whole rant, so I'll restate it, in case you were dozing off at my boring lecture. Don't do it! We have a magic trick coming soon. You won't want to miss it!

Restated: when you see two events happening that appear to have a cause-and-effect relationship, then your brain naturally and automatically looks for reinforcement. Also known as "success stories". The chicken, completely by accident, got the right notes at roughly the same time it twisted. Then it received a pellet, and from then on it believed increasingly that the twisting was one of the things that "caused" pellets to appear. The belief was only reinforced from then on.

That's it. That's how superstition happens. And wouldn't you know it, most of the time it's right. That's why it works so well as a pattern-formation strategy. You touch the stove, you burn your finger. Do it again, sure enough, you burn it again. Cause and effect. But just because you've drawn the correct inference doesn't mean it's not superstition! You never really know if it's a "fact" until science has promoted it. Getting things promoted from superstition to accepted fact is the fundamental endeavor of all science.

Ironically, the chicken story is an Urban Legend. I have no idea whether my prof was telling the truth when she said it was "her" Abnormal Psych student who trained the chicken. Maybe she said it was another prof, and I just remembered it as being her class. I suppose I could go dig up the details (with a LOT of effort; this was 20 years ago) and possibly find out whether it's true or not. Until then, it's an Urban Legend.

That doesn't actually mean it's not true. "Urban Legend" doesn't refer to the message content; it refers to the transmission protocol. As it happens, many urban legends are true, if you believe Snopes.com at any rate. What makes them urban legends is that they're passed from person to person without keeping any of the original details correct about who, where, when. So even if they're true, the way you heard about them makes them superstition. Until they've been verified (if necessary, by repeatable experiment), that's all they are.

So superstition isn't necessarily a bad thing! We couldn't possibly go and verify every single fact we've heard that we think is probably true. We'd never make any progress (as individuals or as a civilization). We have to take most things we know for granted. Most of what we know at any given time is superstition. It's normal.

And of course many of our superstitions are transmitted to us from other people, through virus-like propagation. One propagation type is the urban legend. Another one, sadly, is marketing. One way or another, ideas make their way to us, and our tendency is to believe anything that seems plausible.

As I was thinking last week about what I'd write about in this essay, I Googled for "technological superstition" and came across this lovely little ACM article by Jef Raskin from 2003. In it, he talks a little about the psychological foundations of superstition, and goes on to poke fun at a popular and widespread superstition, namely that expensive audio cables sound better than cheap ones. It's definitely worth a read, and it has a little jewel at the end that I'm sure will make you gasp with appreciation for the author's Nostradamus-like predictive powers.

It's all fun and games until someone loses a testicle

Superstition is harmless, right?

Well, sort of. Mostly. Usually. In fact, the point of the previous section was the superstition is not only inevitable (for all of us), it's actually a critical brain function. Most of what we believe is superstitious, and if we really want to, we can go track down any particular belief and do some reference-checking on the sources (e.g. is the source Wikipedia or the National Enquirer? Both are sometimes right and sometimes wrong, but the confidence levels differ dramatically) or even repeat the experiments ourselves.

Superstition is strongest when people desperately want to influence or control events around them, but it's too hard for them, so they turn to the magic of wishful thinking: Feng Shui, pennies in fountains, shooting stars, Agile Methodologies.

Nothing wrong with that.

The fundamental cultural problem with superstition arises when people don't know how to differentiate between reliable and unreliable data-verification sources, so they treat non-science as science. If they don't recognize or admit that they're being superstitious, then they'll feel no qualms at all about trying to propagate their beliefs to YOU.

In fact, if they can do it successfully, if everyone believes something, then it's no longer superstition. (This is why philosophers wonder whether even scientifically accepted facts are still just superstition. Quantum mechanics has revived that line of inquiry in the last century.) That's why they want you to believe what they believe: it reduces the nagging doubt that they might be wrong. It's why superstitious communities spring up and clump together and get all insular — they self-reinforce within the group.

This, folks, is what's been gradually going wrong with Agile over the past decade. It's a complicated problem. And it has NOTHING TO DO with whether it "works" for you or not.

The problem is that if it kept spreading, it was eventually going to be a majority opinion, at which point the non-believers would start losing their jobs because they weren't playing along with the Establishment. Cowboys, indeed. Now you know why Agile folks call them Cowboys. They want to brand us all as rogues, as non-team-players. Scary, eh?

I don't want to have to wear an armband just because I don't share their techno-religious beliefs. I'm already wearing enough of them as it is. The Church of C++, for instance, is a physical and moral majority, and you're a lesser programmer if you don't use it. Get used to it. You live in an unenlightened age.

As far as I'm concerned, Agile Methodologies are fine for you to use, but if you try to push them on a coworker, it should be treated as an HR violation, just as if you were discriminating against them because they don't believe in Feng Shui.

If it's a manager pushing Agile on a subordinate, it should be treated with extra severity. Reprimand, goes on the employee record, mandatory training, and 3 strikes you're out.

You have to fight fire with fire. I want it to be their career problem before it becomes your career problem.

I joke about it a lot, and I'll continue to do so, but this is actually pretty serious stuff.

But let's go ahead and close out on a lighter note. I promised you a pony, right? No, wait, it was a magic trick. That's right. Sigh. All right, then. You asked for it, you'll get it!

Anagrams and Hidden Meanings

Here's one laaaaaaaaast superstition for you: anagrams! Ooooh, they're one of the most popular of all. You rearrange the letters of a word or phrase to see if you can find hidden meanings. That's all there is to it!

Unless you're truly superstitious — and I mean the Old World kind, where you still believe in crazy stuff like leprechauns and numerology and a steroid-free Olympics — then you know that anagrams don't really have hidden meanings in them; it's just random coincidence.

I mean, let's face it: if you look hard enough, you can always find some phrase describing your intended target that also has a really unfortunate anagram. It's just blind (usually bad) luck. I think Spiro Agnew knows this better than anyone in history.

And yet... sometimes it's soooo hard to believe it's really random. When I was in my mid-twenties, I ran my full legal name (Stephen Francis Yegge) through an anagram solver to see what came up. Ouch! I mean, here I was, struggling with my weight, and all the meaningful anagrams of my name seemed to be about being fat. The worst one was "piggery, hence fatness". Gosh. Was I cursed? It sure seemed that way.

Meanwhile, my little brother Dave was all pissed off. "Why are all mine about being sick? This is lame!" I remember it like it was yesterday. "David Francis Yegge" (my parents were so creative) came back with anagrams like "gas acid fever dying" and "dying grief cave sad", and Dave was NOT happy about it. Hidden meanings my ass, he announced, and he stopped playing with the program. I don't think he ever used one again for the rest of his life.

Are "hidden messages" like these a coincidence? Yeah. Yes. They are. From any rational, sane perspective, they're just sheer bad luck. Problem is, our brains sometimes just don't want to be rational.

I'll close with one last example, and it's a dirty trick — one that would be utterly beneath a more principled blogger than myself. Because after all my ranting about superstition, I'm about to use superstition for my own nefarious purposes.

Let's say, hypothetically speaking of course, you were to rearrange the letters of — oh, how about Agile Manifesto, just for pure curiosity's sake. An innocent exercise, nothing more. ("Hey, it can't hurt! It works for me! I've been rearranging letters since 1990!" Dorks.)

Well, chances are pretty good that you'll find lots of meaningless and ho-hum anagrams, and possibly one or two moderately interesting ones.

But if you're hoping for a really clear message, especially with bigger words in it — you know, something that would be unmistakable as a hidden message — you know you're likely to be disappointed.

So let's try it. Can't hurt.

How many 2-word anagrams of "Agile Manifesto" do you think there are?

Guess what? There's exactly one:   Egomania Itself

Yup. I thought the exact same thing you just did. :-)

Sure, you can argue about probability and statistics and the fundamental nature of randomness, and you can cry foul play! no fair! poor sportsmanship! low blow! until you're blue in the face, and you'd probably be right.

But in the end, I mean, come on, you have to admit, it's a pretty spooky coincidence...

(cue haunted mansion bat music)

...or IS it?

(cue happy end-of-show music)

Tune in next time! I think I've pretty much finished up with Agile for now; I've stuffed garlic in its mouth and pounded a wooden stake through its heart and it's making melodramatic retreating gestures, shaking its fist at me and promising revenge in the inevitable sequel.

So now I can go back to writing about boring stuff that won't make me infamous. Like, you know, Google.

Hope you enjoyed the show!

Wednesday, September 27, 2006

Good Agile, Bad Agile



  Scrums are the most dangerous phase in rugby, since a collapse or improper engage can lead to a front row player damaging or even breaking his neck.
Wikipedia


When I was growing up, cholesterol used to be bad for you. It was easy to remember. Fat, bad. Cholesterol bad. Salt, bad. Everything, bad. Nowadays, though, they differentiate between "good" cholesterol and "bad" cholesterol, as if we're supposed to be able to distinguish them somehow. And it was weird when they switched it up on us, because it was as if the FDA had suddenly issued a press release announcing that there are, in fact, two kinds of rat poison: Good Rat Poison and Bad Rat Poison, and you should eat a lot of the Good kind, and none of the Bad kind, and definitely not mix them up or anything.

Up until maybe a year ago, I had a pretty one-dimensional view of so-called "Agile" programming, namely that it's an idiotic fad-diet of a marketing scam making the rounds as yet another technological virus implanting itself in naive programmers who've never read "No Silver Bullet", the kinds of programmers who buy extended warranties and self-help books and believe their bosses genuinely care about them as people, the kinds of programmers who attend conferences to make friends and who don't know how to avoid eye contact with leaflet-waving fanatics in airports and who believe writing shit on index cards will suddenly make software development easier.

You know. Chumps. That's the word I'm looking for. My bad-cholesterol view was that Agile Methodologies are for chumps.

But I've had a lot of opportunity to observe various flavors of Agile-ism in action lately, and I now think I was only about 90% right. It turns out there's a good kind of Agile, although it's taken me a long time to be able to see it clearly amidst all the hype and kowtowing and moaning feverishly about scrums and whatnot. I have a pretty clear picture of it now.

And you can attend my seminar on it for the low, low price of $499.95! Hahaha, chump!

No, just kidding. You'll only find seminars about the Bad kind of Agile. And if in the future you ever find me touring around as an Agile Consultant, charging audiences to hear my deep wisdom and insight about Agile Development, you have my permission to cut my balls off. If I say I was just kidding, say I told you I'd say that. If I then say I'm Tyler Durden and I order you not to cut my balls off, say I definitely said I was going to say that, and then you cut 'em right off.

I'll just go right ahead and tell you about the Good Kind, free of charge.

It's kinda hard to talk about Good Agile and Bad Agile in isolation, so I might talk about them together. But I'll be sure to label the Good kind with a happy rat, and the Bad kind with a sad dead rat, so you'll always know the difference.

The Bad Kind


Back in Ye Olden Dayes, most companies approached software development as follows:

- hire a bunch of engineers, then hire more.
- dream up a project.
- set a date for when they want it launched.
- put some engineers on it.
- whip them until they're either dead or it's launched. or both.
- throw a cheap-ass pathetic little party, maybe. This step is optional.
- then start over.

Thank goodness that doesn't happen at your company, eh now? Whew!

Interestingly, this is also exactly how non-technical companies (like, say, Chrysler) handled software development. Except they didn't hire the engineers. Instead, they contracted with software consultants, and they'd hand the consultants 2-year project specs, and demanded the consultants finish everything on time plus all the crap the customer threw in and/or changed after signing the contract. And then it'd all fall apart and the contractors wouldn't get paid, and everyone was really miffed.

So some of the consultants began to think: "Hey, if these companies insist on acting like infants, then we should treat them like infants!" And so they did. When a company said "we want features A through Z", the consultants would get these big index cards and write "A" on the first one, "B" on the second one, etc., along with time estimates, and then post them on their wall. Then when the customer wanted to add something, the consultant could point at the wall and say: "OK, boy. Which one of these cards do you want to replace, BOY?"

Is it any wonder Chrysler canceled the project?

So the consultants, now having lost their primary customer, were at a bar one day, and one of them (named L. Ron Hubbard) said: "This nickel-a-line-of-code gig is lame. You know where the real money is at? You start your own religion." And that's how both Extreme Programming and Scientology were born.

Well, people pretty quickly demonstrated that XP was a load of crap. Take Pair Programming, for instance. It's one of the more spectacular failures of XP. None of the Agileytes likes to talk about it much, but let's face it: nobody does it. The rationale was something like: "well if ONE programmer sitting at a terminal is good, then TEN must be better, because MORE is ALWAYS better! But most terminals can only comfortably fit TWO programmers, so we'll call it PAIR programming!"

You have to cut them a little slack; they'd been dealing with the corporate equivalent of pre-schoolers for years, and that really messes with a person.

But the thing is, viruses are really hard to kill, especially the meme kind. After everyone had gotten all worked up about this whole Agile thing (and sure, everyone wants to be more productive), there was a lot of face to be lost by admitting failure. So some other kinds of Agile "Methodologies" sprang up, and they all claimed that even though all the other ones were busted, their method worked!

I mean, go look at some of their sites. Tell me that's not an infomercial. C'mon, just try. It's embarrassing even to look at the thing.

Yeah. Well, they make money hand over fist, because of P.T. Barnum's Law, just like Scientology does. Can't really fault 'em. Some people are just dying to be parted with their cash. And their dignity.

The rest of us have all known that Agile Methodologies are stupid, by application of any of the following well-known laws of marketing:

- anything that calls itself a "Methodology" is stupid, on general principle.
- anything that requires "evangelists" and offers seminars, exists soley for the purpose of making money.
- anything that never mentions any competition or alternatives is dubiously self-serving.
- anything that does diagrams with hand-wavy math is stupid, on general principle.

And by "stupid", I mean it's "incredibly brilliant marketing targeted at stupid people."

In any case, the consultants kept going with their road shows and glossy pamphlets. Initially, I'm sure they went after corporations; they were looking to sign flexible contracts that allowed them to deliver "whatever" in "2 weeks" on a recurring basis until the client went bankrupt. But I'm equally sure they couldn't find many clients dumb enough to sign such a contract.

That's when the consultants decided to take their road show to YOU. Why not take it inside the companies and sell it there, to the developers? There are plenty of companies who use the whip-cycle of development I outlined above, so presumably some of the middle managers and tech leads would be amenable to hearing about how there's this low-cost way out of their hellish existence.

And that, friends, was exactly, precisely the point at which they went from "harmless buffoons" to "potentially dangerous", because before they were just bilking fat companies too stupid to develop their own software, but now the manager down the hall from me might get infected. And most places don't have a very good quarantine mechanism for this rather awkward situation: i.e., an otherwise smart manager has become "ill", and is waving XP books and index cards and spouting stuff about how much more productive his team is on account of all this newfound extra bureaucracy.

How do we know it's not more productive? Well, it's a slippery problem. Observe that it must be a slippery problem, or it all would have been debunked fair and square by now. But it's exceptionally difficult to measure software developer productivity, for all sorts of famous reasons. And it's even harder to perform anything resembling a valid scientific experiment in software development. You can't have the same team do the same project twice; a bunch of stuff changes the second time around. You can't have 2 teams do the same project; it's too hard to control all the variables, and it's prohibitively expensive to try it in any case. The same team doing 2 different projects in a row isn't an experiment either.

About the best you can do is gather statistical data across a lot of teams doing a lot of projects, and try to identify similarities, and perform some regressions, and hope you find some meaningful correlations. But where does the data come from? Companies aren't going to give you their internal data, if they even keep that kind of thing around. Most don't; they cover up their schedule failures and they move on, ever optimistic.

Well if you can't do experiments and you can't do proofs, there isn't much science going on. That's why it's a slippery problem. It's why fad diets are still enormously popular. People want fad diets to work, oh boy you bet they do, even I want them to work. And you can point to all these statistically meaningless anecdotes about how Joe lost 35 pounds on this one diet, and all those people who desperately want to be thinner will think "hey, it can't hurt. I'll give it a try."

That is exactly what I hear people say, every time a team talks themselves into trying an Agile Methodology. It's not a coincidence.

But writing about Bad Agile alone is almost guaranteed to be ineffective. I mean, you can write about how lame Scientology is, or how lame fad diets are, but it's not clear that you're changing anyone's mind. Quitting a viral meme is harder than quitting smoking. I've done both. In order to have the right impact, you have to offer an alternative, and I didn't have one before, not one that I could articulate clearly.

One of the (many) problems with Bad Agile is that they condescendingly lump all non-Agile development practices together into two buckets: Waterfall and Cowboy. Waterfall is known to be bad; I hope we can just take that as an axiom today. But what about so-called Cowboy programming, which the Agileers define as "each member of the team does what he or she thinks is best"?

Is it true that this is the only other development process? And is Cowboy Programming actually bad? They say it as if it's obviously bad, but they're not super clear on how or why, other than to assert that it's, you know, "chaos".

Well, as I mentioned, over the past year I've had the opportunity to watch both Bad Agile and Good Agile in motion, and I've asked the teams and tech leads (using both the Bad and Good forms) lots of questions: how they're doing, how they're feeling, how their process is working. I was really curious, in part because I'd consented to try Agile last Christmas ("hey, it can't hurt"), and wound up arguing with a teammate over exactly what metadata is allowed on index cards before giving up in disgust. Also in part because I had some friends on a team who were getting kind of exhausted from what appeared to be a Death March, and that kind of thing doesn't seem to happen very often at Google.

So I dug in, and for a year, I watched and learned.

The Good Kind


(cue happy rat)

I'm going to talk a little about Google's software development process. It's not the whole picture, of course, but it should suffice for today. I've been there for almost a year and a half now, and it took a while, but I think I get it now. Mostly. I'm still learning. But I'll share what I've got so far.

From a high level, Google's process probably does look like chaos to someone from a more traditional software development company. As a newcomer, some of the things that leap out at you include:

- there are managers, sort of, but most of them code at least half-time, making them more like tech leads.

- developers can switch teams and/or projects any time they want, no questions asked; just say the word and the movers will show up the next day to put you in your new office with your new team.

- Google has a philosophy of not ever telling developers what to work on, and they take it pretty seriously.

- developers are strongly encouraged to spend 20% of their time (and I mean their M-F, 8-5 time, not weekends or personal time) working on whatever they want, as long as it's not their main project.

- there aren't very many meetings. I'd say an average developer attends perhaps 3 meetings a week, including their 1:1 with their lead.

- it's quiet. Engineers are quietly focused on their work, as individuals or sometimes in little groups or 2 to 5.

- there aren't Gantt charts or date-task-owner spreadsheets or any other visible project-management artifacts in evidence, not that I've ever seen.

- even during the relatively rare crunch periods, people still go get lunch and dinner, which are (famously) always free and tasty, and they don't work insane hours unless they want to.

These are generalizations, sure. Old-timers will no doubt have a slightly different view, just as my view of Amazon is slightly biased by having been there in 1998 when it was a pretty crazy place. But I think most Googlers would agree that my generalizations here are pretty accurate.

How could this ever work?

I get that question a lot. Heck, I asked it myself. What's to stop engineers from leaving all the trouble projects, leaving behind bug-ridden operational nightmares? What keeps engineers working towards the corporate goals if they can work on whatever they want? How do the most important projects get staffed appropriately? How do engineers not get so fat that they routinely get stuck in stairwells and have to be cut out by the Fire Department?

I'll answer the latter question briefly, then get to the others. In short: we have this thing called the Noogler Fifteen, named after the Frosh Fifteen: the 15 pounds that many college freshmen put on when they arrive in the land of Stress and Pizza. Google has solved the problem by lubricating the stairwells.

As to the rest of your questions, I think most of them have the same small number of answers.

First, and arguably most importantly, Google drives behavior through incentives. Engineers working on important projects are, on average, rewarded more than those on less-important projects. You can choose to work on a far-fetched research-y kind of project that may never be practical to anyone, but the work will have to be a reward unto itself. If it turns out you were right and everyone else was wrong (the startup's dream), and your little project turns out to be tremendously impactful, then you'll be rewarded for it. Guaranteed.

The rewards and incentives are too numerous to talk about here, but the financial incentives range from gift certificates and massage coupons up through giant bonuses and stock grants, where I won't define "giant" precisely, but think of Google's scale and let your imagination run a bit wild, and you probably won't miss the mark by much.

There are other incentives. One is that Google a peer-review oriented culture, and earning the respect of your peers means a lot there. More than it does at other places, I think. This is in part because it's just the way the culture works; it's something that was put in place early on and has managed to become habitual. It's also true because your peers are so damn smart that earning their respect is a huge deal. And it's true because your actual performance review is almost entirely based on your peer reviews, so it has an indirect financial impact on you.

Another incentive is that every quarter, without fail, they have a long all-hands in which they show every single project that launched to everyone, and put up the names and faces of the teams (always small) who launched each one, and everyone applauds. Gives me a tingle just to think about it. Google takes launching very seriously, and I think that being recognized for launching something cool might be the strongest incentive across the company. At least it feels that way to me.

And there are still other incentives; the list goes on and ON and ON; the perks are over the top, and the rewards are over the top, and everything there is so comically over the top that you have no choice, as an outsider, but to assume that everything the recruiter is telling you is a baldfaced lie, because there's no possible way a company could be that generous to all of its employees, all of them, I mean even the contractors who clean the micro-kitchens, they get these totally awesome "Google Micro-Kitchen Staff" shirts and fleeces.

There is nothing like it on the face of this earth. I could talk for hours, days about how amazing it is to work at Google, and I wouldn't be done. And they're not done either. Every week it seems like there's a new perk, a new benefit, a new improvement, a new survey asking us all if there's any possible way in which life at Google could be better.

I might have been mistaken, actually. Having your name and picture up on that big screen at End of Quarter may not be the biggest incentive. The thing that drives the right behavior at Google, more than anything else, more than all the other things combined, is gratitude. You can't help but want to do your absolute best for Google; you feel like you owe it to them for taking such incredibly good care of you.

OK, incentives. You've got the idea. Sort of. I mean, you have a sketch of it. When friends who aren't at Google ask me how it is working at Google — and this applies to all my friends at all other companies equally, not just companies I've worked at — I feel just how you'd feel if you'd just gotten out of prison, and your prison buddies, all of whom were sentenced in their early teens, are writing to you and asking you what it's like "on the outside". I mean, what would you tell them?

I tell 'em it's not too bad at all. Can't complain. Pretty decent, all in all.

Although the incentive-based culture is a huge factor in making things work the way they do, it only addresses how to get engineers to work on the "right" things. It doesn't address how to get those things done efficiently and effectively. So I'll tell you a little about how they approach projects.

Emergent Properties versus The Whip


The basic idea behind project management is that you drive a project to completion. It's an overt process, a shepherding: by dint of leadership, and organization, and sheer force of will, you cause something to happen that wouldn't otherwise have happened on its own.

Project management comes in many flavors, from lightweight to heavyweight, but all flavors share the property that they are external forces acting on an organization.

At Google, projects launch because it's the least-energy state for the system.

Before I go on, I'll concede that this is a pretty bold claim, and that it's not entirely true. We do have project managers and product managers and people managers and tech leads and so on. But the amount of energy they need to add to the system is far less than what's typically needed in our industry. It's more of an occasional nudge than a full-fledged continuous push. Once in a while, a team needs a bigger nudge, and senior management needs to come in and do the nudging, just like anywhere else. But there's no pushing.

Incidentally, Google is a polite company, so there's no yelling, nor wailing and gnashing of teeth, nor escalation and finger-pointing, nor any of the artifacts produced at companies where senior management yells a lot. Hobbes tells us that organizations reflect their leaders; we all know that. The folks up top at Google are polite, hence so is everyone else.

Anyway, I claimed that launching projects is the natural state that Google's internal ecosystem tends towards, and it's because they pump so much energy into pointing people in that direction. All your needs are taken care of so that you can focus, and as I've described, there are lots of incentives for focusing on things that Google likes.

So launches become an emergent property of the system.

This eliminates the need for a bunch of standard project management ideas and methods: all the ones concerned with dealing with slackers, calling bluffs on estimates, forcing people to come to consensus on shared design issues, and so on. You don't need "war team meetings," and you don't need status reports. You don't need them because people are already incented to do the right things and to work together well.

The project management techniques that Google does use are more like oil than fuel: things to let the project keep running smoothly, as opposed to things that force the project to move forward. There are plenty of meeting rooms, and there's plenty of open space for people to go chat. Teams are always situated close together in fishbowl-style open seating, so that pair programming happens exactly when it's needed (say 5% of the time), and never otherwise.

Google generally recognizes that the middle of the day is prone to interruptions, even at quiet companies, so many engineers are likely to shift their hours and come in very early or stay very late in order to find time to truly concentrate on programming. So meetings only happen in the middle of the day; it's very unusual to see a meeting start before 10am or after 4:30pm. Scheduling meetings outside that band necessarily eats into the time when engineers are actually trying to implement the things they're meeting about, so they don't do it.

Google isn't the only place where projects are run this way. Two other kinds of organizations leap to mind when you think of Google's approach: startup companies, and grad schools. Google can be considered a fusion of the startup and grad-school mentalities: on the one hand, it's a hurry-up, let's get something out now, do the simplest thing that could work and we'll grow it later startup-style approach. On the other, it's relatively relaxed and low-key; we have hard problems to solve that nobody else has ever solved, but it's a marathon not a sprint, and focusing requires deep concentration, not frenzied meetings. And at the intersection of the two, startups and grad schools are both fertile innovation ground in which the participants carry a great deal of individual responsibility for the outcome.

It's all been done before; the only thing that's really surprising is that Google has managed to make it scale.

The scaling is not an accident. Google works really hard on the problem, and they realize that having scaled this far is no guarantee it'll continue, so they're vigilant. That's a good word for it. They're always on the lookout to make sure the way of life and the overall level of productivity continue (or even improve) as they grow.

Google is an exceptionally disciplined company, from a software-engineering perspective. They take things like unit testing, design documents and code reviews more seriously than any other company I've even heard about. They work hard to keep their house in order at all times, and there are strict rules and guidelines in place that prevent engineers and teams from doing things their own way. The result: the whole code base looks the same, so switching teams and sharing code are both far easier than they are at other places.

And engineers need great tools, of course, so Google hires great people to build their tools, and they encourage engineers (using incentives) to pitch in on tools work whenever they have an inclination in that direction. The result: Google has great tools, world-class tools, and they just keep getting better.

The list goes on. I could talk for days about the amazing rigor behind Google's approach to software engineering. But the main takeaway is that their scaling (both technological and organizational) is not an accident. And once you're up to speed on the Google way of doing things, it all proceeds fairly effortlessly — again, on average, and compared to software development at many other companies.

The Tyranny of the Calendar


We're almost done. The last thing I want to talk about here is dates. Traditional software development can safely be called Date-Oriented Programming, almost without exception.

Startup companies have a clock set by their investors and their budget. Big clients set target dates for their consultants. Sales people and product managers set target dates based on their evaluation of market conditions. Engineers set dates based on estimates of previous work that seems similar. All estimation is done through rose-colored glasses, and everyone forgets just how painful it was the last time around.

Everyone picks dates out of the air. "This feels like it should take about 3 weeks." "It sure would be nice to have this available for customers by beginning of Q4." "Let's try to have that done by tomorrow."

Most of us in our industry are date-driven. There's always a next milestone, always a deadline, always some date-driven goal to it.

The only exceptions I can think of to this rule are:

1) Open-source software projects.
2) Grad school projects.
3) Google.

Most people take it for granted that you want to pick a date. Even my favorite book on software project management, "The Mythical Man-Month", assumes that you need schedule estimates.

If you're in the habit of pre-announcing your software, then the general public usually wants a timeframe, which implies a date. This is, I think, one of the reasons Google tends not to pre-announce. They really do understand that you can't rush good cooking, you can't rush babies out, and you can't rush software development.

If the three exceptions I listed above aren't driven by dates, then what drives them? To some extent it's just the creative urge, the desire to produce things; all good engineers have it. (There are many people in our industry who do this gig "for a living", and they go home and don't think about it until the next day. Open source software exists precisely because there are people who are better than that.)

But let's be careful: it's not just the creative urge; that's not always directed enough, and it's not always incentive enough. Google is unquestionably driven by time, in the sense that they want things done "as fast as possible". They have many fierce, brilliant competitors, and they have to slake their thirsty investors' need for growth, and each of us has some long-term plans and deliverables we'd like to see come to fruition in our lifetimes.

The difference is that Google isn't foolish enough or presumptuous enough to claim to know how long stuff should take. So the only company-wide dates I'm ever aware of are the ends of each quarter, because everyone's scrambling to get on that big launch screen and get the applause and gifts and bonuses and team trips and all the other good that comes of launching things with big impact at Google.

Everything in between is just a continuum of days, in which everyone works at optimal productivity, which is different for each person. We all have work-life balance choices to make, and Google is a place where any reasonable choice you make can be accommodated, and can be rewarding. Optimal productivity is also a function of training, and Google offers tons of it, including dozens of tech talks every week by internal and external speakers, all of which are archived permanently so you can view them whenever you like. Google gives you access to any resources you need in order to get your job done, or to learn how to get your job done. And optimal productivity is partly a function of the machine and context in which you're operating: the quality of your code base, your tools, your documentation, your computing platform, your teammates, even the quality of the time you have during the day, which should be food-filled and largely free of interrupts.

Then all you need is a work queue. That's it. You want hand-wavy math? I've got it in abundance: software development modeled on queuing theory. Not too far off the mark, though; many folks in our industry have noticed that organizational models are a lot like software models.

With nothing more than a work queue (a priority queue, of course), you immediately attain most of the supposedly magical benefits of Agile Methodologies. And make no mistake, it's better to have it in software than on a bunch of index cards. If you're not convinced, then I will steal your index cards.

With a priority queue, you have a dumping-ground for any and all ideas (and bugs) that people suggest as the project unfolds. No engineer is ever idle, unless the queue is empty, which by definition means the project has launched. Tasks can be suspended and resumed simply by putting them back in the queue with appropriate notes or documentation. You always know how much work is left, and if you like, you can make time estimates based on the remaining tasks. You can examine closed work items to infer anything from bug regression rates to (if you like) individual productivity. You can see which tasks are often passed over, which can help you discover root causes of pain in the organization. A work queue is completely transparent, so there is minimal risk of accidental duplication of work.

And so on. The list goes on, and on, and on.

Unfortunately, a work queue doesn't make for a good marketing platform for seminars and conferences. It's not glamorous. It sounds a lot like a pile of work, because that's exactly what it is.

Bad Agile in More Detail


I've outlined, at a very high level, one company's approach to software development that is neither an Agile Methodology, nor a Waterfall cycle, nor yet Cowboy Programming. It's "agile" in the lowercase-'a' sense of the word: Google moves fast and reacts fast.

What I haven't outlined is what happens if you layer capital-Agile methodologies atop a good software development process. You might be tempted to think: "well, it can't hurt!" I even had a brief fling with it myself last year.

The short answer is: it hurts. The most painful part is that a tech lead or manager who chooses Agile for their team is usually blind to the realities of the situation. Bad Agile hurts teams in several ways.

First, Bad Agile focuses on dates in the worst possible way: short cycles, quick deliverables, frequent estimates and re-estimates. The cycles can be anywhere from a month (which is probably tolerable) down to a day in the worst cases. It's a nicely idealistic view of the world.

In the real world, every single participant on a project is, as it turns out, a human being. We have up days and down days. Some days you have so much energy you feel you could code for 18 hours straight. Some days you have a ton of energy, but you just don't feel like focusing on coding. Some days you're just exhausted. Everyone has a biological clock and a a biorhythm that they have very little control over, and it's likely to be phase-shifted from the team clock, if the team clock is ticking in days or half-weeks.

Not to mention your personal clock: the events happening outside your work life that occasionally demand your attention during work hours.

None of that matters in Bad Agile. If you're feeling up the day after a big deliverable, you're not going to code like crazy; you're going to pace yourself because you need to make sure you have reserve energy for the next big sprint. This impedance mismatch drives great engineers to mediocrity.

There's also your extracurricular clock: the set of things you want to accomplish in addition to your main project: often important cleanups or other things that will ultimately improve your whole team's productivity. Bad Agile is exceptionally bad at handling this, and usually winds up reserving large blocks of time after big milestones for everyone to catch up on their side-project time, whether they're feeling creative or not. Bad Agile folks keep their eye on the goal, which hurts innovation. Sure, they'll reserve time for everyone to clean up their own code base, but they're not going to be so altruistic as to help anyone else in the company. How can you, when you're effectively operating in a permanent day-for-day slip?

Bad Agile seems for some reason to be embraced by early risers. I think there's some mystical relationship between the personality traits of "wakes up before dawn", "likes static typing but not type inference", "is organized to the point of being anal", "likes team meetings", and "likes Bad Agile". I'm not quite sure what it is, but I see it a lot.

Most engineers are not early risers. I know a team that has to come in for an 8:00am meeting at least once (maybe several times) a week. Then they sit like zombies in front of their email until lunch. Then they go home and take a nap. Then they come in at night and work, but they're bleary-eyed and look perpetually exhausted. When I talk to them, they're usually cheery enough, but they usually don't finish their sentences.

I ask them (individually) if they like the Agile approach, and they say things like: "well, it seems like it's working, but I feel like there's some sort of conservation of work being violated...", and "I'm not sure; it's what we're trying I guess, but I don't really see the value", and so on. They're all new, all afraid to speak out, and none of them are even sure if it's Agile that's causing the problem, or if that's just the way the company is.

That, my friends, is not "agile"; it's a just load of hooey. And it's what you get whenever any manager anywhere decides to be a chump.

Good Agile Should Drop the Name


I would caution you to be skeptical of two kinds of claims:

- "all the good stuff he described is really Agile"
- "all the bad stuff he described is the fault of the team's execution of the process"

You'll hear them time and again. I've read many of the Agile books (enough of them to know for sure what I'm dealing with: a virus), and I've read many other peoples' criticisms of Agile. Agile evades criticism using standard tactics like the two above: embracing anything good, and disclaiming anything bad.

If a process is potentially good, but 90+% of the time smart and well-intentioned people screw it up, then it's a bad process. So they can only say it's the team's fault so many times before it's not really the team's fault.

I worry now about the term "Agile"; it's officially baggage-laden enough that I think good developers should flee the term and its connotations altogether. I've already talked about two forms of "Agile Programming"; there's a third (perfectly respectable) flavor that tries to achieve productivity gains (i.e. "Agility") through technology. Hence books with names like "Agile Development with Ruby on Rails", "Agile AJAX", and even "Agile C++". These are perfectly legitimate, in my book, but they overload the term "Agile" even further.

And frankly, most Agile out there is plain old Bad Agile.

So if I were you, I'd take Agile off your resume. I'd quietly close the SCRUM and XP books and lock them away. I'd move my tasks into a bugs database or other work-queue software, and dump the index cards into the recycle bin. I'd work as fast as I can to eliminate Agile from my organization.

And then I'd focus on being agile.

But that's just my take on it, and it's 4:00am. Feel free to draw your own conclusions. Either way, I don't think I'm going to be an Early Riser tomorrow.

Oh, I almost forgot the obvious disclaimer: I do not speak for Google. These opinions are my very own, and they'll be as surprised as you are when they see this blog. Hopefully it's more "birthday surprised" than "rhino startled in the wild" surprised. We'll see!