Good Agile, Bad Agile

Scrums are the most dangerous phase in rugby, since a collapse or improper engage can lead to a front row player damaging or even breaking his neck. — Wikipedia

When I was growing up, cholesterol used to be bad for you. It was easy to remember. Fat, bad. Cholesterol bad. Salt, bad. Everything, bad. Nowadays, though, they differentiate between "good" cholesterol and "bad" cholesterol, as if we're supposed to be able to distinguish them somehow. And it was weird when they switched it up on us, because it was as if the FDA had suddenly issued a press release announcing that there are, in fact, two kinds of rat poison: Good Rat Poison and Bad Rat Poison, and you should eat a lot of the Good kind, and none of the Bad kind, and definitely not mix them up or anything.

Up until maybe a year ago, I had a pretty one-dimensional view of so-called "Agile" programming, namely that it's an idiotic fad-diet of a marketing scam making the rounds as yet another technological virus implanting itself in naive programmers who've never read "No Silver Bullet", the kinds of programmers who buy extended warranties and self-help books and believe their bosses genuinely care about them as people, the kinds of programmers who attend conferences to make friends and who don't know how to avoid eye contact with leaflet-waving fanatics in airports and who believe writing shit on index cards will suddenly make software development easier.

You know. Chumps. That's the word I'm looking for. My bad-cholesterol view was that Agile Methodologies are for chumps.

But I've had a lot of opportunity to observe various flavors of Agile-ism in action lately, and I now think I was only about 90% right. It turns out there's a good kind of Agile, although it's taken me a long time to be able to see it clearly amidst all the hype and kowtowing and moaning feverishly about scrums and whatnot. I have a pretty clear picture of it now.

And you can attend my seminar on it for the low, low price of $499.95! Hahaha, chump!

No, just kidding. You'll only find seminars about the Bad kind of Agile. And if in the future you ever find me touring around as an Agile Consultant, charging audiences to hear my deep wisdom and insight about Agile Development, you have my permission to cut my balls off. If I say I was just kidding, say I told you I'd say that. If I then say I'm Tyler Durden and I order you not to cut my balls off, say I definitely said I was going to say that, and then you cut 'em right off.

I'll just go right ahead and tell you about the Good Kind, free of charge.

It's kinda hard to talk about Good Agile and Bad Agile in isolation, so I might talk about them together. But I'll be sure to label the Good kind with a happy rat, and the Bad kind with a sad dead rat, so you'll always know the difference.

The Bad Kind

Back in Ye Olden Dayes, most companies approached software development as follows:

- hire a bunch of engineers, then hire more.
- dream up a project.
- set a date for when they want it launched.
- put some engineers on it.
- whip them until they're either dead or it's launched. or both.
- throw a cheap-ass pathetic little party, maybe. This step is optional.
- then start over.

Thank goodness that doesn't happen at your company, eh now? Whew!

Interestingly, this is also exactly how non-technical companies (like, say, Chrysler) handled software development. Except they didn't hire the engineers. Instead, they contracted with software consultants, and they'd hand the consultants 2-year project specs, and demanded the consultants finish everything on time plus all the crap the customer threw in and/or changed after signing the contract. And then it'd all fall apart and the contractors wouldn't get paid, and everyone was really miffed.

So some of the consultants began to think: "Hey, if these companies insist on acting like infants, then we should treat them like infants!" And so they did. When a company said "we want features A through Z", the consultants would get these big index cards and write "A" on the first one, "B" on the second one, etc., along with time estimates, and then post them on their wall. Then when the customer wanted to add something, the consultant could point at the wall and say: "OK, boy. Which one of these cards do you want to replace, BOY?"

Is it any wonder Chrysler canceled the project?

So the consultants, now having lost their primary customer, were at a bar one day, and one of them (named L. Ron Hubbard) said: "This nickel-a-line-of-code gig is lame. You know where the real money is at? You start your own religion." And that's how both Extreme Programming and Scientology were born.

Well, people pretty quickly demonstrated that XP was a load of crap. Take Pair Programming, for instance. It's one of the more spectacular failures of XP. None of the Agileytes likes to talk about it much, but let's face it: nobody does it. The rationale was something like: "well if ONE programmer sitting at a terminal is good, then TEN must be better, because MORE is ALWAYS better! But most terminals can only comfortably fit TWO programmers, so we'll call it PAIR programming!"

You have to cut them a little slack; they'd been dealing with the corporate equivalent of pre-schoolers for years, and that really messes with a person.

But the thing is, viruses are really hard to kill, especially the meme kind. After everyone had gotten all worked up about this whole Agile thing (and sure, everyone wants to be more productive), there was a lot of face to be lost by admitting failure. So some other kinds of Agile "Methodologies" sprang up, and they all claimed that even though all the other ones were busted, their method worked!

I mean, go look at some of their sites. Tell me that's not an infomercial. C'mon, just try. It's embarrassing even to look at the thing.

Yeah. Well, they make money hand over fist, because of P.T. Barnum's Law, just like Scientology does. Can't really fault 'em. Some people are just dying to be parted with their cash. And their dignity.

The rest of us have all known that Agile Methodologies are stupid, by application of any of the following well-known laws of marketing:

- anything that calls itself a "Methodology" is stupid, on general principle.
- anything that requires "evangelists" and offers seminars, exists soley for the purpose of making money.
- anything that never mentions any competition or alternatives is dubiously self-serving.
- anything that does diagrams with hand-wavy math is stupid, on general principle.

And by "stupid", I mean it's "incredibly brilliant marketing targeted at stupid people."

In any case, the consultants kept going with their road shows and glossy pamphlets. Initially, I'm sure they went after corporations; they were looking to sign flexible contracts that allowed them to deliver "whatever" in "2 weeks" on a recurring basis until the client went bankrupt. But I'm equally sure they couldn't find many clients dumb enough to sign such a contract.

That's when the consultants decided to take their road show to YOU. Why not take it inside the companies and sell it there, to the developers? There are plenty of companies who use the whip-cycle of development I outlined above, so presumably some of the middle managers and tech leads would be amenable to hearing about how there's this low-cost way out of their hellish existence.

And that, friends, was exactly, precisely the point at which they went from "harmless buffoons" to "potentially dangerous", because before they were just bilking fat companies too stupid to develop their own software, but now the manager down the hall from me might get infected. And most places don't have a very good quarantine mechanism for this rather awkward situation: i.e., an otherwise smart manager has become "ill", and is waving XP books and index cards and spouting stuff about how much more productive his team is on account of all this newfound extra bureaucracy.

How do we know it's not more productive? Well, it's a slippery problem. Observe that it must be a slippery problem, or it all would have been debunked fair and square by now. But it's exceptionally difficult to measure software developer productivity, for all sorts of famous reasons. And it's even harder to perform anything resembling a valid scientific experiment in software development. You can't have the same team do the same project twice; a bunch of stuff changes the second time around. You can't have 2 teams do the same project; it's too hard to control all the variables, and it's prohibitively expensive to try it in any case. The same team doing 2 different projects in a row isn't an experiment either.

About the best you can do is gather statistical data across a lot of teams doing a lot of projects, and try to identify similarities, and perform some regressions, and hope you find some meaningful correlations. But where does the data come from? Companies aren't going to give you their internal data, if they even keep that kind of thing around. Most don't; they cover up their schedule failures and they move on, ever optimistic.

Well if you can't do experiments and you can't do proofs, there isn't much science going on. That's why it's a slippery problem. It's why fad diets are still enormously popular. People want fad diets to work, oh boy you bet they do, even I want them to work. And you can point to all these statistically meaningless anecdotes about how Joe lost 35 pounds on this one diet, and all those people who desperately want to be thinner will think "hey, it can't hurt. I'll give it a try."

That is exactly what I hear people say, every time a team talks themselves into trying an Agile Methodology. It's not a coincidence.

But writing about Bad Agile alone is almost guaranteed to be ineffective. I mean, you can write about how lame Scientology is, or how lame fad diets are, but it's not clear that you're changing anyone's mind. Quitting a viral meme is harder than quitting smoking. I've done both. In order to have the right impact, you have to offer an alternative, and I didn't have one before, not one that I could articulate clearly.

One of the (many) problems with Bad Agile is that they condescendingly lump all non-Agile development practices together into two buckets: Waterfall and Cowboy. Waterfall is known to be bad; I hope we can just take that as an axiom today. But what about so-called Cowboy programming, which the Agileers define as "each member of the team does what he or she thinks is best"?

Is it true that this is the only other development process? And is Cowboy Programming actually bad? They say it as if it's obviously bad, but they're not super clear on how or why, other than to assert that it's, you know, "chaos".

Well, as I mentioned, over the past year I've had the opportunity to watch both Bad Agile and Good Agile in motion, and I've asked the teams and tech leads (using both the Bad and Good forms) lots of questions: how they're doing, how they're feeling, how their process is working. I was really curious, in part because I'd consented to try Agile last Christmas ("hey, it can't hurt"), and wound up arguing with a teammate over exactly what metadata is allowed on index cards before giving up in disgust. Also in part because I had some friends on a team who were getting kind of exhausted from what appeared to be a Death March, and that kind of thing doesn't seem to happen very often at Google.

So I dug in, and for a year, I watched and learned.

The Good Kind

(cue happy rat)

I'm going to talk a little about Google's software development process. It's not the whole picture, of course, but it should suffice for today. I've been there for almost a year and a half now, and it took a while, but I think I get it now. Mostly. I'm still learning. But I'll share what I've got so far.

From a high level, Google's process probably does look like chaos to someone from a more traditional software development company. As a newcomer, some of the things that leap out at you include:

- there are managers, sort of, but most of them code at least half-time, making them more like tech leads.

- developers can switch teams and/or projects any time they want, no questions asked; just say the word and the movers will show up the next day to put you in your new office with your new team.

- Google has a philosophy of not ever telling developers what to work on, and they take it pretty seriously.

- developers are strongly encouraged to spend 20% of their time (and I mean their M-F, 8-5 time, not weekends or personal time) working on whatever they want, as long as it's not their main project.

- there aren't very many meetings. I'd say an average developer attends perhaps 3 meetings a week, including their 1:1 with their lead.

- it's quiet. Engineers are quietly focused on their work, as individuals or sometimes in little groups or 2 to 5.

- there aren't Gantt charts or date-task-owner spreadsheets or any other visible project-management artifacts in evidence, not that I've ever seen.

- even during the relatively rare crunch periods, people still go get lunch and dinner, which are (famously) always free and tasty, and they don't work insane hours unless they want to.

These are generalizations, sure. Old-timers will no doubt have a slightly different view, just as my view of Amazon is slightly biased by having been there in 1998 when it was a pretty crazy place. But I think most Googlers would agree that my generalizations here are pretty accurate.

How could this ever work?

I get that question a lot. Heck, I asked it myself. What's to stop engineers from leaving all the trouble projects, leaving behind bug-ridden operational nightmares? What keeps engineers working towards the corporate goals if they can work on whatever they want? How do the most important projects get staffed appropriately? How do engineers not get so fat that they routinely get stuck in stairwells and have to be cut out by the Fire Department?

I'll answer the latter question briefly, then get to the others. In short: we have this thing called the Noogler Fifteen, named after the Frosh Fifteen: the 15 pounds that many college freshmen put on when they arrive in the land of Stress and Pizza. Google has solved the problem by lubricating the stairwells.

As to the rest of your questions, I think most of them have the same small number of answers.

First, and arguably most importantly, Google drives behavior through incentives. Engineers working on important projects are, on average, rewarded more than those on less-important projects. You can choose to work on a far-fetched research-y kind of project that may never be practical to anyone, but the work will have to be a reward unto itself. If it turns out you were right and everyone else was wrong (the startup's dream), and your little project turns out to be tremendously impactful, then you'll be rewarded for it. Guaranteed.

The rewards and incentives are too numerous to talk about here, but the financial incentives range from gift certificates and massage coupons up through giant bonuses and stock grants, where I won't define "giant" precisely, but think of Google's scale and let your imagination run a bit wild, and you probably won't miss the mark by much.

There are other incentives. One is that Google a peer-review oriented culture, and earning the respect of your peers means a lot there. More than it does at other places, I think. This is in part because it's just the way the culture works; it's something that was put in place early on and has managed to become habitual. It's also true because your peers are so damn smart that earning their respect is a huge deal. And it's true because your actual performance review is almost entirely based on your peer reviews, so it has an indirect financial impact on you.

Another incentive is that every quarter, without fail, they have a long all-hands in which they show every single project that launched to everyone, and put up the names and faces of the teams (always small) who launched each one, and everyone applauds. Gives me a tingle just to think about it. Google takes launching very seriously, and I think that being recognized for launching something cool might be the strongest incentive across the company. At least it feels that way to me.

And there are still other incentives; the list goes on and ON and ON; the perks are over the top, and the rewards are over the top, and everything there is so comically over the top that you have no choice, as an outsider, but to assume that everything the recruiter is telling you is a baldfaced lie, because there's no possible way a company could be that generous to all of its employees, all of them, I mean even the contractors who clean the micro-kitchens, they get these totally awesome "Google Micro-Kitchen Staff" shirts and fleeces.

There is nothing like it on the face of this earth. I could talk for hours, days about how amazing it is to work at Google, and I wouldn't be done. And they're not done either. Every week it seems like there's a new perk, a new benefit, a new improvement, a new survey asking us all if there's any possible way in which life at Google could be better.

I might have been mistaken, actually. Having your name and picture up on that big screen at End of Quarter may not be the biggest incentive. The thing that drives the right behavior at Google, more than anything else, more than all the other things combined, is gratitude. You can't help but want to do your absolute best for Google; you feel like you owe it to them for taking such incredibly good care of you.

OK, incentives. You've got the idea. Sort of. I mean, you have a sketch of it. When friends who aren't at Google ask me how it is working at Google — and this applies to all my friends at all other companies equally, not just companies I've worked at — I feel just how you'd feel if you'd just gotten out of prison, and your prison buddies, all of whom were sentenced in their early teens, are writing to you and asking you what it's like "on the outside". I mean, what would you tell them?

I tell 'em it's not too bad at all. Can't complain. Pretty decent, all in all.

Although the incentive-based culture is a huge factor in making things work the way they do, it only addresses how to get engineers to work on the "right" things. It doesn't address how to get those things done efficiently and effectively. So I'll tell you a little about how they approach projects.

Emergent Properties versus The Whip

The basic idea behind project management is that you drive a project to completion. It's an overt process, a shepherding: by dint of leadership, and organization, and sheer force of will, you cause something to happen that wouldn't otherwise have happened on its own.

Project management comes in many flavors, from lightweight to heavyweight, but all flavors share the property that they are external forces acting on an organization.

At Google, projects launch because it's the least-energy state for the system.

Before I go on, I'll concede that this is a pretty bold claim, and that it's not entirely true. We do have project managers and product managers and people managers and tech leads and so on. But the amount of energy they need to add to the system is far less than what's typically needed in our industry. It's more of an occasional nudge than a full-fledged continuous push. Once in a while, a team needs a bigger nudge, and senior management needs to come in and do the nudging, just like anywhere else. But there's no pushing.

Incidentally, Google is a polite company, so there's no yelling, nor wailing and gnashing of teeth, nor escalation and finger-pointing, nor any of the artifacts produced at companies where senior management yells a lot. Hobbes tells us that organizations reflect their leaders; we all know that. The folks up top at Google are polite, hence so is everyone else.

Anyway, I claimed that launching projects is the natural state that Google's internal ecosystem tends towards, and it's because they pump so much energy into pointing people in that direction. All your needs are taken care of so that you can focus, and as I've described, there are lots of incentives for focusing on things that Google likes.

So launches become an emergent property of the system.

This eliminates the need for a bunch of standard project management ideas and methods: all the ones concerned with dealing with slackers, calling bluffs on estimates, forcing people to come to consensus on shared design issues, and so on. You don't need "war team meetings," and you don't need status reports. You don't need them because people are already incented to do the right things and to work together well.

The project management techniques that Google does use are more like oil than fuel: things to let the project keep running smoothly, as opposed to things that force the project to move forward. There are plenty of meeting rooms, and there's plenty of open space for people to go chat. Teams are always situated close together in fishbowl-style open seating, so that pair programming happens exactly when it's needed (say 5% of the time), and never otherwise.

Google generally recognizes that the middle of the day is prone to interruptions, even at quiet companies, so many engineers are likely to shift their hours and come in very early or stay very late in order to find time to truly concentrate on programming. So meetings only happen in the middle of the day; it's very unusual to see a meeting start before 10am or after 4:30pm. Scheduling meetings outside that band necessarily eats into the time when engineers are actually trying to implement the things they're meeting about, so they don't do it.

Google isn't the only place where projects are run this way. Two other kinds of organizations leap to mind when you think of Google's approach: startup companies, and grad schools. Google can be considered a fusion of the startup and grad-school mentalities: on the one hand, it's a hurry-up, let's get something out now, do the simplest thing that could work and we'll grow it later startup-style approach. On the other, it's relatively relaxed and low-key; we have hard problems to solve that nobody else has ever solved, but it's a marathon not a sprint, and focusing requires deep concentration, not frenzied meetings. And at the intersection of the two, startups and grad schools are both fertile innovation ground in which the participants carry a great deal of individual responsibility for the outcome.

It's all been done before; the only thing that's really surprising is that Google has managed to make it scale.

The scaling is not an accident. Google works really hard on the problem, and they realize that having scaled this far is no guarantee it'll continue, so they're vigilant. That's a good word for it. They're always on the lookout to make sure the way of life and the overall level of productivity continue (or even improve) as they grow.

Google is an exceptionally disciplined company, from a software-engineering perspective. They take things like unit testing, design documents and code reviews more seriously than any other company I've even heard about. They work hard to keep their house in order at all times, and there are strict rules and guidelines in place that prevent engineers and teams from doing things their own way. The result: the whole code base looks the same, so switching teams and sharing code are both far easier than they are at other places.

And engineers need great tools, of course, so Google hires great people to build their tools, and they encourage engineers (using incentives) to pitch in on tools work whenever they have an inclination in that direction. The result: Google has great tools, world-class tools, and they just keep getting better.

The list goes on. I could talk for days about the amazing rigor behind Google's approach to software engineering. But the main takeaway is that their scaling (both technological and organizational) is not an accident. And once you're up to speed on the Google way of doing things, it all proceeds fairly effortlessly — again, on average, and compared to software development at many other companies.

The Tyranny of the Calendar

We're almost done. The last thing I want to talk about here is dates. Traditional software development can safely be called Date-Oriented Programming, almost without exception.

Startup companies have a clock set by their investors and their budget. Big clients set target dates for their consultants. Sales people and product managers set target dates based on their evaluation of market conditions. Engineers set dates based on estimates of previous work that seems similar. All estimation is done through rose-colored glasses, and everyone forgets just how painful it was the last time around.

Everyone picks dates out of the air. "This feels like it should take about 3 weeks." "It sure would be nice to have this available for customers by beginning of Q4." "Let's try to have that done by tomorrow."

Most of us in our industry are date-driven. There's always a next milestone, always a deadline, always some date-driven goal to it.

The only exceptions I can think of to this rule are:

1) Open-source software projects.
2) Grad school projects.
3) Google.

Most people take it for granted that you want to pick a date. Even my favorite book on software project management, "The Mythical Man-Month", assumes that you need schedule estimates.

If you're in the habit of pre-announcing your software, then the general public usually wants a timeframe, which implies a date. This is, I think, one of the reasons Google tends not to pre-announce. They really do understand that you can't rush good cooking, you can't rush babies out, and you can't rush software development.

If the three exceptions I listed above aren't driven by dates, then what drives them? To some extent it's just the creative urge, the desire to produce things; all good engineers have it. (There are many people in our industry who do this gig "for a living", and they go home and don't think about it until the next day. Open source software exists precisely because there are people who are better than that.)

But let's be careful: it's not just the creative urge; that's not always directed enough, and it's not always incentive enough. Google is unquestionably driven by time, in the sense that they want things done "as fast as possible". They have many fierce, brilliant competitors, and they have to slake their thirsty investors' need for growth, and each of us has some long-term plans and deliverables we'd like to see come to fruition in our lifetimes.

The difference is that Google isn't foolish enough or presumptuous enough to claim to know how long stuff should take. So the only company-wide dates I'm ever aware of are the ends of each quarter, because everyone's scrambling to get on that big launch screen and get the applause and gifts and bonuses and team trips and all the other good that comes of launching things with big impact at Google.

Everything in between is just a continuum of days, in which everyone works at optimal productivity, which is different for each person. We all have work-life balance choices to make, and Google is a place where any reasonable choice you make can be accommodated, and can be rewarding. Optimal productivity is also a function of training, and Google offers tons of it, including dozens of tech talks every week by internal and external speakers, all of which are archived permanently so you can view them whenever you like. Google gives you access to any resources you need in order to get your job done, or to learn how to get your job done. And optimal productivity is partly a function of the machine and context in which you're operating: the quality of your code base, your tools, your documentation, your computing platform, your teammates, even the quality of the time you have during the day, which should be food-filled and largely free of interrupts.

Then all you need is a work queue. That's it. You want hand-wavy math? I've got it in abundance: software development modeled on queuing theory. Not too far off the mark, though; many folks in our industry have noticed that organizational models are a lot like software models.

With nothing more than a work queue (a priority queue, of course), you immediately attain most of the supposedly magical benefits of Agile Methodologies. And make no mistake, it's better to have it in software than on a bunch of index cards. If you're not convinced, then I will steal your index cards.

With a priority queue, you have a dumping-ground for any and all ideas (and bugs) that people suggest as the project unfolds. No engineer is ever idle, unless the queue is empty, which by definition means the project has launched. Tasks can be suspended and resumed simply by putting them back in the queue with appropriate notes or documentation. You always know how much work is left, and if you like, you can make time estimates based on the remaining tasks. You can examine closed work items to infer anything from bug regression rates to (if you like) individual productivity. You can see which tasks are often passed over, which can help you discover root causes of pain in the organization. A work queue is completely transparent, so there is minimal risk of accidental duplication of work.

And so on. The list goes on, and on, and on.

Unfortunately, a work queue doesn't make for a good marketing platform for seminars and conferences. It's not glamorous. It sounds a lot like a pile of work, because that's exactly what it is.

Bad Agile in More Detail

I've outlined, at a very high level, one company's approach to software development that is neither an Agile Methodology, nor a Waterfall cycle, nor yet Cowboy Programming. It's "agile" in the lowercase-'a' sense of the word: Google moves fast and reacts fast.

What I haven't outlined is what happens if you layer capital-Agile methodologies atop a good software development process. You might be tempted to think: "well, it can't hurt!" I even had a brief fling with it myself last year.

The short answer is: it hurts. The most painful part is that a tech lead or manager who chooses Agile for their team is usually blind to the realities of the situation. Bad Agile hurts teams in several ways.

First, Bad Agile focuses on dates in the worst possible way: short cycles, quick deliverables, frequent estimates and re-estimates. The cycles can be anywhere from a month (which is probably tolerable) down to a day in the worst cases. It's a nicely idealistic view of the world.

In the real world, every single participant on a project is, as it turns out, a human being. We have up days and down days. Some days you have so much energy you feel you could code for 18 hours straight. Some days you have a ton of energy, but you just don't feel like focusing on coding. Some days you're just exhausted. Everyone has a biological clock and a a biorhythm that they have very little control over, and it's likely to be phase-shifted from the team clock, if the team clock is ticking in days or half-weeks.

Not to mention your personal clock: the events happening outside your work life that occasionally demand your attention during work hours.

None of that matters in Bad Agile. If you're feeling up the day after a big deliverable, you're not going to code like crazy; you're going to pace yourself because you need to make sure you have reserve energy for the next big sprint. This impedance mismatch drives great engineers to mediocrity.

There's also your extracurricular clock: the set of things you want to accomplish in addition to your main project: often important cleanups or other things that will ultimately improve your whole team's productivity. Bad Agile is exceptionally bad at handling this, and usually winds up reserving large blocks of time after big milestones for everyone to catch up on their side-project time, whether they're feeling creative or not. Bad Agile folks keep their eye on the goal, which hurts innovation. Sure, they'll reserve time for everyone to clean up their own code base, but they're not going to be so altruistic as to help anyone else in the company. How can you, when you're effectively operating in a permanent day-for-day slip?

Bad Agile seems for some reason to be embraced by early risers. I think there's some mystical relationship between the personality traits of "wakes up before dawn", "likes static typing but not type inference", "is organized to the point of being anal", "likes team meetings", and "likes Bad Agile". I'm not quite sure what it is, but I see it a lot.

Most engineers are not early risers. I know a team that has to come in for an 8:00am meeting at least once (maybe several times) a week. Then they sit like zombies in front of their email until lunch. Then they go home and take a nap. Then they come in at night and work, but they're bleary-eyed and look perpetually exhausted. When I talk to them, they're usually cheery enough, but they usually don't finish their sentences.

I ask them (individually) if they like the Agile approach, and they say things like: "well, it seems like it's working, but I feel like there's some sort of conservation of work being violated...", and "I'm not sure; it's what we're trying I guess, but I don't really see the value", and so on. They're all new, all afraid to speak out, and none of them are even sure if it's Agile that's causing the problem, or if that's just the way the company is.

That, my friends, is not "agile"; it's a just load of hooey. And it's what you get whenever any manager anywhere decides to be a chump.

Good Agile Should Drop the Name

I would caution you to be skeptical of two kinds of claims:

- "all the good stuff he described is really Agile"
- "all the bad stuff he described is the fault of the team's execution of the process"

You'll hear them time and again. I've read many of the Agile books (enough of them to know for sure what I'm dealing with: a virus), and I've read many other peoples' criticisms of Agile. Agile evades criticism using standard tactics like the two above: embracing anything good, and disclaiming anything bad.

If a process is potentially good, but 90+% of the time smart and well-intentioned people screw it up, then it's a bad process. So they can only say it's the team's fault so many times before it's not really the team's fault.

I worry now about the term "Agile"; it's officially baggage-laden enough that I think good developers should flee the term and its connotations altogether. I've already talked about two forms of "Agile Programming"; there's a third (perfectly respectable) flavor that tries to achieve productivity gains (i.e. "Agility") through technology. Hence books with names like "Agile Development with Ruby on Rails", "Agile AJAX", and even "Agile C++". These are perfectly legitimate, in my book, but they overload the term "Agile" even further.

And frankly, most Agile out there is plain old Bad Agile.

So if I were you, I'd take Agile off your resume. I'd quietly close the SCRUM and XP books and lock them away. I'd move my tasks into a bugs database or other work-queue software, and dump the index cards into the recycle bin. I'd work as fast as I can to eliminate Agile from my organization.

And then I'd focus on being agile.

But that's just my take on it, and it's 4:00am. Feel free to draw your own conclusions. Either way, I don't think I'm going to be an Early Riser tomorrow.

Oh, I almost forgot the obvious disclaimer: I do not speak for Google. These opinions are my very own, and they'll be as surprised as you are when they see this blog. Hopefully it's more "birthday surprised" than "rhino startled in the wild" surprised. We'll see!

Blogger's Block #4: Ruby and Java and Stuff

Part 4 of a 4-part series of short posts intended to clear out my bloggestive tract. Hold your nose!

Well, I held out for a week. Then I read the comments. Argh! Actually they were fine. Nice comments, all around. Whew.

I don't have any big themes to talk about today, but I've got a couple of little ones, let's call 'em bloguettes, that I'll lump together into a medley for today's entree.

Bloguette #1: Ruby Sneaks up on Python

I was in Barnes today, doing my usual weekend stroll through the tech section. Helps me keep up on the latest trends. And wouldn't you know it, I skipped a few weeks there, and suddenly Ruby and Rails have almost as many books out as Python. I counted eleven Ruby/RoR titles tonight, and thirteen for Python (including one Zope book). And Ruby had a big display section at the end of one of the shelves.

Not all the publishers were O'Reilly and Pragmatic Press. I'm pretty sure there were two or three others there, so it's not just a plot by Tim O'Reilly to sell more books. Well, actually that's exactly what it is, but it's based on actual market research that led him to the conclusion that Rails and Ruby are both gathering steam like nobody's business.

I like a lot of languages. Really, I do. But I use Ruby. I'm not even sure if I like Ruby. The issue might just be irrelevant to whether I use it. I like OCaml, for instance, but I don't use it. I don't like Java, but I do use it. Liking and using are mostly orthogonal dimensions, and if you like the language you're using even a little bit, you're lucky. That, or you just haven't gotten broad enough exposure to know how miserable you ought to be.

I use Ruby because it's been the path of least resistance for most of my programming tasks since about 3 days after I started messing with it, maybe 4 years ago.

I don't even really know Ruby all that well. I never bothered to learn it. I did read "Ruby in a Nutshell" cover-to-cover, but it's a short read (and it's a bit out of date now.) Then I read bits of "Programming Ruby", but not all of it. And now I use Ruby for everything I can, any time I have any choice in the matter. I don't even mind that I don't know the language all that well. It has a tiny core that serves me admirably well, and it's easy to look things up when you need to.

I do a lot more programming in Python than in Ruby -- Jython in my game server, and Python at work, since that's what everyone there uses for scripting. I have maybe 3x more experience with Python than with Ruby (and 10x more experience with Perl). But Perl and Python both have more unnecessary conceptual overhead, so I find I have to consult the docs more often with both of them. And when all's said and done, Ruby code generally winds up being the most direct and succinct, whether it's mine or someone else's.

I have a lot of trouble writing about Ruby, because I find there's nothing to say. It's why I almost never post to the O'Reilly Ruby blog. Ruby seems so self-explanatory to me. It makes it almost boring; you try to focus on Ruby and you wind up talking about some problem domain instead of the language. I think that's the goal of all programming languages, but so far Ruby's one of the few to succeed at it so well.

If only it performed better. *Sigh*. Well, its performance is in the same class as Perl/Python/JavaScript/Lua/Bash/etc., so there are still plenty of tasks Ruby's admirably suited for.

I think next year Ruby's going to be muscling in on Perl in terms of mindshare, or shelf-share, at B&N.

Bloguette #2: Java's Biggest Failing (Literally)

I still do most of my programming in Java -- at least half of it, maybe more. The Java platform continues to make amazing strides. The newest incarnation (JDK 6) has lots of goodies I can't wait to play with. Like Rhino, for instance, and although they appear to have gutted it, it's still awesome. I think it's the best choice they possibly could have made. Thank God they didn't bundle Groovy. What a catastrophe that was, and still is, and would have been for Java if they'd bundled it. Rhino rocks.

The JVM is just getting faster and more stable, and there are even some OK libraries that come with it. I used to think the Java platform libraries were the cat's meow. Heck, I thought they were the whole damn cat. But working with better libraries in miscellaneous other languages has got me thinking that Java's libraries are hit-or-miss.

Example: Java's concurrency libraries (java.util.concurrent[.*]) are to die for. I mean, if you're stuck with threads. I think in the fullness of time, hand-managed threads will be history, but in the meantime, Java's concurrency libraries are just superb.

I recently ported a medium-sized Python program I'd written (about 1200 lines of fairly dense Python code) to Java, because the Python was taking about an hour to run, and I wanted to parallelize the work. I spent about 3 days doing the rewrite: one day on the straight port, a day adding in the threading, and a day fine-tuning it. The straight port wound up as 1300 lines of Java (surprising that it wasn't bigger, but maybe I code in Python with a Java accent?), and ran about 50% faster, down to about 30 minutes. After adding in the threading and state machine, the program ran in 50 to 60 seconds.

So I got an order of magnitude improvement with only about a 50% increase overall in program size. The vast majority of the improvement was attributable to the threading, which in turn would have taken me FAR longer if I'd been using raw synchronization primitives. The java.util.concurrent stuff made it a snap.

On the other hand, Java's DOM implementation completely blows chunks. It quickly became the bottleneck in my application, due to an O(n) algorithm I stumbled across with no good workaround for. I can't remember exactly where it was (this was back in July), but I found a sheepishly apologetic comment from the author in the online docs. It was something to do with setting attributes on nodes while you're doing a traversal of some sort: something you'd definitely want to be fast, but it had at least linear performance, maybe worse, and now accounts for 95+% of my app's processing time.

And of course Java's DOM interface blows too, because you can't create subclasses or decorators or do anything useful with the DOM other than use it as a temp container until you've transfered the data to something more flexible.

Java's collections library is decent, but not superb. It's nice having the data structures they provide, but they're not very configurable, and the language itself makes them often cumbersome. For instance, you can have a WeakHashMap (nice), or an IdentityHashMap (nice), or a ConcurrentHashMap (also nice), but you can't combine any two of those three properties into a single hashtable. Lame.

And java.util is missing implementations and/or interfaces for a bunch of important data types like priority queues (you're stuck using a TreeSet, which is overkill), the disjoint set ADT, splay trees, bloom filters, multi-maps, and of course any kind of built-in graph support. Java hyper-enthusiasts will tell you: "well, go write your own! Or use one of the many hopefully robust implementations on the web!" That seems lame to me. We're talking about data structures here: they're more fundamental than, say, LDAP libraries and much of the other stuff Sun's bundling these days. It's smartest to provide robust, tuned implementations of these things, because it empowers average Java programmers to write faster, more reliable code.

Oh, and let's not even get me started with java.nio. What a mess! It's pretty gross, especially if you come from the comparatively simple background of select() and poll() on Unix. But maybe the grossness was necessary. I'll give them the benefit of the doubt. What bugs me isn't that the API is conceptually weird and complex (and buggy as hell last time I checked); what bugs me is that nobody at Sun bothered to put a layer atop java.nio for ordinary programmers. Like, say, a nonblocking DataInputStream that takes a type to read, a Buffer, and a callback to call when it's finished reading. So every frigging Java programmer on the planet has to write that exact class -- or just flail around with the raw APIs, which is what I think most of them do.

And look what they did to poor LDAP! I mean, the LDAP bindings are dirt-simple in every language I've ever used. It's supposed to be lightweight -- that's what the "L" stands for, fer cryin' out loud. JNDI is this huge monster. So is JMX. I mean, Java libraries have this way of being so bloated and overengineered.

But whatever; I've digressed. Java's libraries are not its biggest failing. The libraries (as I said) are decent, and the platform (in terms of tools, speed, reliability, documentation, portability, monitoring, etc.) really raises the bar on all those other loser languages out there. All of 'em. It's why no better languages have managed to supplant Java yet. Even if the language and its libraries are (on the whole) better than Java's, they also have to contend with the Java platform, and so far nobody's been able to touch it, unless maybe it's .NET, but who cares about .NET? Certainly not Amazon.com or Yahoo! or Google or any other important companies that I'm aware of.

Literals

Anyway, Java's biggest failing, I've decided, is its lack of syntax for literal data objects. It's an umbrella failing that accounts for most of the issues I have with the language.

The idea behind literals is that you have some sort of serialized notation for your data type, and it's part of the language syntax, so you can embed pre-initialized objects in your code.

The most obvious ones are numbers, booleans and strings. It's hard to imagine life without support for numeric literals, isn't it? Well, Java's support is limited at best. There's no syntax for entering a binary value, for instance, like "0b10010100". And there's no BigInteger/BigDecimal syntax, so working with them is a disaster and nobody does it if they can help it. Heck, Java doesn't even have unsigned ints and longs. But Java does more or less the bare minimum for numbers, so people don't notice it much.

Imagine if there were no String literals, so that instead of this:

String s = "Hello, world!";

you had to do this:

  StringBuffer sb = new StringBuffer();
  sb.append('H');
  sb.append('e');
  sb.append('l');
  sb.append('l');
  sb.append('o');
  sb.append(',');
  sb.append(' ');
  sb.append('W').append('o').append('r').append('l').append('d').append('!');
  String s = sb.toString();

Not only is the latter bloated and ugly and error-prone (can you spot the error in mine?), it's also butt-slow. Literals provide the compiler with opportunities for optimization.

Well, unfortunately this OOP garbage is exactly what you have to do when you're initializing a hashtable in Java. Nearly all other languages these days have support for hashtable/hashmap literals, something like:

 my_hashmap = {
  "key1" : "value1",
  "key2" : "value2",
  "key3" : "value3",
  ...
}

That's the syntax used by Python and JavaScript, but other languages are similar. The Java equivalent is this:

Map<String, String> my_hashmap = new HashMap<String, String>();
my_hashmap.put("key1", "value1");
my_hashmap.put("key2", "value2");
my_hashmap.put("key3", "value3");
...

It might not look that much worse from this simple example, but there are definitely problems. One is optimization; the compiler is unlikely to be able to optimize all these method calls, whereas with a literal syntax, it could potentially save on method call overhead during construction of the table (and maybe other savings as well.)

Another is nested data structures. In JavaScript (and Python, Ruby, etc.) you just declare them in a nested fashion, like so:

my_thingy = {
  "key1": { "foo": "bar", "foo2": "bar2"},
  "key2": ["this", "is", "a", "literal", "array"],
  "key3": 37.5,
  "key4": "Hello, world!",
   ...
}

It would be hard to do this particular one in Java 5 because of the mixed value types, though it's probably not an issue since using mixed-type data structures is something you rarely do in practice, even in dynamically-typed languages. But even if all the values were hashes of string-to-string, how are you going to do it in Java without literals? You can't. You're stuck with:

Map<String, Map<String, String>> my_hashmap = \
   new HashMap<String, HashMap<String, String>>();

Map<String, String> value = new HashMap<String, String>();
value.put("foo", "bar");
value.put("foo2", "bar2");
my_hashmap.put("key1, value);

value.clear();
value.put("foo3", "bar3");
value.put("foo4", "bar4");
my_hashmap.put("key2, value);

... 

And then you find out later that your clever clear() optimization (instead of creating a new HashMap object for each value) busted it completely. Whee.

Java programmers wind up dealing with this kind of thing by writing generic helper functions, and it winds up layering even more OOP overhead onto something that ought to be a simple declaration. It also tends to be brutally slow; e.g. you could write a function called buildHashMap that took an array of {key, value, key, value, ...}, but it adds a huge constant-factor overhead.

This is why Java programmers rely on XML so heavily, and it imposes both an impedance mismatch (XML is not Java, so you have to translate back and forth) and a performance penalty.

But the story doesn't end there. What about Vector/ArrayList literals? Java has primitive array literals, which is nice as far as it goes:

String[] s = new String[]{"fee", "fi", "fo", "fum"};

Unfortunately, Java's primitive arrays are a huge wart; they don't have methods, can't be subclassed, and basically fall entirely outside the supposedly beautiful OOP-land that Java has created. It was for performance, to help capture skeptical C++ programmers, and they have their place. But I don't see why they should have all the syntactic support. I mean, the [] array-indexing operator is ONLY available for Java arrays. Sure would be nice to have it for ArrayLists, wouldn't it? And Strings? And FileInputStreams?

But for some reason, Java gave arrays not one, but TWO syntactic sugarings, and then didn't give that sugar to anything else array-like in the language.

So for building ArrayLists, LinkedLists, TreeMaps and the like, you're stuck with Swing-style code assemblages.

I think of them as Swing-style because I used to do a lot of AWT and Swing programming, back when I was a Thick Client kind of guy, and they have a distinct(ly unpleasant) footprint. It looks vaguely like this, in pseudo-Swing:

Panel p = new Panel(new FlowLayout());
JButton b = new JButton("Press me!");
b.setEventListener(somethingOrOther);
p.add(b);
JSomething foo = new JSomething(blah, blah);
foo.setAttribute();
foo.setOtherAttribute();
foo.soGladIDontDoThisKindOfThingAnymore();
p.add(foo);
...

Building UIs in Swing is this huge, festering gob of object instantiations and method calls. It's OOP at its absolute worst. So people have come up with minilanguages (like the TableLayout), and declarative XML replacements like Apache Jelly, and other ways to try to ease the pain.

I was on a team at Amazon many years ago that was planning to port a big internal Swing application to the web, and we were looking at the various ways to do web programming, which at the time (for Java) were pretty much limited to JSP, WebMacro, and rolling your own Swing-like HTML component library.

We experimented with the OOP approach to HTML generation and quickly discarded it as unmaintainable. (Tell that to any OOP fanatic and watch their face contort as they try to reconcile their conflicting ideas about what constitutes good programming practice.)

The right solution in this case is, of course, a Lisp dialect; Lisp really shines at this sort of thing. But Lisp isn't so hot at algebraic expressions, and the best Lisp machines no longer look so cutting-edge compared to the JVM, and blah blah blah, so people don't use Lisp. So it goes.

The next-best solutions are all about equally bad. You have your XML-language approaches (like Jelly, but for the web), but they don't give you sufficient expressiveness for control flow -- presentation logic really does require code, and it gets ugly in XML in a real hurry. You have your JSP-style templating approaches, and they aren't bad, but they can have as many as 4 or 5 different languages mixed in the same source file, which presents various problems for your tools (both the IDEs and the batch tools).

And then you have a long tail of other approaches, none of which manage to be very satisfying, but that's not really the fault of the languages. It's the browsers' fault: they START with three languages (HTML, CSS, and JavaScript), rather than having just one language to control the entire presentation, and it only goes downhill from there.

But NONE of the approaches to web templating is as bad as Swing-style programming, with a huge thicket of calls to new(), addChild(), setAttribute(), addListener(), and the like. The only approach that's worse (and even it might just be tied) is raw HTML printing:

print("<html><body>...</body></html>");

So we're all in agreement. OOP-style assembly of parents and children is the worst way to generate HTML. You want to use declarations; you want a template, something that visually looks like the end result you're trying to create.

Well, it's the exact same situation for data structures, isn't it? You'd rather draw a picture of it (in a sense, that's exactly what you're doing with syntax for literals) than write a bunch of code to assemble it. This is all assuming that you're working with a small data set, of course. But that happens all the time in real-world programs; it's ubiquitous. So you kinda want your language to support it syntactically.

And so far we've only covered literal syntax for HashMaps and ArrayLists (which you can combine to produce various kinds of custom Trees.) Already Java's way behind other languages, and we haven't discussed any richer data types.

Like, say, objects.

JavaScript does it the best here, IMO, in the parity between hashes and objects. It's not really possible in Ruby or Python to declare a class, then create instances of the class using literal notation the way you can in JavaScript, where the keys are the names of instance variables. Fortunately you can accomplish this in either Ruby or Python with just a smidge of metaprogramming, so it's spilt milk at worst.

In Java, you only have one big hammer (instantiation), and one big wrench (the method call), so that's what you use. All you can really do to help is create a constructor that takes arguments that populate the instance variables. But if any of your instance variables are collections (other than arrays), then you're back to the old create-setprops-addchild, create-setprops-addchild pattern again.

And what about functions? Ruby and JavaScript and Lisp and Scheme and Lua and Haskell and OCaml and most other self-respecting languages have function literals. That is, they have a syntax for declaring an instance of a function as a data object in your code that you can assign to a variable, or pass as a parameter.

(Python has them too, but unfortunately they can only be one line, so Python folks prefer to pretend anonymous functions aren't very important. This is one of the 10 or so big problems caused by Python's whitespace policy. Don't ever let 'em tell you it doesn't cause problems. It does. Maybe it's worth the trade-off; that's a personal style preference, but they should at least admit the tradeoff exists.)

Well, Java sort of has them, but Java's static type system doesn't have a literal syntax for a method signature. It's pretty easy to imagine one, e.g. something like:

(int, int) -> String x;

This imaginary syntax declares a variable x that takes 2 ints as parameters and returns a string. Lots of languages have signature-syntax of some sort, and Java's syntax space is definitely sparse enough that they could pick a good syntax for it without fear of collisions, even conceptual collisions. But no such luck. Instead, when you want to do this sort of thing you have to declare a named interface, and then inside of it declare at least one named method (which is where the params and return type show up), and then you're still not done, because when you create the function you have to create an anonymous (or named) class that contains the definition of the function that matches the interface.

Yuck. But at least they let you do it; the alternative of not having it at all is definitely worse.

Still... isn't syntactic sugar nice? I mean, they added the "smart" for-loop, which Java programmers just rave about. So someone, somewhere in the Java community thinks syntax is good. I'm not sure many of them really understand the difference between syntactic sugar (into which category the "smart" for-loop falls) and orthogonal syntax, in which the basic operators apply to all data types for which those operators make sense, and there are literal declarations possible for every data type.

Let alone the next step, which is extensible syntax -- but that idea strikes fear into the hearts of many otherwise brave Java programmers, and Rubyists and Pythonistas as well, so let's back it up a notch to "orthogonal", and keep everyone calm.

So there you have it: Java's biggest failing. It's the literals. No literal syntax for array-lists (or linked lists or tree sets), nothing for hashtables, nothing for objects of classes you've personally defined, none for functions or function signatures. Java programmers all around the world spend a *lot* of their time working around the problem, using XML and YAML and JSON and other non-Java data-declaration languages, and writing tons of code (whole frameworks, even) for serializing and deserializing these declarations to and from Java. For the smaller stuff, they just write helper functions, which wind up being bloated, inefficient, error-prone, and extremely unsatisfying.

Java's next-biggest failing may well be the lack of orthogonality in its set of operators. We can live without operator overloading, I suppose (the simplest form of extensible syntax), but only if Sun makes operators like [] and + actually work for objects other than arrays and Strings, respectively. Jeez.

Epiblogue

You can draw your own conclusions about why suddenly there are all these books on Ruby appearing on the bookshelves. It's a mix of truths, no doubt. And you can draw your own conclusions about why Sun's adding support for scripting languages to the JVM, rather than simply fixing Java so that people don't want (need, really) to use those other languages.

But when you dig down into a programming language, and you get past all the hype and the hooplah, what you find is a set of policies and decisions that affect your everyday life as a programmer in ways you can't ignore, and that no amount of hype will smooth over.

If your language is sitting on you like an invisible elephant, and everyone using the language is struggling to work around the same problems, then it's inevitable that other languages will come into play. Libraries can make you more productive, but they have almost no effect on the scalability of the language.

Every language has a complexity ceiling, and it's determined by a whole slew of policy and design decisions within the language, not the libraries. The slew includes the type system (with its attendant hundreds of mini-policies), and the syntax, and it also includes the language's consistency: the ratio of rules to exceptions.

Java's demonstrating quite clearly that at a certain level of complexity, the libraries and frameworks start to collapse under their own weight. People are always writing "lightweight" replacements for existing fat Java libraries and frameworks, and then the replacements get replaced, ad infinitum. But have you ever seen anyone write a replacement for XPath? Nope. It's not like everyone is rushing out to write the next big XML-querying framework. This is because XPath is a language, not a library, and it's orders of magnitude more conceptually scalable than the equivalent DOM manipulations.

Object-Oriented Programming. Touted even by skeptics as a radical leap forward in productivity, and all OOP really is boils down to a set of organizational techniques. Organization is nice, sure.

But it's pretty clear that OOP alone doesn't cut it; it has to be supplemented with Language-Oriented Programming and DSLs. And all languages, DSLs and general-purpose languages alike, have to be designed to maximize consistency; each inconsistency and special-case in the language adds to its conceptual overhead and lowers the complexity ceiling.

So you can look at the shelves filling up with Ruby books and chalk it up to marketing hype, but I have a different theory. I think it's entirely due to complexity management: Ruby does a better job of helping managing complexity than its competitors. It doesn't do a perfect job, mind you -- far from it. But it's enough of a step forward in productivity (even over Perl and Python) that it's managing to shoulder its way in to a pretty crowded language space.

With that in mind, despite my griping about Java's failings, I think Sun might actually be doing the right thing by introducing scripting languages (and improving support for them in the JVM.) Maybe. Their investment isn't really so much in Java as it is in the JVM; the JVM is their .NET. Java's not really about productivity, not really -- it's got a lot of strengths (performance, deployment, reliability, static checkability, and so on), but productivity isn't high on the list. So maybe the best way to address the productivity issue, for folks who really need it more than raw performance, is to introduce new JVM languages rather than try to pull Java in two directions.

We'll see. And with that, I think I've officially un-blocked myself; I seem to be able to blog again. So I'm declaring the Blogger's Block series finished!

BloggersBlock block = new BloggersBlock();
block.setFinished(true);
block.tieOffAndStuff();
blog.addChild(block);
...

Blogger's Block #3: Dreaming in Browser Swamp

Part 3 of an N-part series of short (well, fast, anyway) posts intended to clear out my bloggestive tract.

I've been doing a lot of JavaScript and DHTML and AJAX programming lately. Increasing quantities of it. Boy howdy. The O'Reilly DHTML book has gotten big enough to crush a Volkswagon Bug, hasn't it? And my CSS book has gone from pristine to war-torn in under a month. I've managed to stay in the Dark Ages of web programming for the past 10 years: you know, HTML, a little CGI, font color="red". Way old school. And now I'm getting the crash-course. The more I learn, the more I wish I'd known it for longer. Although then I'd have had to live through the long transition from Dark Ages to the muchly-improved situation we have today. Far from good, to be sure, but it's improved dramatically since last I looked.

JavaScript is probably the most important language in the world today. Funny, huh? You'd think it would be Java or C++ or something. But I think it just might be JavaScript.

For one thing, despite JavaScript's inevitable quirks and flaws and warts and hairy boogers and severe body odor, it possesses that magical property that you can get stuff done really fast with it. Well, if you can find any halfway decent tools and libraries for it, that is. Which you can't, not without effort. But the situation is improving. Slow and steady wins the race and all. JavaScript is definitely the tortoise to Java's hare.

See, JavaScript has a captive audience. It's one of those languages you just have to know, or you get to miss out on Web programming, and in case you hadn't noticed, thick clients are like Big Hair these days. Most non-technical people I know pretty much live in their browsers, and they only emerge periodically to stare in puzzlement at iTunes or a game or something, and wonder why isn't it in the browser, because everything else useful seems to be. It's where the whole world is. To non-technical people, of course. Which is, like, practically everyone.

We technical folks like to eye browsers with suspicion, and for a good reason. They're not platforms. They're sort of like platforms, but they're missing all this stuff you normally expect from platforms. The DHTML book (which covers pretty much the only semi-reliable intersection of the browser platforms) is, for all its massive size, still just one book, and any platform worth its salt will need a whole shelf.

It turns out if you dig deep into Mozilla (aka Netscape, aka Firefox, aka SeaMonkey, aka SwampMonster, I mean the thing really has way too many farging names already), you'll find that it actually is a relatively full-featured platform. It's not quite as general-purpose as an OS (or Java), but it's certainly big and hairy enough to be making threats in that general direction.

But my God, it's sooooooo ugly. It's got well over a decade of ugly packed in there. "Hello, World" in Mozilla is six or seven files in as many different languages. I kid you not. It's worse than Hello, World was back in the Petzold days of Win32 programming. You have your XUL file and your JavaScript file and your CSS file and your manifest.rdf and your i18n.something and I can't remember what all else. And then you have to build them together (using some other files) to make even more files: a JAR file and an XPI file at a minimum. That's one gnarly-ass introductory program.

Me, I kinda prefer Python's version:

print "Hello, world!"

Color me silly with font="red", but it just seems cleaner to me.

And then when you try to graduate from Hello, World to something that seems like it should be only epsilon more difficult, like, say, "Get me a list of the user's bookmarks", you officially launch off into this eerie XPCOM world where you have to write dozens and dozens of lines of JavaScript code that looks nothing at all like you'd imagine it should, if you closed your eyes and thought to yourself: "I wonder what the JS code for getting bookmarks would look like?"

Look, I'll even show ya:

/**
 * Returns a sorted list of bookmark objects.  Each object
 * has properties name, url, and (optional) kw.
 * @return bookmark list:  an array of anonymous objects, each
 * with "name", "url" and "kw" fields.  Sorts the list by name
 * if sort is true.
 */
function getBookmarkList(doSort) {
  var rdf = Components.classes["@mozilla.org/rdf/rdf-service;1"].
    getService(Components.interfaces.nsIRDFService);
    
  var bmks = rdf.GetDataSource("rdf:bookmarks");
  var NC_NS = "http://home.netscape.com/NC-rdf#";
  var kwArc = rdf.GetResource(NC_NS + "ShortcutURL");
  var urlArc = rdf.GetResource(NC_NS + "URL");
  var nameArc = rdf.GetResource(NC_NS + "Name");
  var rdfLiteral = Components.interfaces.nsIRDFLiteral;
  var e = bmks.GetAllResources();
  var items = [];

  while(e.hasMoreElements()) {
    var r =  e.getNext().QueryInterface(
        Components.interfaces.nsIRDFResource);
    var urlR = bmks.GetTarget(r, urlArc, true);
    var kwR = bmks.GetTarget(r, kwArc, true);
    var nameR = bmks.GetTarget(r, nameArc, true);
    if (!(nameR && urlR)) {
      continue;
    }
    var item = {};
    item.name = nameR.QueryInterface(rdfLiteral).Value;
    item.url = urlR.QueryInterface(rdfLiteral).Value;
    if (kwR) {
      item.kw = kwR.QueryInterface(rdfLiteral).Value;
    }
    items.push(item);
  }

  if (doSort) {
    items.sort(function(a, b) {
        return (a.name.upcase() < b.name.upcase()) ? -1 : 1;
      });
  }
  return items;
}

Shouldn't there be a "getBookmarks()" in there somewhere? I mean, what is all that crap?

Admittedly this function does a little extra, what with the sorting, but that's only 5 lines of code, because JavaScript came with verbs included, thank you so much Brendan. And those 5 lines are actually fairly sane; I mean, you can read them and say "hey, that's sorting!" But as the rest of the code, what we seem to have is a serious failure to separate infrastructure from business logic. We're trying to take the patient's temperature here, and they're making us saw the poor guy open and grub around in his intestines looking for the right spot to stick the thermometer. So to speak.

In any case, that's not really JavaScript's fault; it's Firefox's fault. I have trouble keeping them separate in my head sometimes, because I, unlike the rest of the Free World, have the balls not to support Internet Explorer. For my personal stuff, anyway. (Or so I believe at the moment. Time will tell.)

That's what's really holding JavaScript back, you know. And it's holding back CSS, and DOM, and all the other standards. We have this little impasse problem. You know the song and dance: Microsoft didn't want Netscape to be a competing platform, so they built IE and gave it away for free, and for a brief while they even had a better product, so they captured enough market share.

And then... nothing. Oh, sure, they've made a few half-assed attempts to make IE standards-compliant, sort of, but only after making many full-assed attempts to distort those standards to give Microsoft competitive advantages. I've heard that directly from folks working on the relevant teams over there. Microsoft cheerfully shows up at the standards meetings to make damn sure they screw up the APIs for everyone else. You know. Microsoft-style. Sorta like how DirectX was bugly compared to OpenGL. Or Win32 compared to *nix. Or MFC compared to any sane object system (e.g. TurboPascal and TurboC). Or COM compared to CORBA. (I mean, you have to work hard to be worse than CORBA.) Microsoft has always been awful at making APIs, always always always, and I've decided over the years to credit this to malice rather than incompetence. Microsoft isn't incompetent, whatever else they might be. Burdened, yes; incompetent, no.

Why am I talking about them side-by-side with JavaScript? Because the standoff between Microsoft and the Forces of Neutrality (open standards and the like) is the main thing that's holding JavaScript back. Nobody wants to build an amazingly cool website that only works in FireFox/Opera/(insert your favorite reasonably standards-compliant browser here). Because they're focused on the short term, not the long term. It would only take one or two really killer apps for Mozilla to take back the market share from Microsoft. That, or a whole army of pretty good ones. People don't like downloading new stuff (in general), and they also don't like switching browsers. But they'll do it if they know they have to in order to use their favorite app.

Everyone knows all this; not a jot of it is news to anyone, but nobody wants to be the one to make a clean break from IE. It might bankrupt them. Great app, nobody sees it, company goes bust. So the killer apps will have to come from the fringe, the margin, the anarchy projects engineers do on the side — at least at companies where engineers have a little free time for innovation. Excepting only go-for-broke startups, most places can't (or won't) bet the farm on a Firefox-only application. So even though the spec is moving forward, or maybe sideways, DHTML in the real world has been in near-stasis for years.

In any event, the whole MSIE standoff isn't the only thing holding JavaScript back. There are definitely other contributors.

One big problem is that it's JavaScript. Nobody wants to use JavaScript.

I'm serious. It's not that it's a bad language (it's not); it's just not the language they want to use. You know. Them. You. Everyone who has a favorite programming language. Most people only want to use one, their favorite, whatever they're best with, and when they switch to a different one, they're slower. They feel stifled, held back, uncomfortable. That feeling goes away in under a month of immersion in a new language, but most engineers begrudge that time fiercely, probably because they don't realize it's only a month.

So many folks who take a stab at browser programming wind up saying "oh GOSH, why can't I use BLUB, man this really SUCKS!" and then instead of writing their killer web app, they go off and write some lame-ass toolkit that compiles their language into JavaScript, or tries to be a Firefox plugin, or tries to be a Frankensteinian compiled-together monstrosity like Apache and mod_perl was.

The funny thing is, they wouldn't be gaining a damn thing with a new language, except maybe libraries, but even that's somewhat doubtful. The standard library and runtime for any respectable general-purpose programming language are pretty big: definitely too big to expect everyone in the world to download. And that's the problem. You'd have to get everyone to do it. Might as well work on your own browser at that point. Oh, you can bet a bunch of people run off to do that, too, utterly overlooking the fact that nobody will use it even if it's great (like Opera), because it's not the browser they're accustomed to, and it has no killer app.

But if they wrote a killer app... oh, but that would have to be in JavaScript... well, maybe we can come up with a way to write it in XYZ instead! That's right — a bunch of would-be web app programmers are stuck in a vicious circle trying to break into a market that has one of the weirdest monopolies in history: IE (and to a tiny extent, Firefox and Safari: the Dr. Pepper and RC Cola of browsers, respectively) maintains its monopolistic lock not through overt control, but through apathy on the part of users worldwide.

And that apathy extends to the browser makers themselves. Microsoft and Apple have no reason to try to make their browsers competing platforms; quite the opposite in fact. So almost no innovation happens in IE and Safari, compared to the innovation (in Apple's case) or purchase and/or me-tooing of other innovations (in Microsoft's) going on outside the browser groups at those companies.

OK, but what about Firefox? Why don't they, you know, innovate? Well, they're trying, I think, but for what I'm guessing are probably tangled historical reasons — which manifest as the developers often being gridlocked politically — Mozilla lacks what Fred Brooks Jr. calls "conceptual integrity" in his classic "The Mythical Man-Month". [Which, incidentally, remains today the most vitally relevant book on software engineering, over 30 years after it was written.] The Mozilla folks would have to do a lot of serious re-thinking in order to reduce XUL's "Hello, World" down to a few lines of code in a single language. And I'm not convinced that kind of thinking is happening in the Firefox camp right now. It's not that they're not thinking at all; don't get me wrong. They're just not thinking about radical, revolutionary user-level simplifications to the basic framework.

Like, the kind of radical simplification Ruby on Rails introduced for server-side web programming. Yeah, yeah, I know, you hate Ruby because it's not the language you learned at your mother's teat, for God's sake. Despite Ruby's near-perfect Conceptual Integrity Index, you'd sooner quit your job and become a sanitation engineer than spend a day or three just learning the language and being done with it. But whatever your feelings on Ruby, Rails has caused a huge stir, because it took something that everyone assumed had to be ugly now and for all time, and it built layers and scaffolding on top that smoothed over a tremendous amount of that ugliness. Made it almost purty, even.

Rails — now that's the kind of simplification I'm talking about. Screw all the XML minilanguages (XUL, XBL, X*L). Screw the RDF. You need to be able to do everything in JavaScript. JSON is just good enough and parseable enough and language-interoperable enough to replace all of them. For that matter, screw CSS. I don't mean the CSS spec, not the relatively elegant constraint system they have in place; I just mean the CSS syntax, since it's one more language piled onto the heap. The whole selector minilanguage is nifty, but does it really need to be different from XPath? I mean, aren't they both doing path expressions to select things in the DOM? Jeez!

Am I spouting heresy or common sense here? It depends who you're asking. There are a lot of "web developers" in the world. Lots and lots. And they, my friends, are the last thing holding JavaScript back.

Because nobody wants to be a web developer. No self-respecting, rootin'-tootin', core-dump-debugging programmer wants to be one, anyway. That's sissy stuff. That's what most non-web programmers think, whether they use Java or C++ or C# or Perl or Python or Haskell or what-have-you. There's a deep-seated cultural perception problem in our industry about this, and I think it has a lot of root causes.

One cause, and let's be honest here, is that a lot of web developers were self-taught, weaned from text to HTML to onclick="foo.hide()" and onwards to CSS and DOM and more complex JavaScript, and thence on to CGI and PHP and VB and ActiveX and SVG and Flash and the rest of the gigantic mess of barely-interoperable languages we have to work with for web programming, not one of which covers the whole spectrum the way C++ or Java does. You have to mix and match them (always poorly) in order to achieve some effect that usually would have been trivial using a thick-client framework.

So we really have two reasons at play here: one is that web developers mostly taught themselves, which means they're generally not very good at what "real" programmers of course consider "real" programming, so diving into web programming hasn't been considered very glamorous, nor a very good career move. The other is that the field is so littered with new, ugly languages and technologies (again, not one of which is a turn-key solution) that most folks who do try it wind up fleeing.

I mean, I'm all in favor of MVC, but I think Common Sense and its kissing-cousin Conceptual Integrity will both tell you that M, V and C don't need to be three separate languages. And the manifest and build information needn't be a fourth and fifth, respectively. And i18n a sixth. And the XPCOM system services an effective seventh. And the server-side languages Nth through Zth. It just doesn't make any sense.

Which is in no small part why you keep hearing about Ruby on Rails, even though you really just wish it would go away so you don't have to learn it. Rails has the laws of physics (or at least economics) on its side. Rails is like one of those bizarre tunnel diodes, where the electrons on one side of the barrier tunnel to the other side without traveling the distance in between (at least in the classical-mechanics sense of "traveling"). They just sort of appear on the other side, instantaneously, because being on the other side is a lower quantum energy state, and tough shit if you don't like it.

People want Rails to go away because it has Ruby in the name: you can't use it like a library, nooooo, it has to be in some other language, and other languages are bad by axiomatic decree. Programmers are lazy: they've found that the greedy algorithm gets them there with the least energy expenditure most of the time, and switching languages requires much more energy than integrating a new library, no matter how godawfully complex and ultimately useless the library is. Which is why the Java community is thrashing around with like 50 competing frameworks for server side web programming. Oh, and the Python community too. And, um, all of the rest of them too.

Well, if you happen to be doing web programming, Ruby on Rails defies classical language mechanics by actually being a lower energy state. That's right; it's more lazy to learn Rails than it is to try to get your web framework to be that productive, so people are just tunneling over to it like so many electrons.

This phenomenon will happen in the browser space. I can assure you it will. It's an economic certainty. There's money at play here, lots and LOTS of money; every company in the world wants a cool website. Not just a cool website; they want cool apps. Companies are realizing — glacially, yes, belatedly, yes, but inexorably — that most people with computer access in the world today live inside their browser, and they'd prefer not to leave it.

"Everyone in the world" — that's an awful lot of money at stake.

So as soon as "Scheme on Skis" or "JavaScript on Jets" or whatever comes along, that Rails-like radical simplification of the huge ugly Browser Swamp, the game will change almost overnight.

I'm not sure exactly how it'll pan out. Rails is basically a big code generator, a big compiler, in a sense. The "language" is Rails itself — there's precious little actual Ruby in a Rails app, surprisingly enough, although there's tons underneath — and the target platform is the Browser Swamp. It's not a seamless abstraction; you still need to know CSS and HTML (at least), and you need to know a fair bit about HTTP and relational databases and web servers and all that crapola you need for "Hello, World" on the web.

But maybe that was the right choice. Maybe if DHH had defined an entire DSL for web programming in pure Ruby, with all the DHTML auto-generated, it wouldn't have been as popular. In the long run, I think the pure-Ruby approach (or pure-anything, as long as it's a single language that supports declarative programming, which rules Java out) is economically superior, because there's less to learn, and more purchase for optimizations, error-checking and the like. But in the short term, meaning today, the hybrid approach seems to be doing well.

The alternative to a Rails-like multi-language hybrid is to do the whole ball of wax in a single language. Lisp and Scheme folks have, of course, warmed to this idea, and they all write macros that generate HTML, which winds up being way cleaner than you might guess. But not many frameworks have taken the approach of generating all the JavaScript. The GWT is one, of course, but you'd have to be a pretty doggurn die-hard Java programmer to go that way. I'm sure it'll improve with time, but the biggest stumbling block, amusingly enough, is that it's not JavaScript. JavaScript is still King in the browser space, and ironically it's like programming to the "bare metal" compared to using a Java-to-JavaScript compiler. So JavaScript what the "real" web programmers prefer to use.

And we've come full circle.

Isn't this great? Me, I'm loving it. Yeah, it's a crap sandwich and we all have to take a bite. But it's also a greenfield opportunity, a land rush, a high-energy state just dying to become a low-energy state. Who's going to solve it? What will the solution look like?

One option I haven't really discussed is the incumbent: maybe the standards committees will eventually evolve cross-browser support into a platform that doesn't drive most programmers into I'll-write-my-own-dammit frenzies. I mean, it's a LOT better than it was in 1997. Look at Google Spreadsheets or Writely or GMail or Google Maps (anyone notice a pattern here?) — would any of these really have been possible in 1997? Heck, I doubt it: they're barely possible today. But they make it pretty obvious which way thick clients are headed, don't they?

Hang on — you're not still a thick-client programmer, are you? Oh dear. You'd better get yourself a DHTML book and an AJAX book, and right-quick. Oh, you're above that client stuff, you're a server-side programmer then, are you? Bully for you! I don't blame you. I hid there for many years myself; server-side is a haven of sanity, isn't it? But I think you'll find that adding web programming to your skills lineup will go a long way; you'll be able to wire up those nifty backends you're writing in ways that real-live people will appreciate. Like, say, your family. And real employers too. They like the web. There's money there. So DHTML/AJAX a great Mixin skill; it complements just about anything else you know how to do.

Anyway, I've pretty much belched out the barest outlines of my browser-ish thoughts for today: just enough for a blog entry, so I'll wrap up.

Ooooh, and it's a Blogger's Block entry! This Blogger's Block series has been great for the ol' creative juices. Just write about whatever you want, no worries, and it all flows nicely.

The only thing that could possibly go wrong is me reading the comments.

What?

I had a realization last night as I was juuuust falling asleep. The realization I had was that I have all my realizations just as I'm falling asleep. Or just as I'm waking up. There's something magical about that time, and I think I know what it is.

I think you (or at least "I", but I think maybe "we") are highly creative when we're nearly asleep because all the pressure's off. You can't go to sleep if you're under a lot of pressure — not easily, at any rate. Nobody can follow us into our dreams, so we go there alone, and when you're alone with yourself, you can be yourself. We've put shields in place to prevent us from saying or doing stupid things in public, and those shields come down when we're asleep.

That's why you always have those amazing dreams that you want to write down in the morning, the ones that would make a great movie screenplay or spy-thriller novel. Your mind is in a creative frenzy, and you mostly suppress it during the day.

Well, you can't say stuff in a public blog without getting some criticism, which is why most employees at most companies (that's you, an all likelihood) are reticent to try it. There's not much to be gained, and a lot of potential downside. People have been fired over their blogs, for instance, although those are relatively rare cases, and the blogger in question is almost invariably a jackass. For most bloggers it's more subtle. If you're speaking in a public forum and sufficient people appear to be listening, then it's hard not to be perceived as a spokesperson (of sorts) for the company. Scoble's the canonical example, but you can easily find others who fit the profile.

If your company has an ad-hoc, out-of-band, quasi-pseudo-spokesperson on the loose, the person is a risk. Sort of. I mean, it's lose-lose. If the person is cheerleading, well, nobody likes a fanboy. (It's a truly ugly word, isn't it? "Fanboy" is the new F-word.) If it's badmouthing, well, that's not too cool either. But hey, one person's fawning is another person's badmouthing, right? There are two sides to any interesting opinion (otherwise it's not going to be very interesting), so anything a blogger says in public is going to be criticized by some percentage of the readers. So no matter what, the company the blogger works for is taking some indirect heat, and it's risky to hope that the positive side of the blog, if any, will make up for it.

Well then. That's why bloggers like me always have to give you the following disclaimer: I don't speak for Google. Not even a little bit. For the reasons I've outlined above, the best I can really hope for is that they choose to look the other way when I blog. It's the best any blogger can hope for. It's what I'm hoping for.

I'm a frigging Google fanboy, though; that's going to be really hard to hide, so I'll just be honest with you. F-word, I hate that F-word. But it fits in this case. I'll try my best to be objective. Google's a terrible place to be when you're on a diet, for instance.

In any case, my sleepy realization has effectively solved my Blogger's Block problem. I couldn't write because I was too worried about what other people (i.e., you) would think. Yeah, I care about you too much. You! I think that gradually piled on resistance, and it was getting harder and harder for me to write past it. You can see it pretty clearly in the "Clothes for the Soul" post, which I don't care all that much about — it was just a thought-exercise, after all, and was supposed to be fun. But then I got all defensive at the end, which made commenters all defensive, and I made a mess, all because I was trying so hard to avoid criticism.

So the solution is simple: I won't read the comments.

ZZZZZzzzzzzzooooooooompf — and just like that, in a flash, I'm alone with my thoughts. Amazing. Really. I can feel a chill; it's like I didn't know I was in a haunted house until all the ghosts left, all at once.

La la la, La la la, I can say whatever I want, and I needn't know 'til I'm dead what anyone else thought of it. Nobody mentions my blog at work, by and large, since it's awkward to do so. (I assume this is the case for all bloggers — we don't have any cultural conventions for it, so it's like the third time you pass someone in the hall and you both carefully avoid eye contact.) So if I don't read the comments, then I really am alone with my thoughts.

That's how I was able to write some of my more interesting things back at Amazon; initially nobody read my blog, so I was writing for an audience of at most about 5 people, and then only if I pestered them to read it, which I almost never did.

So no reading comments. My blogs may not be any better for it, but I'm sure I'll be much happier. Oh, you can bet I'll be tempted. Maybe I won't be able to resist. Must... *pant* ... resiiiiist... augh! But I think I'll be able to hold out. Why? Because for this entire entry I've felt like I'm just about to fall asleep. Well, that's because I was; it's been a long day, and I didn't start until 1am, and it's 4:30am now. So yeah. Sleeeep.

But it's a nice feeling. Much nicer than the Russian Roulette of reading the comments on my blogs. "Uh-huh, uh-huh, yep, *BANG* aaaaah!" <blood pressure shoots up to 190/150>. High blood pressure is a recipe for some pretty questionable blogging, I think, and it's not too good for your health, either.

So! Tell me what you th... er, tell others what you think! I'll be hiding under my desk, hoping the monsters go away.

And learn DHTML! You won't like it, but you'll be glad all the same.

'night.

---

p.s. some light reading: