The collected year

One of my nerdy little ideas for this year is that I'm going to start collating and storing all the great bits of writing that I stumble over during the year, but which tend to get consigned to the ever-growing pile of New Yorkers in our house, or the depths of Instapaper.

The first piece to enter my anthology is Atul Gawande's 'Testing, Testing', from the December 14 2009 issue of the New Yorker. The strapline for the article asks: "The health-care bill has no master plan for curbing costs. Is that a bad thing?"

Gawande's topic is the American healthcare reform package. He points out that medical care currently absorbs 18 cents in every dollar earnt in America, and that if this current trend continues, "the cost of family insurance will reach twenty-seven thousand dollars or more in a decade, taking more than a fifth of every dollar that people earn." He continues:

So what does the reform package do about it? Turn to page 621 of the Senate version, the section entitled “Transforming the Health Care Delivery System,” and start reading. Does the bill end medicine’s destructive piecemeal payment system? Does it replace paying for quantity with paying for quality? Does it institute nationwide structural changes that curb costs and raise quality? It does not. Instead, what it offers is . . . pilot programs.

As Gawande notes, this has been the subject of much criticism. here's the grand master plan? How can this crippled, wasteful system be improved by a bunch of unconnected mini projects?

Gawande then looks at the history of 20th century agricultural reform in America. In 1900, over 40% of a family's income was spent on food. The US government did not institute a grand master plan to overhaul the labour-intensive, un-innovative farming industry. But it did nudge things along by supporting pilot programmes. By 1930 food costs accounted for 24% of family spending, and today it's 8%. Whether you agree or disagree with contemporary farming practices in America - and god knows some of them are bloody ugly - this change has been massively influential.

Reading Gawande's piece has shaped how I've approached two other articles this week. The problem with pilot projects, in my personal experience, is that they're generally projects to try something new that you're given permission to undertake by higher-ups on the basis that (a) they won't cost more than what's available in petty cash and (b) you'll do them on top of your normal work. Often, pilot projects succeed because of the people working on them; because an individual or team with the skills and the commitment makes the choice to undertake them. The problems arise when the pilot is successful, but can't be integrated into the grand master plan - where the budget lives.

I came back to work yesterday and read this post by Koven J Smith, 'On pilot projects and other things that don't work'. In it, Smith argues that

the whole concept of the “pilot project” itself is a fantasy. It’s rarely a project in the conventional sense; it’s a hedge. More often than not, a pilot project is undertaken as a way for technologists to slide a potentially controversial (and yet often technologically mundane) idea past museum administration. It’s a way to fail without actually incurring the costs or benefits of actual failure.

You could replace the word 'technology' there with a few others. Smith concludes:

We need to stop this hedging, and own up to our failures when they occur. If you’re pushing a technology project at your museum, make the case for it at the beginning, rather than hoping that a successful pilot will make the case for you. If you’re proposing the project in the first place, it’s probably because you already have some faith that it will be successful. Don’t ensure that project’s death by committing it to the pilot project graveyard.

Both Gawande and Smith's articles point to the same conclusion - a pilot project can only really succeed when it's evaluated to establish its worth and resourced to become part of what you or your organisation does.

Sometimes though the point of a pilot can be test a hypothesis. In science, a disproved hypothesis isn't a negative outcome, any more than a proven hypothesis is a positive one. It's simply another bit of information that helps guide you towards establishing further pieces of information. Therefore a 'failed' project really should point you towards a different approach.

This is demonstrated in this post by Nina Simon, 'Is Wikipedia Loves Art getting "better"?' The first iteration of the Wikipedia Loves Art project, carried out in the US, was a mixed success. The next iteration, in the Netherlands, made a number of changes - which you could compare to refinements of an experiment - and resulted in a different kind of mixed success. Now a third iteration is about to be undertaken, in Britain.

As Simon notes, the question of whether the project is getting 'better' as a result of these refinements comes down to your judgement about what constitutes 'best' - for example, more freedom for the participating photographers to make their own choices over what to document, versus a what-to-photograph list that reduces personal choice, but also reduces confusion amongst the photographers and the admin overhead for the institutions involved. The interesting thing overall, of course, is that fact that the communities of Wikimedians and collecting institutions didn't give up after the mixed success of the first pilot - they've kept trialling and refining, learning as they go.