16 February 2010

Misreading Data

The world is full of data - the trick is to understand what it means.

For example, some people point to the huge DC snowstorm as proof that global warming is a myth. Um, first of all, data isn't proof. It's evidence. And second, the storm might actually be evidence of a warming trend, because warmer air can hold more moisture and thus create larger storms. Or maybe the blizzard is just the result of an El Nino pattern and it might have nothing to do with climate change at all, man-made or otherwise. And please understand - this post is not about global warming. It's about understanding & interpreting data.

Me, I'm convinced data never speaks for itself. It must always be processed and interpreted. That's particularly true on projects where we collect metrics. And there are many kinds of metrics - predictive, descriptive, etc. Sometimes we simply collect the obvious & easy ones, which may or may not tell us anything about the project team's performance. Other times we misread good data. It can be done well - but it takes deliberate effort (and a grain of salt).

I don't have an easy answer - all I'm trying to say here is that we're fooling ourselves if we think the data is self-interpretive. We're taking a dangerous shortcut when we fail to reflect on the data and the many ways it can be shaped (or mis-shaped), interpreted (or mis-interpreted).


Anonymous said...

Not about Global Warming? Interesting how you make statements from the "there must be global warming" side of the fence and how about. There is not global warming. The Westminster Dog Show is going on. Hey, there was this big snow storm in DC. All three are unrelated....see? All three have data that proves them true. AGREE - data is available for my statements - there is no data for global warming.

The Dan Ward said...

Hmmm... I'm not sure whether to respond to this anonymous comment, ignore it or delete it.

How about I use it to make a related point about data: we humans have a tendency to pay attention to data that supports our beliefs, and to ignore or deny data that does not. This happens on projects as well as in scientific arguments.

So, when this nameless writer says "there is no data for global warming," he or she is arguing that a huge quantity of actual scientific data does not exist. Now, the collected body of centuries-worth of climate data may or may not support global warming. It may or may not be convincing. It may or may not be proof. But the data does exist. Thermometers and log books have been around for a long time. We have information about historical temperatures. What the data *means* is a whole other question... which is precisely the point of the original post.

Similarly, in a system development project, we may say "there is no data that the project is having problems," when in fact the data might actually exist... but it hasn't been interpreted or understood yet.

As another example, the mysterious commenter says I "make statements from the 'there must be global warming' side of the fence..." when I actually mention arguments from both sides. This assertion leads me to suspect (but I cannot prove) that the writer does not believe global warming is happening, and because of this belief he or she interprets my post to be about global warming & its reality.

It's not about global warming - not at all. It's about interpreting data and metrics. It's about the danger of believing that "the data speaks for itself." It's about project leadership and the need to be reflective, thoughtful & deliberate in the way we collect, understand and apply data.

Is it getting hot in here or is it just me?

David said...

Well, Dan, I've made a living interpreting data, and I have to agree with you here: Very often, delving into the data provides very counterintuitive facts. Not results, facts. You should never be afraid to go where the data and science lead you. Sound bites are of no real interest.

I've said it before: Metrics can be useful, but you have to be clear on the question, and the manner in which you answer it. And the assumptions you make along the way.

As an interesting aside, many of the global warming experts predicted higher variation in *weather* as an effect of *climate* change. I find it interesting that anyone would use an accurate prediction to refute an assertion by an expert...

Enjoy the Blog -- haven't visited in a bit (dissertation is sucking my time like a vacuum) but I do enjoy it when I do.