My first thought was - huh? What about all the GAO reports that say (over and over and over again) that military acquisition projects take too long, cost too much, and under perform when fielded.
And then I realized that it's entirely possible for both to be true.
When we say a project comes in on time & on budget, it sounds like a good thing. But what if it had too much time & money to start with? It's possible to spent too much money AND be on budget.
And what do we really mean by "on budget." Does that mean we delivered the stated capability using the original amount of funding? Or are we taking credit for staying under a budget line that has increased over the years, as additional funds get added to the baseline (because we've increased the schedule, added requirements, etc)? If we define "on budget" broadly enough, then I'll bet almost everyone is on budget. Maybe even 95% of projects.
But there's more to this story. The idea that 95% of projects are on time & on budget doesn't actually mean as much as it might seem. Imagine a portfolio of 10 projects. Nine of them each cost $100, and each deliver on time, on budget. The 10th project was supposed to cost $100,000, but ended up coming in at $150,000. Do we praise this portfolio for being "90% on budget," or do we acknowledge it spent $50K more than planned? I'm thinking the latter assessment is more accurate.
The flaw in the 95% on budget reasoning is that it is measuring performance on a per-attempt basis, using a straight count of the number of projects. What we should be measuring is the performance of our dollars (i.e. the proverbial bang for the buck).
Similarly, let's say we have a project that has 10 steps. Nine of the steps each take one day to accomplish. The 10th step takes a year. At the end of nine days, if we say we're 90% done with the project, because we have finished nine of the steps, we're seriously misrepresenting how much work has been done (and how much is remaining).
Statistics are funny things. It's entirely possible for a statistic to be true and misleading all at once. It's important to be careful and make sure the things we're measuring and reporting provide an accurate representation of the situation.
5 comments:
75% of statistics quoted in blogs are made up on the spot.
@Mark ... 3 out of 4 experts would agree with you!
@Dan ... nice lesson on earned value management too! I'll tell you from my PEM days, I looked at the portfolio and relied on my good programs to average out the bad actors. Not a great way to do business, but it was a direct result of how our success was measured (the almighty OSD Obs & Exps goals...).
Dan,
A statistic is, by definition, just a function of data. Doesn't have to make sense or be meaningful to be a statistic. Usually we're interested in summary statistics, as a filter for useful information. Summarizing usually involves a loss of information.
When a summary statistic is given you should ask a few questions:
1. What data is it based on? Was there a different set of data available to get me the same information? Why was this set of data chosen?
2. What basis is there for applying this function to this data? Mathematical? Historical?
3. How does this statistic help me see a bigger picture? Does it reasonably and accurately represent that bigger picture?
4. Is there bias in the way that the data is summarized?
5. What information was lost in the summarization?
The example you give has problems across the board, some of which you accurately highlight in your questions.
The ethical and useful application of statistics is not trivial -- even experienced people who ought to know better obscure important information by oversummarizing (or worse).
You should always be wary of statistics...and I say this as a statistician.
Love the blog --
dave
This stat sounds like a bunch of PMs are good at change control more than anything else.
@Dave - Great comments - thanks much! I really like the observation that summarization involves a loss of information.
@Craig - Yeah, I think that's the main take-away from that stat. Unfortunately, I'm afraid some people try to use the stat to say "See what a good job we're doing!"
Post a Comment