Earned Value Management

As a follow-on to yesterday's post, I'd like to say a few words in praise of something called Earned Value Management (aka EVM).

EVM is basically a way of measuring progress in terms of actual work accomplished, not simply in terms of the number of discrete tasks that have been marked Done. It seems to me that EVM is just another word for honest accounting.

It boggles the mind that a project leader would use anything other than EVM to measure, assess and report progress towards their goals.

Some people try to say it's too complicated, too cumbersome and frankly too dang hard to do EVM. I've also heard that EVM isn't well suited for small projects, but in fact there are ways to do simple implementations, without relying on excessive overhead. For that matter, the simple implementations can probably be useful even on big projects.

Now, applying EVM to certain types of research projects, where future work paths are unknown, requires a slightly different tack... but it's possible to use some of the principles even in this area.

EVM - it's not just for breakfast anymore. Check it out!

Statistics

I recently read a quote that said something along the lines of "95% of acquisitions are done on time, on budget."

My first thought was - huh? What about all the GAO reports that say (over and over and over again) that military acquisition projects take too long, cost too much, and under perform when fielded.

And then I realized that it's entirely possible for both to be true.

When we say a project comes in on time & on budget, it sounds like a good thing. But what if it had too much time & money to start with? It's possible to spent too much money AND be on budget.

And what do we really mean by "on budget." Does that mean we delivered the stated capability using the original amount of funding? Or are we taking credit for staying under a budget line that has increased over the years, as additional funds get added to the baseline (because we've increased the schedule, added requirements, etc)? If we define "on budget" broadly enough, then I'll bet almost everyone is on budget. Maybe even 95% of projects.

But there's more to this story. The idea that 95% of projects are on time & on budget doesn't actually mean as much as it might seem. Imagine a portfolio of 10 projects. Nine of them each cost $100, and each deliver on time, on budget. The 10th project was supposed to cost $100,000, but ended up coming in at $150,000. Do we praise this portfolio for being "90% on budget," or do we acknowledge it spent $50K more than planned? I'm thinking the latter assessment is more accurate.

The flaw in the 95% on budget reasoning is that it is measuring performance on a per-attempt basis, using a straight count of the number of projects. What we should be measuring is the performance of our dollars (i.e. the proverbial bang for the buck).

Similarly, let's say we have a project that has 10 steps. Nine of the steps each take one day to accomplish. The 10th step takes a year. At the end of nine days, if we say we're 90% done with the project, because we have finished nine of the steps, we're seriously misrepresenting how much work has been done (and how much is remaining).

Statistics are funny things. It's entirely possible for a statistic to be true and misleading all at once. It's important to be careful and make sure the things we're measuring and reporting provide an accurate representation of the situation.