What Is Good?

Apparently my theme this week has something to do with measuring progress (I didn't plan to have a theme - it just sorta happened). While previous posts have mostly focused on the measurement part, let's take a look at the "progress" part today.

Specifically, I'd like to talk about priorities and values. What attributes do we consider desirable for our projects? What are our measures of merit? What indicators give us reassurance that we've got a good program?

Regular readers of this blog know I think it's "important and good" for a program to be Fast, Inexpensive, Simple and Tiny. I apply these values to every part of the program, from the requirements documents and system architectures to the organizational structure and operational processes. My research indicates that the FIST values not only support programmatic success (i.e. delivering on time), but also operational success (i.e. performing well in the field).

The funny thing about values is they are often assumed, not discussed. I seldom see explicit statements about a project's values, but there are plenty of indicators that show what the project leaders really think are important.

Look at any number of recently cancelled projects, and I think you'll find that nobody thought it was particularly important for the thing to be developed quickly or inexpensively, or for it to be simple. Rather, complexity is often treated as a sign of sophistication. Big budgets are a sign that the system is important. Take a long time to build it? Great! That means you're clearly doing a good job and exercising due diligence.

Here's the thing: when we aren't deliberate with our values, when we don't examine and discuss them, we run the risk of being guided by values that are counterproductive and ineffective. I'm not saying that FIST is the only value set that works - I'm just saying we should be purposeful with our values, and should be aware of the way they shape our decisions and behavior.

Earned Value Management

As a follow-on to yesterday's post, I'd like to say a few words in praise of something called Earned Value Management (aka EVM).

EVM is basically a way of measuring progress in terms of actual work accomplished, not simply in terms of the number of discrete tasks that have been marked Done. It seems to me that EVM is just another word for honest accounting.

It boggles the mind that a project leader would use anything other than EVM to measure, assess and report progress towards their goals.

Some people try to say it's too complicated, too cumbersome and frankly too dang hard to do EVM. I've also heard that EVM isn't well suited for small projects, but in fact there are ways to do simple implementations, without relying on excessive overhead. For that matter, the simple implementations can probably be useful even on big projects.

Now, applying EVM to certain types of research projects, where future work paths are unknown, requires a slightly different tack... but it's possible to use some of the principles even in this area.

EVM - it's not just for breakfast anymore. Check it out!

Statistics

I recently read a quote that said something along the lines of "95% of acquisitions are done on time, on budget."

My first thought was - huh? What about all the GAO reports that say (over and over and over again) that military acquisition projects take too long, cost too much, and under perform when fielded.

And then I realized that it's entirely possible for both to be true.

When we say a project comes in on time & on budget, it sounds like a good thing. But what if it had too much time & money to start with? It's possible to spent too much money AND be on budget.

And what do we really mean by "on budget." Does that mean we delivered the stated capability using the original amount of funding? Or are we taking credit for staying under a budget line that has increased over the years, as additional funds get added to the baseline (because we've increased the schedule, added requirements, etc)? If we define "on budget" broadly enough, then I'll bet almost everyone is on budget. Maybe even 95% of projects.

But there's more to this story. The idea that 95% of projects are on time & on budget doesn't actually mean as much as it might seem. Imagine a portfolio of 10 projects. Nine of them each cost $100, and each deliver on time, on budget. The 10th project was supposed to cost $100,000, but ended up coming in at $150,000. Do we praise this portfolio for being "90% on budget," or do we acknowledge it spent $50K more than planned? I'm thinking the latter assessment is more accurate.

The flaw in the 95% on budget reasoning is that it is measuring performance on a per-attempt basis, using a straight count of the number of projects. What we should be measuring is the performance of our dollars (i.e. the proverbial bang for the buck).

Similarly, let's say we have a project that has 10 steps. Nine of the steps each take one day to accomplish. The 10th step takes a year. At the end of nine days, if we say we're 90% done with the project, because we have finished nine of the steps, we're seriously misrepresenting how much work has been done (and how much is remaining).

Statistics are funny things. It's entirely possible for a statistic to be true and misleading all at once. It's important to be careful and make sure the things we're measuring and reporting provide an accurate representation of the situation.

The One Thing

If there was one thing I could accomplish professionally, one contribution to the corpus of program management practice and theory, it would be to dispell the Myth of Inevitability.

What is this myth, you ask? It's the idea that high tech system development projects (weapons, spacecraft, commercial products, etc) inevitably take a long time, cost a lot and are complex. The myth is expressed in phrases like "better, faster, cheaper - pick two." A corollary to this myth is the idea that adding time and money to the project improves the outcome, as if the problem with our failed system development efforts was that our schedule was too short and our budget too small.

No kidding -that's what they said when the 18 year, $7B (billion-with-a-b) Comanche helicopter was cancelled. They needed more money. They needed more time. Yeah, that would have helped. Right. A similar chorus rises from any number of failed high tech projects. My assessment is that they had too much time & money, not too little. And at the core, they believed that the costs and delays were inevitable, simply an inherent part of this kind of work. It's a tragic belief and a self-fulfilling prophecy. It's also not grounded in reality.

The truth is, there is nothing inherent in military technology (for example) that requires it to cost so much, take so long or be so complex. Yes, systems like the F-22 took a long time and cost a lot. But there's a difference between "it took a long time" and "it takes a long time." We could have done it better (both programmatically and operationally). If we set up our values correctly, if we cherish our constraints and pursue intelligent simplicity, we don't have to spend billions and decades. We can do it for millions and in years (or thousands in months). The F-117, the SR-71, NASA's Near Earth Asteroid Rendezvous (NEAR) mission and the Pathfinder Mars mission are all examples of high-tech projects built on a FIST (Fast, Inexpensive, Simple, Tiny) foundation.

Despite the evidence of a significant portfolio of FISTy projects (documented in my master's degree thesis), the Myth of Inevitability insists that things like this have to be complicated and expensive. Believers in the myth see no alternative to bureaucracies, technologies and processes that are complex, expensive and slow.

Their failure to see through the myth reflects ignorance of the past and a lack of imagination. There are alternatives. Download a free copy of The FIST Handbook for more details on the principles, activities and examples of how to use constraints to foster creativity and deliver systems that are simultaneously faster, better and cheaper.

New Wars, New Friend

A hearty Rogue Welcome to blogger Mike Burlson, who writes the New Wars blog! His blog is, as Gabe would say, awesome. Be sure to check it out.

Burlson's vision for the future Navy lines up very closely with my own imaginings of a future AF. Specifically, he's saying we should move away from small fleets of big, expensive, complex systems, and towards larger fleets of systems that are, to coin a phrase, fast-inexpensive-simple-tiny. Here's a short excerpt:

We contend here at New Wars that modern computer technology added to guided missiles has doomed the heavily armored, over-priced weaponry of the Cold War/WW 2 eras. With this in mind we could safely cut such complicated arms as the manned fighter, the heavy tank, and large surface warships. Their replacements would be unmanned aerial vehicles, light armored cars, plus submarines and light patrol ships.

Amen, brother!

Capabilities

Let's say I can hold my breath for two minutes. That would be an impressive capability.

However, being able to hold my breath that long would not make me a more capable writer or engineer - the two main areas of professional expression I'm currently engaged in. I would not include "can hold breath for 2 minutes" in a resume looking for an engineering or writing job -it's just not relevant to those tasks.

So, when I hear people say that the F-22 Raptor is "the most capable aircraft ever," I have to object and ask what they mean by "most capable."

When I assess a system's capability, I'm looking at its ability to contribute to the fight. Since 2001, any system that doesn't provide capabilities we need in Iraq or Afghanistan is, by my definition, not very capable. And that's precisely the case with the Raptor. It first went operational in 2005, and it has yet to fly a combat mission in either war. It's not bringing anything (a-n-y-t-h-i-n-g) to the fight. In what sense is it, therefore, the "most capable" jet around?

To put it plainly, the F-22 does things we don't currently need to do. We may need to do them someday, although I think that's not as likely as some people do. But it is a demonstrable fact that we don't need these capabilities today... or any time soon. The SECDEF himself said the Raptor is "irrelevant" to the fights in Iraq and Afghanistan. To my mind, that makes it one of the least capable aircraft in the inventory (and that's not even factoring in its maintenance issues, low availability rates, etc). Combine that with the $65B we've spent on the thing and you can see why we've decided to not buy any more of them.

So yeah, it can hold its breath for a long time, but that capability doesn't line up with the near- or mid-term needs. What we need are aircraft that are capable of accomplishing the mission. The Raptor clearly is not. I'm not saying we should scrap the whole fleet. I'm just saying we should stop trying to paint it as the "most capable" system we've got.

Catching Up With BRITE

One of the first projects I really had an impact on was a little imagery dissemination project for the National Geospatial-Intelligence Agency called BRITE (originally Broadcast-Request Imagery Technology Experiment... I think they changed the E to something else later). It was a small system designed to provide overhead imagery to forward deployed SpecOps guys who had very limited bandwidth and were highly mobile.

The BRITE project was one of the keys to developing my FIST approach to technology development. We had a very small team - in fact, I was the only government guy working on the project (and I was a junior Captain). On the contractor side, we just had a handful of people, and most of them were only assigned to the project part time, as I recall. The budget was quite small, the deployment schedule was short (this was 2001 - 2002). The system was designed to operate in austere conditions and it worked like magic.

That experience had an enormous impact on my thinking and my perspective on what can be done with small teams of talented people, working on tight budgets and tight schedules. It also hammered home the importance of simplicity - organizationally, technically and procedurally.

So, just for fun I googled it and came up with a great story about using BRITE to support the Hurricane Katrina disaster relief effort. It's nice to see that my little project still had legs after I'd moved on to other things.

Congress Redux

One other comment while we're talking about Congress.

Representatives and Senators are elected to represent and protect the interests of their constituencies. That's their job, and we should not complain and object when they do what they were elected to do. The problems in military acquisitions are not Congress' fault.

See, when we deliberately spread out development of a project across 44 states and several hundred congressional districts, we are making a cynical move that's designed to ensure the project can't be cancelled. We can't then turn around and object that those doggone people in Congress are forcing us to do something. If we built smaller, more focused projects, we'd get a lot less Congressional involvement.

Similarly, when we launch a hugely expensive project, it is entirely appropriate for Congress to insist on performing oversight. It's a lot of money, and they owe it to their constituencies to make sure it's spent appropriately. If we spent less money, we'd get less Congressional involvement.

The FIST approach is a good way to reduce intrusive oversight without denying legislators the right and ability to perform their legitimate, constitutionally appointed roles.

The Realm of the Possible

In a recent discussion about improving defense acquisitions, someone suggesting things would be better if we could get rid of Congress. Everyone laughed at this humerous suggestion, and one particularly literal-minded person said "Oh, that's too big of a change, it can't be done, let's move on..."

Well, hold on.

Maybe there's a way to "eliminate" Congress from the project without disbanding one third of our constitutionally-established government.

See, the problem isn't Congress' existence. The problem is that sometimes Congress provides, as Shrek so delightfully put it, "the opposite of help." So, what if we could minimize this anti-help?

It turns out, we can. The FIST (Fast, Inexpensive, Simple, Tiny) approach does exactly that.

A FISTy project has a small budget, which attracts less interest and oversight from our congressional leadership. A short schedule provides fewer chronological opportunitites for, let's call it involvement (instead of meddling). A small team and simple organization doesn't get spread across multiple congressional districts and states, and voila, that means fewer legislators with a dog in the fight.

These decreases are all appropriate and above-board. It's not a matter of shutting Congress out, but rather of keeping the project sufficiently small that it doesn't require their, ahem, help.

Change

I like to say that FIST isn't a new way to do acquisitions. It's just how we do acquisitions when we do it well.

But frankly, I'm not convinced my current attempt to drive FISTy change into the system is going to succeed. I am very aware that the probability of success is low. Five or ten years from now, there's a pretty good chance the defense acquisition community is going to look very much like it does right now.

Having said that, the environment does seem more change-friendly than it ever has before. Our experiences in Iraq and Afghanistan are a big part of that, along with the new administration, changes in the economy and advances in technology - particularly UAV's. This combination of operational, political, economic and technical factors is sort of a perfect storm and gives me hope that significant, meaningful change is possible. If the FIST approach ever had a chance at widespread adoption, it's in the current climate.

Our troops overseas are increasingly emphasizing the importance of rapid access to systems that are simple and effective. They have less tolerance for the complexity and delays of previous years. The president ran on a platform of change, and military leaders from the SECDEF on down are reimagining the way we do business. The economy is forcing people to think really hard before spending money. And along come these UAV's, which offer relatively inexpensive and simple ways to accomplish missions that used to require much more time, money and effort.

So will FIST take off? I don't know. I certainly hope so. All I know is that I've got to try to make a difference, even if the odds are against me, 'cause these are the best odds I'm ever likely to see.

FIST Video!

Yesterday's post was the short version of the FIST (Fast, Inexpensive, Simple, Tiny) approach to system development.

Today, I've got a short post with a link to a long video. Yes, this is the much anticipated Rogue-a-palooza aka Rogue Fest 2009 aka the briefing that Gabe and I did at the Defense Acquisition University back in July.

I confess I haven't actually watched the whole thing yet, but my mom and dad did and they said I did real good. They also said the video includes the Q&A at the end of the presentaiton - I wasn't sure if that part would be included, but it was. That's cool.

So, if you've got an hour or so to kill, and you want to see me and Gabe jump around like monkeys and tell stories about why FIST is so good, click the link.

It's Not Inevitable

OK, let me summarize the whole FIST thing as concisely as possible. It really comes down to this:

"There is nothing inherent in military technology that requries it to cost so much, take so long, or be so complex."

There, I wrote it down for everybody on the Interwebs to see, and it feels good.

The more I read, the more I do, the more I hear people's stories, the more convinced I am that complexity is not inevitable. We do it to ourselves, technologically and organizationally and procedurally... but we dont' have to. Similarly, these huge cost overruns and schedule delays are not inevitable. Until we believe that - I mean REALLY believe it - we're going to keep getting the results we've been getting.

Incidentally, this applies to NASA's space projects as much as the DoD's systems... and no doubt to commercial & industrial projects as well.

CogBlog Homerun

Check out this (rather lengthy) post over at the CogBlog. It's an insightful look at the saying-doing gap in defense acquisitions, primarily focused on the IT side of things. A few nuggets to whet your appetite:

DoD acquisition regulations do not actually mandate stupidity … they just encourage it.

...the Joint Capability Integration Development System, or “JCIDS“, mandates a change from “system-based” to “capability-based” requirements... However formal OPTEST evaluates systems, not enterprise capability.

What is the gap between the capability provided by Travelocity vs. Defense Travel System? [Dan here - holy cow DTS is a pain to use!]

We need pre-approved catalog offerings available on GSA schedules and IDIQ vehicles. We need “dating services” to put consumers with like requirements in touch with each other and expert providers. We need “consumer reports”... to provide basis of objective, convenient comparison.

TSPR, Trust & Strength

In the late 90's, before I really had any idea what was going on in the world of defense acquisitions, some in the DoD tried an approach called Total System Performance Responsibility (TSPR). It came and went largely unnoticed by little ole me. Now, like Total Quality Management and other past attempts at making things better, TSPR has become a bad word among military technology circles.

Since people are still throwing spears at it, I figured I should know more about it than I do. It turns out, the basic idea was to reduce the government's involvement and allow contractors to do things their own way, as efficiently as possible, without undue influence or involvement by the government.

Maj Henry Pandes wrote a pretty good article about it, if you want to read more. James Gill wrote a short response to Maj Pandes' article. Also a good read.

Unfortunately, many people today seem to have decided that the problem with TSPR was that it involved too much trust. Trust is bad, they conclude. We can't trust those slimy contractors. They're bad, they're bad!

Hold on. Let's not get the wrong conclusion here.

After a bit more investigation and reflection, it seems to me that the problem with TSPR wasn't trust, but rather ignorance. Instead of reasonable delegation, we turned it into abdication. The government wasn't merely hands-off, it was eyes-shut. Anyone surprised that didn't work out very well?

The truth is, I think the government should trust contractors more than we do (as I've written elsewhere). But that doesn't mean we should take the "wake me up when it's ready" approach. We can still be involved and informed, without turning it into excessive meddling.

More to the point, trust isn't weakness, naivete and stupidity. Trust requires strength, judgment and wisdom. A lack of trust indicates, among other things, a lack of trustworthiness and a significant degree of unreflective foolishness. Yeah, I said it - foolishness. Real rogues trust their partners, 'cause that's a smart and strong thing to do.

The TSPR approach may have been fatally flawed from the start, or it may have been a good idea badly executed. I still don't know. But what I do know is that if we think the lesson of TSPR's failure is to not trust people, we've learned a horribly wrong lesson.

Team Rogue in the Danger Room

For those who haven't heard yet, Wired magazine's Danger Room blog did a very nice post about Team Rogue and our FIST approach to acquisitions.

They even sort of made it sound like the SECDEF is an advocate of the FIST approach. OK, it didn't sort of sound like that... it totally sounded like that. What else does the phrase "high level advocate" mean?

Anyway, it's fun to see our ideas getting this kind of visibility.

Cancelling Projects

From a justice perspective, it's important for an accused person to have a good lawyer. Even if the defense attorney thinks the client is guilty, they are still supposed to provide a robust defense and make the strongest case possible.

Unfortunately, program managers often act like defense attorneys for their projects. They sometimes act as if their job is to keep the project alive, and defend it to all comers... even if the project is a dog and needs to be cancelled.

The program manager (arguably) knows the most about the health and viability of the program. If it's heading over a cliff, aiming for irrelevance or otherwise doomed, the PM is usually among the first to know, and should speak up. Sadly, this does not always happen.

PM's are not defense attorneys. Their job is not to keep the program alive and out of jail. It's to guide and shepherd the development. And if the program needs to be cancelled, the PM should lead the charge.

FIST Handbook - Updated

Team Rogue and I put together an updated version of the FIST Handbook (pun intended - sorry!), which collects the best of our articles from Defense AT&L.

Unfortunately, the file format has some incompatabilities with Lulu, so the version you can download on RoguePress is out of date. The new version is 12Mb, and rather than try to email it, I stuck it on Rapid Share. You can download your own very version right here.

Enjoy!

SOCOM Truths

A former classmate of mine is about to deploy with the Spec Ops guys, and he just sent me an excerpt from a briefing about SOCOM acquisitions. It contained the following truths:

FAST Requires MORE DISCIPLINE
RISK Must Be Managed NOT AVOIDED
FASTER Does Not Have To Increase COST/RISK
COMPETITION Can Be Done QUICKLY
UNCONVENTIONAL THINKING Is An ENABLER
CREDIBILITY Enables FREEDOM OF ACTION

I've always said that SOCOM is one of the shining stars in the defense acquisition community. Their track record of delivering on time, on schedule, AND being operationally effective is quite impressive. Embracing FISTy truths like these probably has a lot to do with that.

Courage!

Hey true believers!

The latest action-packed issue of Defense AT&L is posted online - click on over to read my contribution, The Courage Imperative.

I have to admit, I'm a little nervous to see what sort of reaction this one generates. It's a bit, um, pointed... (yeah, unlike all my others, right?)

USAF gets FISTy with new Light Attack Fighter

As reported at the DEW line (and other places), the AF sent out “a request for information on July 27 that calls for first aircraft deliveries to start in Fiscal 2012 and the first operational squadron to activate a year later.”

Let’s see, requesting information in Jul 2009, for deliveries to start in FY12 (which starts in Oct 2011, a little more than two years from now), and an operational squadron just one year later… now we’re talking! The question of course is whether we’ll be able to stick with these short timelines, well-focused mission and emphasis on simple, mature technology, but this project definitely seems to be off on the right foot.

A similar phenomenon is happening in the area of light mobility airlifters. We’re talking about picking up 60 of these “cost effective” aircraft, with an initial operational capability date of FY12.

Strikes me as precisely the approach to capability acquisition we should be taking. I hope it sticks!