How to measure success on agile projects from a customer point of view

This topic is a thread on Scrum-Dev (the yahoo list). Excellent topic. With many good posters. Go and see.

This is an involved topic, so I will give some views in this post, and more views later.

I start with the word "measure". Because I am concerned that we are measuring too many things (not good), and the wrong things (even worse). Many people don't take to being measured like a thing. (People have lots of characteristics and variability, but being a "thing" is something they are especially not good at, or so I have found.)

So, what to do if one wants a simple measure?

The simplest "measure" would be to ask the Product Owner on the team..."is this team effective and are you guys producing business value?"

This is one measure. It might even be binary (yes or no). And it is low cost to collect, except that the Product Owner might give you (let's say for now that you are a senior manager) more of the truth than you want. And ask you to assist in making things better (and things can always be better). (As an aside, if the Product Owner were really good, I might accept a yes/no answer. From a less strong Product Owner, I would always want a longer answer, probably with follow-on questions.) And it might be a conversation that is forward-looking and action-oriented.

The next problem becomes "what is business value really"? To me this is the key direction to take this question (finding a simple measure). (There are other directions we must also take, but for later.)

Let me emphasize this: efforts (projects) are successful to the degree that they deliver business value. There are no technical successes ("we did what we were asked to do", "we're within scope and budget", "it's a beautiful system", etc, etc). There is only delivery of business value...at least somewhere along the chain of value to the ultimate customers.

Therefore: "what is business value really?" There are many views on what business value is. We have discussed some in earlier posts.

Lean says that value is defined by the customer in the context of a specific product (service) at a specific time. That says, among other things, that customers are redefining value all the time. If you take your own personal experience, you know that your "values" (at least in the
material world) are changing all the time. Your own wants and needs change, with great rapidity sometimes.

OK, so what do we do about this in Scrum? Well, Scrum does not prescribe any specific method. (That's a good thing, because there are so many situations.)

What I often suggest is this...and it seems to fit many situations well. Let's assume you can determine a dollar amount representing the business value of a release or for the whole project. Then, assign 1,000 (or 10,000) "gummi bears" to all the existing user stories (or the equivalent if your Product Backlog is not in stories). Perhaps reserve some gummi bears for stories yet to be discovered. You might put those BV gummi bear numbers on one corner of each Story Card. (I do not call these BV numbers "Story Points" because too many people understand that term to mean points of size/effort, which is different.)

Then, as each iteration is completed, count the gummi bears "done". Say 100 are done after the first iteration and the total is 10,000. And say that the total business value is $26 million. 100 divided by 10,000 says you have completed 1% of the business value. Or $260,000 in business value (if I did the math right).

It's a good enough approximation for most situations.

The next problem is if total business value changes during the project. Which it always does to some degree. And new business value is discovered (if anyone is paying attention). Usually.

Is that a simple enough answer?

Of course there is much more to say on this topic. I particularly recommend Software by Numbers by Denne and Cleland-Huang.

"Things should be as simple as possible, and not simpler." I think Einstein said that.

Let me deal with one more issue (in this post) that I have often seen. The issue: the Product Owner is not always "good enough". This is always a risk. But, in a way, unavoidable. Someone must be the proxy for our customer set, trying to determine the overall business value of the product (or our change to the product). Scrum prefers, to avoid long delays in making decisions, to name only one Product Owner. Who serves as that customer proxy (in the context I mentioned). Of course, one or more people (perhaps on the team, perhaps not) might still assist that Product Owner in determining the business value.

Just as the one Product Owner can fail, so also can any group of people fail. Scrum has no silver bullet to eliminate failure by people. What it does do is make clear and visible where the (relative) failures are. Often that can allow corrections to be made.