Wednesday, March 9, 2011

Scrum and Release Planning

I went to the Agile Ottawa group last night. Mostly I very much enjoyed the meeting. A lot of smart people, a lot of good discussions. But...

A Sad Tale

A gentleman there, a serious business guy, new to Scrum, said something like this: "It seems to me that Scrum has no up-front planning. And for those of us in the business world who must give some up-front estimate of time and dollars, this is not going to work for us. Does Scrum have some up-front planning?"

Several people answered. And I gave an answer: YES it does have up-front 'Release Planning.' Scrum does normally include up-front planning. This is somewhat controversial in Agile, but we have it. It is called Release Planning.

And I went on to say some of the things I say below. But my comments were brief, and it seemed to me they were drowned out by Agilists saying "why do you want to do up-front planning?" (and similar comments)...which is a good question, but again, it likely implied to the gentleman that all we 'scrummers' hate up-front planning. Perhaps I am wrong, but it seemed to me that he heard 'Scrum does not do much Release Planning'.

I discussed this with a person in my CSM class the next day. His comment was "yes, based on the stuff I have read [and he had read several things] Scrum does not say much about the up-front planning. I can totally see how that person had that impression."

To me this is very unfortunate.

[Aside: I call this a sad tale, with my tongue somewhat in my cheek. These are relatively normal human misunderstandings. Perhaps a bit sad, but nothing to compare to the sadness in Libya right now.]

Let me now make two references.

In the Scrum Guide (located here, among other places), Release Planning and a Release Planning Meeting are specifically mentioned and discussed.

In "Agile Project Management with Scrum" by Ken Schwaber there are only two mentions of Release Planning (according to my Kindle) in the book. Both not favorable (really about bad release planning). So, one can see why some people might get a wrong impression from that book. And some other sources as well.

What's Wrong With Up-Front Planning

Now, let me guess why Ken Schwaber minimized the mention of Release Planning in the book mentioned above. First, Schwaber and Sutherland note in the Scrum Guide that a lot of the details about how to do Release Planning are outside the purview of Scrum. Meaning: Scrum is a framework only, and as such, it is probably not useful or necessary for Scrum to give guidance on this that could be seen as Scrum's "answer" for everyone. Starting to do that risks making Scrum too big to implement in one step, relatively quickly. And risks giving great advice (for one effort) inappropriately to another effort or situation.

To me, these are quite reasonable concerns, and I think it stretches the point, though, to be so quiet about Release Planning more generally. And, indeed, the Scrum Guide, short as it is, does give a fair amount of guidance on Release Planning.

Another hypothesis I have is that Schwaber was concerned at the time he wrote that book that he was seeing too much Release Planning in 'scrum' implementations that looked too much like waterfall release planning. (I think I have seen this some too, although perhaps not as much.)

So, what's wrong with that?

Well, first, the customer sees no value in Release Planning per se. It is, to use lean terms, at most a 'necessary' waste. So, we want it to be as short as possible (but not shorter).

Second, too often people think the main (only) value from the initial release plan is the initial date and budget. Usually, as indicated above, that initial date and budget are necessary information to make some initial basic decisions. Maybe even accurate information if there were to be no changes. But anyone with much experience with projects has a near 100% experience of changes to projects after the initial Release Planning. Usually quite a number of changes and quite significant. In which case, the initial date and budget quickly become moot (in terms of accuracy; possibly still "directionally" useful, if you have a firm that thinks in those terms).

Third, often the initial Release Planning date and budget are used to drive people on Death Marches. This is an horrendous and utterly scandalous thing in our industry. It is shameful that we let stupid managers use this information this way, but that's the way it often happens. And unfortunately, this gives Release Planning a bad name too.

And partly because of this, many coaches recommend that Release Planning not be done until the Team has established a real velocity. (To me, for reasons I will not explain here, this is the wrong solution to a very real problem.)

Fourth, because of the rapidity of change in many areas, many agile people feel that the YAGNI (You Ain't Gonna Need It) principle means that up-front planning should be minimized. Very minimized. Often these people are from business situations where keeping Release Planning quite minimal will actually work. But, in my professional opinion, other business situations actually greatly benefit from some up-front long as everyone uses the plan (and all its refactorings) to drive good behavior (such as minimizing the stories that get in the real "minimum marketable feature set" of the next release).

I guess we need to be concrete here, to avoid silly discussions. I won't give all the parameters here (and, to be more accurate, one should), but let me ballpark and say that a typical 3 month release for a Scrum team can probably be planned usefully using agile methods (Release Planning) in about 3 days or less. Not a half-day, but again, not 2 weeks or a whole week either. Again, there are potentially a host of possible factors; these are fairly typical real-life situations I have in mind.

What's Right About Release Planning

OK. So we acknowledge that there are 'issues' with up-front planning. (And more than we said above.) These are not issues that either Scrum or waterfall create, but issues nonetheless.

Now, let's look at some of the positives.

A. Businesses need some way of comparing efforts, to decide which efforts should get started, and which should not start. Exact or even somewhat 'accurate' numbers are not essential here...just some rough sense of date/budget and level of accuracy. In virtually all businesses that I work with, this is (or should be, in my professional judgment) a requirement.

B. The team needs to start to see, at a middle level, the interrelationship of the benefits and costs of the work (features) it will be building. To make many other decisions (architecture decisions, among them). Again, rough numbers are sufficient usually. And even very rough numbers are better than nothing. Release Planning provides this information.

C. A sense of the rough time trade-offs is needed. Customers always want features immediately. When the initial conception of the first release looks, as an example, a long way off, it forces the mind into innovative solutions about how to scope down the first release. Release Planning can give the team enough information to start being innovative this way.

D. I think the biggest value of the initial Release Planning is that the whole team starts to see the same elephant. The same effort. In multiple dimensions. The stories (functionality), the business value (at a story level), the effort (at a story level), the risks and dependencies, etc. This is extremely valuable and, in my experience, the level of 'same-page-ness' is so very much better than what I always did in waterfall planning (and what I still hear). Again, extremely valuable, even though it will change.

E. Re-Planning. The initial Release Plan sets up a base from which re-planning canbe done easily. Each Sprint. I like to call this Release Plan Refactoring (and sometimes we use other names for it, such as: Product Backlog Grooming, Product Backlog Refactoring, etc, etc.).

Re-planning, as Eisenhower implied, is essential. The way I will put it (maybe mixing some metaphors) is: We have to use Pareto to deliver just the most essential stuff so the customer gets the coffee while it's still hot. A continual daily-hourly fight to see that most essential stuff. Mark Denne's "Software By Numbers" talks more about why it is so essential to release a "minimum marketable feature set." A great phrase.

Let me emphasize that in Scrum, we are always doing "Release Plan Refactoring" every Sprint. It includes re-planning in multiple ways, such as: adding new stories, using a new known velocity of the team, refactoring the story points on a few badly estimated stories, adjusting some business value points, slicing rising epics into smaller stories, etc, etc.

Having mentioned all these things: Scrum does not define these activities. Again, Scrum is a framework only. I have given you what I see and teach and advise for almost all situations. (And other people whom I respect also concur with these.) But the activities just mentioned are not defined in the Scrum framework. Scrum only defines a 'short' release planning, and then release plan refactoring (although the Scrum literature does not use a consistent term for release plan refactoring).

E. The final value is motivation. The team no longer feeling like they are getting the mushroom treatment. They see the 'big picture' at a more meaningful and richer middle-level. The team feels like they are respected. They see themselves as adults making an adult, well-considered commitment to build that product. This motivational impact is quite important. And is not affected by the inevitable changes much.


More to say, but enough for now. For those interested, see our earlier posts on Release Planning. And future ones.


Dave Rooney said...

It's interesting that Scrum seems to appeal more to the project management and business folks, but XP explicitly deals with up-front release planning.

Coming from that world, spending time prior to starting actual development work figuring out what a group is actually going to build seems quite natural to me.

Perhaps, Joe, you may want to point out in your courses that Release Planning and indeed even Planning Poker originated in the XP world. ;)

Dave Rooney said...
This comment has been removed by a blog administrator.
Joe Little said...

Note: The second, removed post by Dave Rooney was a duplicate.

Joe Little said...

Hi Dave,

Not too concerned about history.

Scrum started before Kent Beck wrote his book, and in all the courses I have done with Jeff Sutherland, I never got the impression that Release Planning was 'added later.'

I do want to say that everyone I know who teaches or coaches Scrum also supports XP for their teams. Or at least a strong move toward XP and its great engineering practices.

Planning Poker as I understand it originated as 'wide-band delphi estimation' at the Rand Corporation many many years ago.

But to me, the history does not matter. (Albeit interesting to us as coaches and trainers.) What works? What can we adopt now? How do we improve? These are the key questions.

I think others (and I) would be interested in your take on the benefits and possible pitfalls of up-front estimation. And how to go about re-estimating. There is more to say about doing it, as you know. And about how different parts of the agile community now view it.


Dave Rooney said...

re: Not concerned with history - Neither am I, except that Agile == Scrum for most people who are currently just learning it. If the Scrum Guide is where they start, they could get the same impression as the gentleman at Agile Ottawa - you don't do any up front planning. You & I both know that isn't the case, but someone who doesn't have 7+ years in the "Agile world" may not know that.

Interestingly, a somewhat similar message popped up on the Scrum Development group today about how to cost out a project up-front in order to get approval if you're using story points to measure things.

There seems to be a definite need for that sort of information to make it to those who are new to Agile.

Joe Little said...

Hi Dave,

Yes, sorry. Did not mean to imply negatively that you were into history.

And for the record, I am slightly concerned that my post was too negative about others' comments at the meeting: I did not hear everyone else's (and certainly not your) comments as negative about Release Planning. But I was left concerned that that gentleman did not hear really hear positive things associating Release Planning and Scrum. (Maybe I was wrong, I hope, for him.)

I know lots of people who talk about Release Planning (eg, Jeff Sutherland). But, yes, it appears we need more of it. And, for those interested in Scrum, the words should be linked to Scrum.

As I tried to say in the post, there are definitely some potential bad ideas or practices associated with up-front planning...but that does not mean "don't do it at all." Which is obvious to you.

Anyway, thanks, and good night.

Ernest Ruppenthal said...

The thing about history is that it helps to answer the other questions (What works? What can we adopt now? How do we improve?). And hopefully prevents us from repeating past mistakes.

Good article, though. Thanks for posting it. Wish I could have made it to the meeting. Sounds like it has some very interesting and thought-provoking discussion.

Joe Little said...

Hi Ernest,

Yes, history is important.

Yes, many other good discussions. Hope to be back soon myself. And maybe meet you there (Agile Ottawa).