Lean Agile Open - April 15th, 2016.

We had a great event in Charlotte yesterday, and I hope you did not miss it. Self-organizing people did the best they could (which is actually a lot) to learn as much as they could with other smart people. About Lean and Agile topics. With, I think, the goals of (a) becoming more effective, and (b) making their own loves, making the lives of their team members, and making the lives of their customers ...better. I don't want the event to sound too goody-goody. We all had fun too. But what I want to emphasize first is: we self-organized. And then: it was good. Thanks to many people! Let me name at least these people: Cassandra Wagner, Riddhi Gupta, Murali Varadarajan, and Bob Petruska. But many others also contributed. What was the value?

1. Senior people at 'waterfall' companies could see that regular people could self-organize.
2. We all saw that we all can help each other.
3. Each of us learned some important things to use back at the ranch to make lean or agile work better at our real work. Each person got to focus his learning where he or she wanted.
4. Most or maybe all of us were 'enlivened.' We felt more hope, more belief that things can change, less alone, more hopeful. We had a sense of a brighter future. This is something that you can never 'buy.'

We did we learn? As I said, we each learned some specific things about specific areas of lean and agile. All good, but not my focus for right now. For some who were new to Open Space, we learned to trust 'open space.' So, I hope we learned to be more courageous next time. As your parents always told you, it is important to trust people. Of course, not to trust too much. But to have some trust, and to be trustworthy. Because more people will have more trust next time, the event will be bigger and better. More people now know how to 'use' Open Space effectively. If we use these events well, this will start to change the community. No man is an island. We are all inter-connected. And we all influence each other. These events, repeated regularly, will change the community. And they can, perhaps more slowly than you may want, change the culture of your company. We will use these events to continue to .... Well, the mundane way to say it is: To get better and better results from lean and agile. But I hope in your heart or stomach (those of you who were there) you feel a stronger energy, an energy you cannot ignore, and that it draws you much more strongly than those simple, mundane words.    

Scrum makes work a game.

You know, of course, that Scrum is named for the Scrum formation in rugby.

More generally, Takeuchi and Nonaka were inspired by the 'rugby' they saw in several great companies and how they did new product innovation.  And Sutherland and Schwaber read that article in HBR: The New New Product Development Game. And that was a key input to creating Scrum.

Here's where we are I think: "Life is hard, life is confusing, and I can never tell if I am making progress. It starts to be no fun, no real people to talk to usefully, no way to see if we are making progress. I am always late and it seem never-ending!"

If some of your colleagues feel that way sometimes, you can imagine that that is: Not Good.

So, Scrum changes it to a 'game.'  It is still real, so not a game in a sense that it is divorced from real life.  But it is, as they say, gamified.

We have a 2 week sprint. We have some defined roles. We put a person on a Team.  We ask the Team (not one person) to be successful.  We suggest that they self-organize and work together (collaborate).

And we allow they to make first downs (as in America football). They get feedback quickly (as in any game)...are they making progress.

Are the measures of progress that we have in Scrum perfect?  Well, actually I think they are very good, but they are not perfect.  But something that is fairly frequent feedback and fairly accurate is far far better than 'nothing.'  Or a couple of 'at-a-boys' that were said half-heartedly.

The key thing is that we have made things 'fun' in a way.  And that feedback is useful for course correction, but perhaps more useful as motivation.

Why?

• we get positive feedback
• we see that other people care
• we can feel proud of ourselves with small wins
• we work together, and it no longer feels lonely
• usually they team starts laughing during the day.  It is fun in that way too.

Let me remind you how games are seductive.  As any behaviorist psychology major can tell you, they give intermittent (positive) feedback.  To be honest, not always positive feedback.  But the positive feedback in intermittent.

And this, according to the research, is key to making the game 'fun.' Now, when they are playing the game, they do not always have a smile on their faces, but still they are engaged, and typically more focused, more 'giving their all' than they usually do working 'in their cube' (as we often see in waterfall).

As with any good game, we need to be able to see frequently, transparently, easily if we are making progress or not.

The managers need to talk about work as a game.  A game we wish to win.

Poppendiecks: The Tyranny of the Plan

Here is a transcription of Mary Poppendieck’s talk. (The tyranny of the plan).

You will find this insightful on many levels.

You might find it more fun to see the video.  Your choice.

Agile Manifesto - History

As you know, the Agile Manifesto and the Agile Principles define agile, to many of us.

These statements were written and agreed to in 2001. There has been some debate about them, but they provide some excellent principles to remember as you are doing agile.  If your practice embodies these principles, then I think your agile will be much more effective.

So, we do well to remind ourselves of these and think through what they really mean.

In this regard, there was a recent discussion…what were the original writers thinking when they wrote these words.  In a google group discussion, Jeff Sutherland said quickly ….

They evolved in about 15 minutes in the order described.

The first one had data from Bell Labs showing a 50x increase in performance by getting individuals and interactions right. Martin Fowler was writing on the white board and proposed we say OVER processes and tools so we did not rule out their use, but focused on what added more value.

The second one I think was brought up immediately by Ron Jeffries. We have seen in Silicon Valley and in Europe for complex development that failure to test and fix inside the sprint fully integrated can take 24x [more] time to test and fix in the next sprint. So you can go 24 times faster with working software.

The third one I was the most passionate advocate as I was CTO of what is now GE Healthcare at the time with 3000 screaming hospitals on my case. We know now that a good Product Owner can generate a 4x increase in production.

The final one was the mantra of the XP folks in the room so we all agreed it had to be there. 65% of features are never or rarely used and waterfall builds them instead of letting new requests into the plan.

You should read Jeff Sutherland’s blog post on this: The Agile Manifesto, Elaborated.  There is also a link there to a yet longer discussion.

Why are we doing Agile?

We just had Southern Fried Agile in Charlotte.  See SouthernFriedAgile.com

I have 5 take-aways.  For today, the first one.

1. Why are we doing this?

We are not doing Agile for Agile's sake. We are doing it because we think it will help.

And the most important people to help are.... well, first, everyone.  Everyone at the same time.  Unbelievable.  But that is too broad for most of us.

First, the customers.

Secondly, the widows and orphans that own the firm (and, ultimately, it is the widows and orphans who own the firm).

Third, the workers and the managers (although in knowledge work, where you separate worker from manager is hard to see...).

Making people's lives better is often overly abstracted into the notion of delivering Business Value.  And that is mostly correct.  But remember that we are delivering business value to real people.  We are real people, and even they (the customers) are real people.

So, we talk about how lean-agile-scrum is so important, and it is, but only as a means to helping the lives of real people get better.

***

Change is always hard.  They say birth is painful.

If it is 'too hard' to become truly agile in X period of time, should we compromise?  Is it better to compromise?

The logic being: the benefits of X (agile, for us) are not worth the pain of the transition.

(Fortunately for human life, most women do not find this argument compelling enough.  Although you see in the eyes of 'new' mothers some thoughts like this. Before the baby is really real.)

Umm.  Very difficult question this is.  Worry about it you will.  Act you must.

And if we compromise (of course we humans always do), should we become complacent with what we changed to 'in the first jump'?   NO!!!!!

***

One of the key points...   We agile people too quickly assume that others (eg, executives) naturally understand why agile is important, and 'care' about agile.

And, one point is, they often do not.  They care about people, they care about business goals.  But, they do not see the necessary connection between business goals and agile.

We must always remember this.

Even with people who understood agile a month or two ago, we always should remind them, and show them well, how agile gives them the benefits they need and want.  Agile is a means, not an end in itself.  Let us not conflate the two.

***

Do I still think that, in most situations, by doing less Scrum-Butt people will get more benefits.  Yes, strongly!!

Do I think that everyone everywhere must immediately do, and only do, pure Scrum?    Well, I think that is too broad or aggressive a statement and not worth answering. For example, I will never talk to or even see 'everyone'.

 

Can we make a promise?

Well, can we?

There are some who do ‘waterfall’ (well, to be fair, something very roughly like the waterfall that Dr. Royce defined in 1970), and some of them seem to believe that we know enough early on, and change will be small enough for the rest of the effort, that we can promise a relatively precise date at the very beginning.  And reliably hit it.   And then, if we do not, they blame ‘scope creep.’

There are some in ‘agile’ (again, we must qualify it) who seem to think that change is always so large and we know so little up front, that we can never promise any release date for our product.  Not at the beginning, and not even along the way.

Of course, we have to agree that the dumbest day of the project is Day 0, at the very beginning.

Next, let us first recognize how important an early release date is.  It is very important to our customers, and it is often extremely important vis-a-vis competition and other factors.  If we release earlier (eg, on time), then the present value of the business value we will earn will be greater (assuming other things roughly equal).

So, clearly there is a great need to release earlier.  And, along with that, although arguably not as much in every case, a need to predict the release date earlier.  And with reasonable accuracy.

Can we do that?

The short answer is yes, to some degree.

In Agile, we can do that, in my opinion, somewhat better than we did in waterfall.  Which also means, still, not all that well on Day 0.  We still have problems in predicting the future.  We know the detailed requirements will change, in many ways and for several reasons.  We know both good and bad change will happen.  It does every time.  The only question is how much things will change.

Still, after some time and experience and learning, we can make some prediction.  And often, with a reasonable contingency, we can hit that date.

(I do not mean to suggest that the first attempt at predicting the date is accurate enough.  That itself is an important business decision — when do we feel we are accurate enough?  (after adding some contingency). Very good and difficult question.)

Can we predict ‘accurately enough’ every time?  Of course not.  And there are many reasons why.  Sometimes we get hit with the ‘Japanese tsunami’ of change.  A lot more change than we could have reasonably predicted.  And then, usually, other people will fully understand if we do not deliver on time.

In another scenario, it is often not one piece of change, but the accumulation of multiple changes that overwhelms us as a Team.  And, because it is not one big public change, but rather many factors, it is harder to understand from a distance.

Some situations are by nature more variable, subject to greater amounts of change (ex: subject to more learning about competition).

Still, even so, predicting, and using the information and learning we get from planning, can be very useful.  It can give us a better basis from which to learn more, in many dimensions.

Finally, one more factor that must be managed well (and often is not).

We need sustainable pace.

First, it is a moral issue.  We cannot treat people as slaves.  Second, we must have sustainable pace because it will make things better.  A pace that enables people to rest, and then, when they are working, to give their best creativity.  So that the innovation is more impressive.

So, what is the problem?  Well, we also know that people need some pressure, some challenge, to do their best work.  And we also know that many many teams, if the deadlines are tight and people say ‘go fast’, the Team often hears that as ‘cut corners.’  In other words, build technical debt.  Write sloppy code, let bugs in, do not document enough, do not refactor anything….and many more bad practices.

So, we the Team, we the Product Owners, we the managers, must be very careful in agreeing on dates and how we talk about this stuff.  Or you will quickly have a buggy ‘legacy’ system, and everyone will resign from over-work.

And it is hard.  You should (in my opinion), ask for dates and predictions.  And the Team needs a challenge.  But we must also talk about sustainable pace, about simplifying the work we do (smaller features), but NOT about cutting corners on quality.  Or maybe better said as ‘still, what we do build, it will be of good technical excellence.’

I will not talk here and now of how we keep our promises.  I have hinted already that breaking down the features and doing as little as necessary on the ‘big’ features is key.  Fixing impediments is key.  And identifying the minimum marketable feature set is key.  And hence, making the right trade-offs is key.  This includes cost-benefit trade-offs.

There is an art to doing the continuous research more effectively to make and revise ‘promises’.
And there is an art to working with the Team to change numerous factors so that we can improve the probability of keeping our promises.

People care about time.  Tempus fugit.

Is Scrum about spirit?

I hear people talk about Scrum quite a bit.  And I talk about Scrum.  Usually we are talking about the basics of Scrum, which sometimes seem so mundane.  Get a Team, make them a real Team, do the SM role well, do the PO role well, have some better meetings, build some artifacts, etc, etc.

But is Scrum all about these practicalities?  Is Scrum just the roles, the meetings and the artifacts?

I hate to say it, because I know it bothers some people to the point of offending them.  But there is a spirit about Scrum, and if you ignore it, you miss what is most important.

I often talk about Scrum-Butt.  “We do Scrum, but…”   Often this is really about how people can do Scrum and do it so poorly that they only get a 20% improvement (as  though in real life a 20% improvement were not huge).  But by not doing Scrum fully, they reduce the improvement they might get.  And we call this Scrum-Butt, from the phrase ‘we do Scrum, but…[we don’t do a Daily Scrum, etc, etc).’

And I talk about how, to avoid Scrum-Butt, the Team must have the values and principles of lean-agile-scrum embedded in ‘muscle memory.’  And so, I put a focus on the values and principles.

But there is a spirit about Scrum too.

I think Jeff Sutherland is in part talking about spirit when he says that ‘they must be having more fun – it they aren’t having more fun, they aren’t doing Scrum right.’

I think there are many aspects to the ‘spirit’ within Scrum.  In part you can see them through the rules or the values or the principles (and there are many more than just those articulated in the Agile Manifesto and the Agile Principles).

There is something magical and wonderful when a Team of 7 rights itself through self-organization, and becomes a unit that can accomplish great work.  And, without putting too fine a point on it, the spirit of the Team is clear, and one feels it is that energy, above all else, that gives them their power, and their effectiveness.  As a New Englander (which I am in part) does not know quite what to call that camaraderie, that sense of joint ownership, that happiness, but it feels almost as though the spirit of the Team makes itself known in these ways too.  One finds oneself caring about the others in the
Team, one finds greater respect, one allows each member to be more his true self, and yet that does not tear the Team apart, but rather brings them closer together as a Team.  If one were from another culture, one might almost say the team exhibits the characteristics of a higher love for each other, and for their customers.  But love is a word seldom used in New England except in connection with Hallmark cards, and then usually with a chuckle.  And to speak of agape and caritas sounds too much like some philosophy class, about which we must soon joke.

In any case, until you have been inside a really good team, and until you have observed a really good team from outside, I am not sure you can know what I mean.  And, more importantly, I do not think you can really understand Scrum,  until you have felt that spirit.

What is rather interesting about Scrum, I find, is that if one enters it with an open heart, and just does it, even does it only partially, it seems to me that Scrum fills you with its spirit.  And, as one consequence, you naturally care more for your teammates and more for your customers.  Or, so I find.

***
Your comments are welcome.

6 Myths of Product Development - Take 2

Months ago, I mentioned here a great article. Here is an article about: Six Myths of Product Development.  By Stefan Thomke and Donald Reinertsen.

Strongly recommended.  Especially good for managers and executives.

Here are the fallacies in one list:

• High utilization of resources will improve performance.
• Processing the work in large batches improves he economics of the process.
• Our plan is great; we just need to stick to it.
• The sooner the project is started, the sooner it will be finished.
• The more features we put into a product, the more customers will like it.
• We will be more successful if we get it right the first time.

I wanted to discuss them some, with the hope that I might entice you to read, understand, and act on the article.

Let's do the first two today.  Quickly.

1. High utilization - Myth

Ok, ceteris paribus (other things being equal) high utilization of the resources would be good.  Lower cost per widget.

But in knowledge work especially, we only deceive ourselves. First, it is often based on the false premise that knowledge workers are fundamentally lazy.  Demoralized, demotivated -- maybe.  But not lazy.  (Ok, maybe 1 in 30 has some real laziness, but we do not recommend 'pushing' on high utilization to get that fixed. Fix it another way, and we do in Scrum.)

Second, high utilization, as shown in the article, leads to low through-put from the system (to us, the system is the Team).  Nothing gets delivered quickly.

And in product development (eg, software development) fast delivery is ESSENTIAL. No, it is much much more essential and critical than my capital letters imply.  We barely have a glimpse, in my opinion, about how critical it is.  And in how many ways it is critical.

Once we recognize that, then we must struggle with seeing the utilization level, and adjusting it to the optimum level. (We are not proposing super-low utilization. Just lower, to get the through-put.)  We have some rough techniques for that in basic Scrum. As you become professional in using those techniques, you may want to fine-tune the techniques for each team.

Again, a push for high utilization by managers is killing us. Managers should focus on speedy delivery of smaller things (smaller releases).

2. Large batches - Myth

It often feels as if we become more efficient with large batches. And, it is partly true in a way.

But it is mostly false, and in all the important ways.

Large batches tend to lead to silos. That means we tend not to work together as a team (in all the bad ways we mean with that phrase).

Large batches means that problems are hidden. And hidden problems are much harder to fix. And, in knowledge work, the bad news does not get better with age.

Large batches means that things are not delivered quickly to the customer.  And speedy delivery to the customer is essential.

Large batches allows our knowledge a much greater opportunity to decay in value. In all the many different ways it can decay in value. Often to zero.

I have to mention the 'carrying costs' of large batches.  It leads to much more 'inventory' or 'work-in-process'.  If you could see or visualize the cost of that 'partially done work' and you were a good manufacturing manager, you would be aghast at all the excess cost.

And our work is done by very expensive people.  The accumulated costs are huge.

So, while it seems (and is a bit) more efficient in some ways to have large batches, in net the costs are higher. And the 'time to market' much slower.  A lose-lose.

***

There are two short commentaries on the great article. Please read it.

Deliver Faster

Why do we want to deliver fast, and then faster?

Well, the first reason is...that's what the customer wants. The customer is thirsty. They want a drink now. She does not want to wait for 3 months to get a major delivery of 200 water bottles. A drink now, please.

The second reason: time to market.  In this context, I mean we need a good poduct to market before the opportunity goes away.  Before the competition beats us there, or before the most important problem has changed.

Third.  More and more my clients are having re-organizations rather frequently, so someone in an Agile class quipped "we need to deliver faster than the next re-org". This meant within the next 3 months.

So, yet another reason to deliver fast. If managers change, then you can deliver what the outgoing manager wanted (with a bit of luck), and then start marching forward with what the new manager wants.

But there is more. We need to deliver faster because we get more feedback.  In this way, we have a smaller release, but learn more quickly whether it is on target or not. And, taking that feedback, we can modify the next release. Or perhaps have no more releases at all.

This minimizes risk. This minimizes WIP, whose value...if we wait long enough...will always eventually go to zero.  So, we have minimized the potential loss in value of the WIP, the work-in-progress.  We have reduced the risk.

All the reasons that make us want to deliver something faster.

Yes, it must be a minimum marketable feature set.  But we are learning that 'minimum' is smaller than we thought.

Deliver faster.  This is our goal as a Team. It is not a goal that the managers stupidly foisted upon us. It is a real need in business.

Then the question becomes: how do we do that?

Why I prefer ScrumBan to Kanban

I have spoken before about why I like Lean and why I like ScrumBan, a combination of Scrum and Kanban.

Some people prefer 'Kanban', as it is being called in the software development community.

Of course, what 'Kanban' is actually in the wild varies a lot.

So, here are my concerns about doing 'Kanban' alone, without Scrum. And I make some assumptions
about what Kanban is, and these assumptions may not be correct in many cases.

1. No upfront planning.

It is fine and correct to say that waterfall does too much upfront thinking.  But this does not mean we should do zero upfront thinking.

In general people who do Scrum also advocate (as I do) some upfront planning before starting the first sprint.  I call this Agile Release Planning, and have talked about specific techniques extensively.  These are patterns that I use. Not always, but almost all the time.

In the wild, many Kanban people do not even do any Sprint Planning. Much less 'agile release planning'.  Which I think is almost always sad. Yes, I can imagine a few odd cases where agile release planning is not necessary at all.

If the Team is doing purely maintenance work (bugs and small enhancements) and has ZERO work in backlog at the beginning of the Sprint, well then I guess there is no point in Sprint Planning.  But I have never seen this situation. I have seen where a lot of the work for the Sprint will be defined later, as the high priority problems come in.  So...minimal Sprint Planning might make sense.

People (the customers and the implementers, etc.) need to see the big picture. It affects creativity about designing the better solution, it affects motivation, and makes better able to address dependencies.  But, we also must never again suffer the illusion that we have all the information upfront. That too is patently incorrectly.

2. No Team

Kanban in the wild does not require a Team concept. Certainly some people are involved, but how they are involved and whether they are a real, stable team is left totally open.

The Team concept, which involves many things (self-organization, etc, etc), is so important. And it needs to be a whole, real, stable team.  By whole, we mean it must include the business side, as we usually call it.  At least good people representing the 'customers' well. Inside the Team.

3. No Sprint.

Kanban is often played without any concept of a sprint.  Although frequently people will say "we got 20 cards done in a week", which implies a 1 week sprint concept.

There are many many advantages to having a sprint. One is so that the Team and the people around the team can see productivity per Sprint (a measurement or feedback loop).

4. 'Scrum does not allow any changes to  the Sprint Backlog'

This is a reason people give for using Kanban that is not correct.

First, it is true that Team productivity is getting killed by interruptions and whipsawing.  So, we want to minimize disruptions. And the first, overly simplistic rule to get the 'kids' straightened out is: No changes to the Sprint Backlog.

But Scrum is not opposed to common sense. So, for example, inserting 2 SPs of 'bug' work (higher priority) to replace a 2SP story (lower priority) that has not been started -- this can be done.  It should be explained to the Team why we are doing this. And the Team is not being asked to do a higher velocity than what they 'committed' to. And they get to re-commit to the new work too (eg, sometimes there are technical reasons why the 2 SP of bug work is not 'equal', this sprint at least, to the 2SP story).

So, this need to address to-be-identified-high-priority work is a false reason to switch to Kanban.

5. Dis-engagement from the Business

Kanban as played in the wild is often used to enable dis-engagement between Technology and the Business. Of course that does not have to be so, but it often is.

Often the Business side does not want to engage. The solution to this is not to give up, but to attack the issue, and 'force' them to engage.  So, I am very discouraged when I see Kanban used this way.  People seldom admit overtly this is the purpose, but it often is. IMO.

Scrum forces engagement, by making a business person (well, usually a business person) Product Owner of a real Team (and the PO is part of the Team).  And then, if the PO is not engaged enough, that should become apparent quickly as an impediment, and hopefully fixed (if high enough in priority).

6. No Daily Scrum

Kanban in the wild often has nothing like a Daily Scrum.

All Lean teams do a short daily meeting (AFAIK). Why do 'Kanban' teams not do this?  Well, they often say they don't want to stop 'continuous flow'.  But then, still, everyone goes home at night and flow stops.  Lean has a short daily meeting, because it helps the Team stay in sync, etc. And the stoppage 'cost' is repaid the same day by higher productivity.

So, I think the Daily Scrum adds a lot in enabling the Team to sync up, self-organize and self-manage.  As a Team.

(Kanban does not require any Team concept.  But sometimes it is played with a Team. So, a Daily Scrum would enable them to do a better job in collaborating, etc as a Team.)

One Team (and I think I believe they are a real Team) that uses Kanban does a kind of Daily Scrum every day at lunch. Still, I am thinking this is less effective than a regular Daily Scrum.  If done professionally.  Although, to be fair, many so-called Daily Scrums are...not really what they should be -- they are not done professionally.

7. No Sprint Review

Kanban in the wild is often played with no Sprint Review or Demo. AFAIK, Kanban does not require any demo.

Related to that, Scrum requires a Demo every Sprint. That includes the business stakeholders.

Now, in Kanban one could have demos.  Ad-hoc. For example, of each 'card' when it completes.

But this is not nearly as good, usually, because the best people to give the best feedback are the business stakeholders. They are often fairly senior or at least very busy. So, it is better for them if, fairly far in advance, they can schedule that every two weeks (or every Sprint), they will come to the Demo (Sprint Review).  This leads to better attendance and better feedback, based upon all the stories, seen together.  'The whole is greater than the sum of the parts.'

In Scrum one could still add additional feedback during the Sprint. For individual stories, as one example. There is no restriction on additional feedback.

8. No Retrospective

Kanban does not require any Retrospective.

Of course, something like a Retrospective could happen 'naturally'.

But from lots of experience, we think the odds of a regular and good retrospective go down considerably if there is no 'required' meeting to make it happen.

Again, to be fair, lots of 'scrum' Retrospectives are done in name only. And are not good or not very good. I want them not to call that kind of unprofessional work 'scrum'. But of course I cannot enforce that.

***

It is fine to add more Kanban ideas to Scrum. And specifically the Scrum board (which out-of-the-box is a basic Kanban board). And to focus on flow, pull, single-piece flow, minimizing WIP, etc.  Excellent ideas.

And sometimes you can't get a Team to do all of Scrum.  At first.  So you start with Kanban. I totally understand.  Getting people to change is hard.  But as soon as you have that change made, start working on them to make some more change.  And adding each part of Scrum, piece by piece.

Comments?

Store Points rather than Hours

Jeff Sutherland has a great post about this, here.  A must read.

Predictable project or innovative project?

Mike Cottmeyer spoke at Agile Carolinas last night. And said many good and useful things.

One thing he talked about is this:

What kind of project do you have?  At one extreme, do you have a project that is pure innovation? 
Or, at the other extreme, do you have a project that it completely 'predictable', and the main problem is good professional execution?

By 'innovative' we mean that the project or effort starts off with barely a vision, and we discover 'what we want to be when we grow up'.  Barely a vision, and only a minimal, inadequate sketch of the scope.  And we innovatively get a bucket of money to go discover some business value in that general area.

Predictable means that the managers feel, at least, that the scope and the business value are pretty well defined. They feel, at least, 'this is clearly what I want you to do....just do it.'  Usually at a fairly high level, but in their minds, it is a known, predictable thing.  Nothing close to 'pure R&D.'

Now, in practice, if you constrain a poet into sonnet format, give him a scope or domain of love and summer time, still, to him, he has realms and realms of room for creativity.  In fact, if you only said 'write a poem', he would be far less creative. But let is avoid for the moment the contradictions of creativity, beauty and innovation.

***

His (Mike Cottmeyer's) main point is that in large organizations, the agile teams are given mostly more predictable projects.

I would state the same thing slightly differently.  But it is really the same basic idea.

There is a continuum.  At one extreme, we know everything up-front. At least all the features needed.  Down to the sprint-sized story level.  And, if we were truly good, and all other factors predictable, we could accurately predict when the product would be released, or the next release.  A lot more accurately.  Etc.

At the other extreme, we no knowing up-front. Everything can and will change.  Everything is unpredictable. So, virtually any up-front planning is useless.
Which raises the issue we are driving at.

How useful is any up-front work?

And Mike Cottmeyer says (or I say he says), and I agree, that in large organizations, we at least think we know a fair amount up-front.

So, I will say most projects are between 30% and 70% known on Day Zero. By known, I mean as an example that we could identify 30-70% of the detailed stories accurately.  If we took the time.

The key point: We know a damn sight more than zero. And still a damn sight less than 100%.  In my experience.

My conclusion: We must do some up-front planning. In a timebox.  And, as we learn, we must re-plan.  I call the 're-planning'....Release Plan Refactoring.

And we must help the Scrum Team, and any other layers or people involved, learn how to manage in this 'partially known/partially emerging' situation.

Part of the learning is how to become more predictable, at least at some level.  And how to manage innovation and disruptive learnings.  These may seem very opposite skills, but in fact they are similar.

But let me go back and say it again. If we know nothing today that will remain true through the effort, then almost any up-front planning is waste.  Maybe not completely, since we must acknowledge man's irrational need to believe he knows and is in some control of the universe and his own life. So, even if we know nothing that will remain true, maybe, just to satisfy this human fantasy, we need to discuss some things upfront.  Or the morale of the team will suffer.  But mostly any upfront planning is waste.

Again, from the other side.  Any classic waterfall person or manager who de facto implies that we know almost everything upfront needs to be read a bedtime story and put to bed early.  That person is still a child, and we should get them out of the office.  They are a hazard. We never know enough to predict very accurately.

Will some effort on estimating and planning enable us to improve the probabilities of making some useful business decisions?  Yes.

Do we know enough to enable us as managers, in justice, to 'hold the team accountable for their estimates'?  No, virtually never.  And to try to do so is usually foolish, counter-productive, immoral and rather stupid.

So, again, I think we are in a middle state. We know well more than nothing and well less than everything.  And, in both ways, we must accept the consequences of this level of knowledge of the future.

Empirical Process Control

Ken Schwaber and others talk of Empirical Process Control ideas as being key to understanding Scrum.  I think this makes some sense.

Mr. Schwaber got these ideas from Babtunde Ogunnaike and W. Harmon Ray, who wrote the process bible: Process Dynamics, Modeling and Control.  Big ole book, mainly about chemical processes.

We are talking about how to build new products.  How to get business results, in the form of new products.  Innovation.  Some of you don't even want to call it a 'process'.

But to process geeks, if you build something in a half-way regular way, then that way that you build things, even if fairly irregular, is a process. Of a sort.

Ken Schwaber uses this (to the Scrum world) famous quote from Ogunnaike and Ray's book:

It is typical to adopt the defined (theoretical)
modeling approach when the underlying
mechanisms by which a process operates are
reasonably well understood. When the process
is too complicated for the defined approach, the
empirical approach is the appropriate choice.”

In very simple terms, we Agile folks take 'defined' to mean 'waterfall', roughly as defined by the famous waterfall article by Dr. Winston Royce.

Two things must be said about 'waterfall'.  First, Dr. Royce defines and shows lots of feedback loops, and most people, when they speak of waterfall, do not mean that.  Or they mean that those feedback loops are very weak and work poorly, very poorly.

Two, Dr. Royce calls for builders to 'build it once, throw it away, and build it again (correctly).'  This is virtually never done in real life, and is typically not meant when people say 'waterfall.'

And by 'empirical', we take that to mean, very simply, Scrum, as defined too quickly in the Scrum Guide by Jeff Sutherland and Ken Schwaber.

So, how do we connect the dots?  A later question is: have we connected them fairly, usefully, and with as much rigor as possible.  And a final question: is there any more light that 'process control' ideas can give us, to enable us to do our innovation/new product development work better?
***
Here are what I think of as the basics of process control, as applicable and useful to understanding Scrum better.  And the two basic methods or approaches (defined vs empirical).

This is what I think I have been told over the years.  By several different people. In fact, I may be adding and subtracting what others have said to me.

It is a very simple theory or set of ideas.  Even though it is simple, it is still (IMO) useful.

There is also a lot it does not 'explain.'  Although perhaps one could start to use these basics concepts to discuss these other things or issues.

In the simplest model, process control consists of a flow across three 'elements'.
1. Inputs
2. The black box ('the process')
3. Output

If 1 and 2 are both 'in control' and highly reliable, then 3 is likely to be reliable.  Ceteris paribus.  These are the conditions for a defined (waterfall) approach.

At the other 'end', if both 1 and 2 are 'out of control' or unreliable, then 3 is, by the definition of this model, unreliable. (Unless there is some other element that magically makes it reliable.)  These are the conditions for an empirical approach.

What does empirical approach mean?
a. We inspect 3 often (this is only common sense -- when 3 is unreliable one naturally wants to inspect it more often; one needs to), with the 'best possible' eyes, expecting it often, even usually, not to be what we want.
b. When 3 is not what we want, if we can, we pull it back to the beginning, and run it through again.  And try to 'adapt' either 1 or 2, to make them temporarily more reliable.  And pray that 3 is or becomes -- after we run it through again -- actually what we want.

We are assuming for now 3 (the widget) can be run through again, usefully. Of course, this may not always be the case.  For software, this is true.  For some physical products, this may be a bad option.

Now, the empirical approach is terrible, obviously. If one (the 'God' of the process) had any sense at all, one would change things so that 1 and 2 were both highly reliable.  And then 3 would become reliable.  That is, one would change things so that one could use the defined approach.

But, what we are saying with new product innovation with human beings, is that we never have 1 or 2 in a reliable state.

We are not God, and there is too much stuff hitting the fan.  From every direction.
So, while we might control the inputs and the black box for 3 minutes, after about 3 minutes things are unreliable again.  Sadly.  So we are always stuck using an empirical process.

But at least we understand the process we do have.

***

Personally, I consider human beings highly unreliable inputs to any process. We humans often compare ourselves to machines, but in fact we are highly unreliable.  Now, innovation and creativity are 'the unexpected'.  So, in innovation 'unreliability' is actually a good thing.  So, humans aren't that bad after all.

This very simple theory or paradigm seems very real and accurate to me. It makes sense, to me, of the mess that we are in. The tar pit, as Fred Brooks calls it.
I think this is basically what Tunde Ogunnaike and Harmon Ray meant.  At least for us.  When they compared 'defined' and 'empirical'.  But, I will ask them.

***

Now, some questions that this very simple theory does not answer.
1. What if only 1 or 2 is 'unreliable'?  (And the other one is reliable.)
2. How unreliable do 1 or 2 need to be before one uses an empirical approach (as you call it)?  One imagines that, at very low levels of 'variation' in 1 and 2, ...that the 'defined' approach would still work or be better.
(As a practical matter for software development, I find that 1 and 2 are so 'out of control' that this question becomes moot.)
3. How do you know there is not another 'element'?
4. What if you can't adapt 'enough' (on 1 or 2)?
(Logically, in the simple case, one stops working or trying to produce anything, since 3 is highly likely to be wrong.  Unless one can live with totally random success.)

Is release planning worth it?

In a word: Yes, if done professionally.

How is release planning, and release plan refactoring…how are they useful?

A few ideas:

• It enables the Team to share ideas
• It allows the Team to see the same elephant
• It enables knowledge creation
• It enables cost, benefit, time trade-offs to become apparent
• It enables everyone to start to distinguish minor decisions from major decisions

Any professional also knows that planning is not and never will be perfect. So, we must put areasonable time box on doing the planning.

It is also useful to plan with good ‘agile’ people. Meaning people who will use the information developed from planning in a useful way (eg, will not do the ‘stupid waterfall manager trick’ of expecting the Team to always hit the date they planned on Day Zero).

Let’s talk about this last one (minor versus major)…

To put things simplistically, there are two types of decisions, which I will call minor and major.  Minor decisions has a minor cost if we make them incorrectly.  If we are clever, we soon learn that we are wrong, and we correct the decision.

But some decisions are major. To change the decision later, it can be very costly in terms of money or time or reputation or some other factor.  Once we identify a major decision, we want to do our best to decide it correctly.  This means first being sure we have framed the decision question correctly.  Then, assuming this is a difficult decision, we want to make the decision at the ‘last responsible moment’.

***

Can planning be useless or worse?

Well, of course.  If you have people who will not learn.  If you have people who will take longer than the current knowledge can justify.  If you too many people who want to use the information for ‘stupid waterfall tricks’.

But if done with good people, using useful concepts, the Scrum Team (and others) can learn a lot doing Release Planning, followed by Release Plan Refactoring every sprint.

It is also true that we can learn by doing ‘real work’.  This is not to say ‘do only real work’…but one balances and shortens release planning (release plan refactoring) in the knowledge that we can and will learn some things faster by doing real work.

Joe’s Agile Release Planning

I have written a new booklet that I want you to have (I think you will find it useful) and also to comment on.

It is about Agile Release Planning.

It proposes that agile release planning consists of these major steps (at least):

1. Assemble the Team and some business stakeholders
2. Agree on the Vision (of the release or maybe more)
3. Create the Product Backlog
4. Determine the Business Value
5. Estimate the Effort
6. Consider Risks, Dependencies, Learning, MMFS and other factors
7. Order the Work
8. Do the scope-date trade-off
9. Finalize the initial plan
10. Consider the ‘communication plan’

Note that the purpose of the work is not focused on the release plan itself. The most important purpose is the Team together with the business stakeholders start seeing the same elephant.

So, I wrote a booklet to show and explain how I teach teams to do each of these steps. And to address some of the ‘lessons learned’ in doing this approach in the real world.

I hope you will read it and comment. Please download it here. It is 28 pages.

Self-organization

We have this idea in Agile, that the Team should self-organize.  This is an important idea. And also an important occurrence (eg, the reality precedes the idea).

In Agile, self-organization is compared to command-and-control.

We think self-org is an important thing to study, both in general and in your Team.

Why

Well, one, because it is just right.  People are free, and self-organization is saying that the Team is allowed to be free.

I guess this needs to be explained a bit.  Some will say: well, the company has bought their time as employees, so the company gets to define what the Team does.  Well, let is concede this at a high level; let us say that the company may define the vision or the goal or the general product.  Perhaps even the user stories.  The Team members are not slaves, but we can assume that, as employees, they agree to do the company's work for pay.  A contract.

But, devising the work, figuring out how they will get to the goal, they should have the freedom to do.

The second main answer is: self-organizing humans tend to do better work than human 'slaves' (humans directed by one or a few command-and-control people).

There is lots of evidence of this. The first book I recommend is The Wisdom of Teams by Katzenbach and Smith.  This is a rather old book.  But the basic evidence is that a self-organizing team out-performs, almost always, an individual.  And to such a degree that the extra cost is well worth it.  But, you must given them some degree of 'freedom'.

There are also a lot of more recent evidence.

Some topic you may wish to research:
Self-organization
Complex Adaptive Systems
Maneuver Warfare
Free Enterprise
Knowledge Creation

Note: Some people say that a free enterprise system is mainly characterized by (mostly) private ownership of capital (means of production). And attribute its successes or failures to that. Others focus on the freedom of individuals to make their own decisions (much like the "wisdom of crowds" idea) and on the view of the national economy as a complex adaptive system, trying to accomplish high-level economic success via many individual agents (people or companies) making their own decisions.  In our view, successful free enterprise countries exhibit many of the successful characteristics related to self-organization.

Command-and-Control Managers

Lots of managers have been taught, explicitly or implicitly, that the 'workers' are dumb, and the manager must tell the worker how to work.  In our view, especially for virtually 100% of knowledge workers (our domain), this is very very incorrect teaching.  But it is nonetheless, what they have been taught.

Now, some of them also understand freedom to some degree, and understand that they want the people in their group to 'think for themselves'. But, we can say with relative assurance, most companies have a lot of people who are relatively command-and-control in their style.

And, in pressure situations (our common situation), they want to use too much command-and-control.

Again, this is not true of all managers, but of many. It depends, in part, on the company's culture.

It is hard to convince these people to be patient for self-organization.

Teams that won't self-organize

This is seen in real teams.

There seem to be several root causes. One, the team has been beat down by command-and-control managers so much, that they have 'forgotten' how to self-organize.  Another: that they Team does not believe it when managers say 'self-organize'.  Another: The Team is fearful that by self-organizing they will be 'held accountable' with punishment a likely outcome.  To avoid pain, they refuse to self-organize.

Whatever the reason, two things can be said.

Many Team (perhaps with Agile coaches) do eventually break out from being 'stuck' (not self-organizing).

And some Teams may take a very long time or perhaps never do it.

The key advice is this: If the Team will not self-organize, or seems to want to self-organize to mediocrity, then a good manager should intervene.  Temporarily.

Perhaps prior key advice: Be patient. Often Teams will self-organize in two or three sprints.  Keep talking about it, and ask them if they need help.

Third advice: In the view of some (including me), some Teams lack some key skills to self-organize. For example, a new junior Team may not know really how to break down work into tasks for the sprint planning meeting. Sometimes 'holding their hand' as they mature is a very successful approach.  And 3 or 4 sprint later, they can be very good at self-organizing their own Sprint Backlog.

Learning to decide as a Team

I think each Team learns how to decide.

The first, most useful thing, is that everyone in the Team gets to offer input on a decision.

Next, the Team needs to accept that no decision is ever perfect.  Thus, decisions in general must be made, perhaps not always quickly, but promptly.

The Team needs to understand the impact of decisions on the morale of the Team and on the success of the Team.

In our view, most teams go through a forming and storming period of decision-making.  And then later get better.

Good self-organization... and better

Lots of Teams seems to self-organize well.

What is also common is for most Teams to plateau.

In part, we may say that a root cause is that most Teams want to reach a stasis... a place where things are balanced and where change slows down.

But have they reached the height of improvement?  We think not.  Some Teams have reached much higher heights.  And some Teams keep on improving.

There seem to be two main factors (or sets of factors):
* magic -- or, more accurately, a bunch of things that are hard to describe.
* a bunch of factors that people talk about, and some teams study and work on

Some of these last factors seem to be quite 'soft'.  Love, listening, creativity, heart seem to be among them.

Lastly

If you have never been on a good team, it is hard to understand what self-organizing is all about.

Within the Team, it might be rather rough and ready. But a lot depends on the specific individuals in the Team.