Why do Scrum?

If we hear about Scrum and want to convince some managers to do it, how should we present the benefits of this approach?

The 2011 Agile Survey by Version One gives some ideas.  See it here:
http://agileconsortium.pbworks.com/w/page/51511217/2011%20State%20of%20Agile%20Survey

Typically the biggest idea is: faster time to market.  And this is very powerful idea.

But it is an idea that is hard to measure for many firms. Or perhaps better to say that the firms, in my experience, do not discuss what they are measuring with the community (eg, the employees) nearly enough. For example, even if we deliver 12 units of functionality in a year, both before and after, it is still a huge win to deliver that in 12 small releases (with agile) rather than in one big release at the end of the year (with waterfall).

Some other key reasons.

Manage Changing Priorities.

Increased productivity.  Scrum should provide greater productivity for the Team per Sprint. This is probably best measured by Velocity, meaning total Story Points 'done' at the end of the Sprint.

Better Align IT/Business.

Enhance Software Quality. We think a strong Definition of Done and more automated testing naturally should lead to higher quality with agile. But no doubt some people who say they are doing "agile" are producing products of lower quality than some waterfall teams.  But in general, Scrum should have higher quality. A lot higher.

Project Visibility. This is one for the managers.  They can just see better; is the project doing well or not.  Whatever the answer is, I just want to know the truth. If things are not good, then I can so something about it.

There are other reasons (in the survey and not), but this is a good start.

Dr Royce and Waterfall

In 1970, Dr. Winston Royce wrote the paper on Waterfall.  And defined it.

Here is the paper:  http://agileconsortium.pbworks.com/w/page/52184647/Royce%20Defining%20Waterfall

He identified 5 things that must be added to reduce most of the risks of doing waterfall.  And I will comment on how Scrum addresses these.


1. Program Design Comes First

Well, I think he may have been mostly right then.  But I would rather say that 'solution design' comes first. Meaning that we most truly try to understand the customer's real problem (not what he says it is) and try to design a full solution to that problem.  Not just, for example, software, but the full solution. And, assuming the product is software, then we understand better where our product plays in the fuller solution.

Scrum does not address this issue directly.  Implicitly Scrum is saying "it is hard to know whether your product will be good. So, build incrementally and try to enhance your feedback, to prove to yourself that the product will promote the solution."

Let me add this: Some people who are doing 'cowboy agile' in fact do not do enough up-front thinking.  In my experience.  Yes, it is fair to say that we learn more from working software than by just 'thinking'.  But some still don't 'look before the leap'.  But this is a hard thing to discuss in the abstract.  Because how much to think up-front will vary depending on many details in your specific situation (for example, the professional instincts of the implementer).

I have also seen Scrum teams get into analysis paralysis, where they are thinking still too much up-front.  In my experience, it is well worth the team's time to discuss continually "well, how much should we think up-front about this piece?"  And they will get better at making intuitive judgments about this that are appropriate for their specific work. 

2. Document the Design

Agile and Scrum do not have Dr. Royce's level of confidence in documentation.

And Agile and Scrum see or assume a greater urgency to meet 'time to market' demands. Typically a high focus on documentation leads to long analysis paralysis periods.

Agile and Scrum do agree that knowledge must be shared. But rather than share knowledge mainly through documentation, the community now suggests doing it many other ways. Here are two: (1) incrementally built working product (2) lightly documented automated tests (where at all possible).  Agile and Scrum people tend to prefer cards (middle-level summaries), white boards, large sheets with drawings in team rooms, etc.

So, information is hopefully shared more, and the sharing is actually better.

Speaking for myself, I do agree with Dr Royce's statement that many implementers tend not to want to write even basic documentation.  And I agree with his statement that much of the purpose of documentation is for use by people later.  So, those needs must be carefully considered and addressed. 

This all results, if done professionally in two things:
1. We write much less documentation than at least I used to when doing 'waterfall'.
2. We ask many people in the Team to write more documentation (in a wiki or somewhere) than they want to write.

3. Do It Twice

Yes, Dr. Royce said "build it once, throw it away, and then build the real thing".  And his son later complained that no one really did this, so no one was really doing waterfall the right way...

The basic problem is that we are always learning, even in the last stages of the work. And that learning needs to be fed back into the final product. This was hos recommendation for doing that in waterfall.  No wonderful the waterfall products are seldom what the customers really need now.

Scrum solves this differently, by continually refactoring the current product to align with the latest knowledge.  And doing this in the 'working product' every sprint. 

This means, yes, Virginia, there is 're-work.'  And the Team should be always asking: 'How could we have avoided some of this re-work?'  And sometimes they will come upon ideas to get better next time.

This is not suggesting or expecting to arrive at a state of 'perfect knowledge up-front'.  This will never happen. But we can learn to consider some things before we leap. Example: Is our design for this story consistent with the rest of the design and also appropriate for our complete and current understanding of what the overall solution needs?

4. Plan, Control, and Monitor Testing

Dr. Royce has some good ideas about testing.  Some of them have been superseded and some new things have come onto the scene as well.

The key thing to say, though, is that Scrum agrees so much with the importance of testing, that Scrum said: let's do it from the very beginning (well, at least from the first Sprint).  In part, to tighten the feedback loop and to reduce the cost of implementing the changes we get from the feedback.

5. Involve the Customer

Yes!  Very important.  Again, Scrum considers this such an important idea that we do it in the form of the Product Owner, who is an essential daily part of Scrum.  In my opinion, the PO role should involve the customer in a much more practical and useful way that the ideas that Dr. Royce proposes for waterfall.

***

Much more to say, but let me keep this post short.

Early in the paper Dr. Royce says: "I believe in this concept [waterfall], but the implementation described above is risky and invites failure."  I don't know, but I think he meant "Compared to cowboy coding [no process], waterfall, with the 5 'additions' that I will propose, seems a lot better."

Now let me quote Dr. Royce's last lines about waterfall:

"[This] summarizes the five steps that I feel necessary to transform a risky development process
into one that will provide the desired product. I would emphasize that each item costs some additional sum
of money. If the relatively simpler process without the five complexities described here would work
successfully, then of course the additional money is not well spent. In my experience, however, the simpler
method has never worked on large software development efforts and the costs to recover far exceeded those
required to finance the five-step process listed." [emphasis added]

Waterfall versus Scrum: How do they compare?

I have recently been asked for advice on how to compare Waterfall (WF) and Scrum.  This is a hard question in some ways: the best way to compare will depend on the person you are talking to.  Another problem is that few people do 'pure' waterfall in actual practice. And the variations of Waterfall are many.

Still here are some thoughts that occur to me.  We think these statements often need explanation, but they can be useful as a start for a discussion, eg, 'should we move from waterfall to Scrum?'

***

Scrum has a prioritized Product Backlog, from which one could execute the Pareto Rule.
WF tends to assume the 100% - 100% rule.   And tends to order the building mainly by technical dependencies or by what the geeks want to do first.

Scrum uses adaptive planning.
WF tries to keep to the initial plan.

Scrum gets feedback from working software early (first sprint, in 2 weeks) and often.
WF does not have working software until very late in the cycle.

WF assumes we know 'everything' upfront.
Scrum assumes there is lots we don't know (yet), and focuses on maximizing learning throughout the project.

WF tries to minimize change (change control board).
Scrum tries to maximize the benefits of good change (eg, learning).

Scrum assumes change is normal, often good, and anyway much of it can't be controlled usefully.
WF assumes change is bad and can be controlled.

WF puts most responsibility on the PM.
Scrum puts most responsibility on the small dedicated Team.
(WF uses a diffuse work group, with only the PM with serious responsibility.)

WF assumes the PM should plan the work.
Scrum assumes it is best if the Team plans its own work and re-plans.

WF prefers 'planning from the center' (e.g., by the PM).  And the workers just execute what they are given.
Scrum prefers more self-organization, and using the full 'complex adaptive system' (a tight thinking adaptive Team).

Scrum optimizes business value, time to market and quality.
WF optimizes conformance to schedule and budget.

WF's typical problem is analysis paralysis and being late. Scrum addresses these.
People using Scrum often err in not thinking quite enough up-front....an easier problem to fix.

When people do WF they typically are thinking: "If I just follow the process and the plan, all will be well."
Scrum tries to force the Team to think for themselves, in their specific situation. Hence, it is a bare framework of things that are essential for every effort. Always other things, in addition to Scrum, must be done.

WF has weak controls (eg, conformance to schedule).
Scrum has strong and frequent controls (e.g., did you all get all those features done this past 2 weeks?)...which enables faster improvement.

Waterfall only allows realisation of value only upon completion.

Scrum supports realisation of value earlier, potentially after every sprint.

Notes:
Unprofessional people often say they are doing 'Scrum'.  Above we are talking about vigorous professional Scrum, not 'cowboy Scrum' or ScrumButt.

Good people forced into WF typically use agile-scrum ideas to make it really work.  Above we are talking about 'true' Waterfall. While 'true' Waterfall rarely exists, we think that one can still see that the ideas and practices of Waterfall, as much as they remain in real practice, impede progress.

Final comment: I am not convinced that these are all the statements one could make. Also, please don't assume I have all the top 5 best comparison statements. Your feedback is very welcome.

Agile Transformation - 1

I was leading an Intermediate CSPO course last week in Winston-Salem.  Some good questions. One theme was about the agile transformation.

The idea is simple, and has many names.  One way to say it: Only by transforming the culture and many of the current 'ways of doing things' can a firm realize the fuller value of lean-agile-scrum.

So, a couple of related sayings:
"Culture eats strategy for breakfast."
"Once you implement Scrum, everything must change."

But before I discuss those (both are useful, but neither fully correct) -- let me discuss what an agile transformation might be.

Most firms typically start with one or two pilot Scrum teams.

Then they expand to say 5 Scrum teams.

Then they can notice that a few managers are working on the impediments of the five teams.  And someone suggests an "impediment removal team" composed of those managers. And that they act like a Scrum team, except that their product backlog derives from the impediments of the 5 Scrum teams.

Then they decide to expand to more teams.  And now people start noticing, in a bigger way, all kinds of impediments that are 'cultural'.  How we compensate people, who reports to whom, performance appraisals, management metrics, how different departments interact, etc, etc.

People also notice that starting teams is serious effort.  Training is costly and difficult to arrange for lots of people. The teams should have agile coaches; how does that get arranged.  The teams need team rooms.  How do you control that old teams don't revert to waterfall.  Are the managers (business and technology) truly understanding the goals of agile?

And the firm hears that other firms at this stage underwent an "agile transformation".  And they wonder what that means and how they might do it.

We, and lots of experienced coaches, recommend an 'agile transformation team'.   The concept here is some more senior managers take on "transformation stories"much like a Scrum team would.

First, we strongly favor this approach, but it is hard, and it needs to be done professionally.

Some problems with this:
1. The stories start off vague or huge.  Example: "Change the waterfall mindset." Nice idea, but how would you ever prove that you did it?  So, you break that into small 'testable' stories (work items).
2. The Team is senior people.  Meaning, that they have forgotten how to work as a real team, and they have forgotten how to do 'real work'.  But some senior people are needed, or this team has no clout.
3. The Team is mostly senior people.  Meaning: who will do some real work. We recommend having some less senior people on the team, who will do more real work.
4. How do you measure the team's productivity?  This is hard, but if they can't show somehow some "data", then how do you decide if the benefits are worth the costs?

Two more key ideas:
A. There should be leadership from the top.  If we are talking about GE, maybe Jeffrey Immelt does not have to talk about agile, but someone pretty senior does.  To get the most benefits.
B. There should be leadership from the bottom.  The senior guys are much more comfortable pulling agile if they know 'real people' also like it.  Leadership from the bottom means an enthusiasm to build and expand agile. An to take some actions to make that happen.

A start on this huge topic.

***

2011 State of Agile Survey

Below is the 2011 Survey by Version One.  One might like yet better participants in the survey, but still I think it is generally indicative of the main trends.

http://agileconsortium.pbworks.com/w/page/51511217/2011%20State%20of%20Agile%20Survey

Some key points to me:
* We have a long way to go
* Too many people "rolling their own"
* Despite the stupid things, still a fair amount of success
* Key prospective reason: Accelerate time to market (achieved by 71%)
* Key retrospective benefit: manage changing priorities (arguably a subset of faster time to market): Achieved by 84%