Joe's Unofficial Scrum Checklist V1.3

It is Memorial Day weekend and time for another edition of Joe's Unofficial Scrum Checklist.

No, not really.

But someone in class asked what he could use to check that his team's were using Scrum well.  I suggested:

1. The ScrumButt Test - 8 points.
2. "A list summarizing Scrum" (V.5)  - 2 sides of one page.  Fairly packed.
3. Joe's Unofficial Scrum Checklist (Now V1.3)

Well, I did not at that time recommend #3.  But I do now.

I guess it bears repeating: We only seek to do Scrum better because we believe in almost all cases that better Scrum leads to better results.

PO Impediments - Charlotte May 2013

Several quick things to say about impediments:

1. Few teams have a public impediment list. They should.
2. Few teams are aggressively attacking impediments, as I see it. (I guess this depends on what one means by aggressive. I mean full time alienation of impediments. Mercilessly.)
3. We must use impediments, and continuous improvement to get higher velocity in the next sprint.  Quick results. Maybe not every time, but most of the time.
4. Most teams do not consider impediments of every type.  For example, often teams will not even think about product owner related impediments.
5. No two people completely agree om what exactly is or is not a PO related impediment. And, some say that every impediment is 'PO related'. ...Fair warning.

Connected to these thoughts, in a recent Scrum class we identified the following Product Owner related impediments:

Introvert or difficulty selling or evangelizing
Process Understanding (lack of)
SME Level (not enough)
Allocation
Wrong priorities = constant change
Inappropriately defined user stories
Inability to negotiate or find middle ground
Heavy Req phase
Rigid UX Framework
Hand-off mentality
Ineffective Scrum
LOB participation (lack of)
Competing priorities
Release Mgmt Std Requirements
Integrating Agile with waterfall teams
Teams too large
Lack of expertise
Lack of empowerment


Perhaps this list suggests some useful thoughts to you.

Predictable project or innovative project?

Mike Cottmeyer spoke at Agile Carolinas last night. And said many good and useful things.

One thing he talked about is this:

What kind of project do you have?  At one extreme, do you have a project that is pure innovation? 
Or, at the other extreme, do you have a project that it completely 'predictable', and the main problem is good professional execution?

By 'innovative' we mean that the project or effort starts off with barely a vision, and we discover 'what we want to be when we grow up'.  Barely a vision, and only a minimal, inadequate sketch of the scope.  And we innovatively get a bucket of money to go discover some business value in that general area.

Predictable means that the managers feel, at least, that the scope and the business value are pretty well defined. They feel, at least, 'this is clearly what I want you to do....just do it.'  Usually at a fairly high level, but in their minds, it is a known, predictable thing.  Nothing close to 'pure R&D.'

Now, in practice, if you constrain a poet into sonnet format, give him a scope or domain of love and summer time, still, to him, he has realms and realms of room for creativity.  In fact, if you only said 'write a poem', he would be far less creative. But let is avoid for the moment the contradictions of creativity, beauty and innovation.

***

His (Mike Cottmeyer's) main point is that in large organizations, the agile teams are given mostly more predictable projects.

I would state the same thing slightly differently.  But it is really the same basic idea.

There is a continuum.  At one extreme, we know everything up-front. At least all the features needed.  Down to the sprint-sized story level.  And, if we were truly good, and all other factors predictable, we could accurately predict when the product would be released, or the next release.  A lot more accurately.  Etc.

At the other extreme, we no knowing up-front. Everything can and will change.  Everything is unpredictable. So, virtually any up-front planning is useless.
Which raises the issue we are driving at.

How useful is any up-front work?

And Mike Cottmeyer says (or I say he says), and I agree, that in large organizations, we at least think we know a fair amount up-front.

So, I will say most projects are between 30% and 70% known on Day Zero. By known, I mean as an example that we could identify 30-70% of the detailed stories accurately.  If we took the time.

The key point: We know a damn sight more than zero. And still a damn sight less than 100%.  In my experience.

My conclusion: We must do some up-front planning. In a timebox.  And, as we learn, we must re-plan.  I call the 're-planning'....Release Plan Refactoring.

And we must help the Scrum Team, and any other layers or people involved, learn how to manage in this 'partially known/partially emerging' situation.

Part of the learning is how to become more predictable, at least at some level.  And how to manage innovation and disruptive learnings.  These may seem very opposite skills, but in fact they are similar.

But let me go back and say it again. If we know nothing today that will remain true through the effort, then almost any up-front planning is waste.  Maybe not completely, since we must acknowledge man's irrational need to believe he knows and is in some control of the universe and his own life. So, even if we know nothing that will remain true, maybe, just to satisfy this human fantasy, we need to discuss some things upfront.  Or the morale of the team will suffer.  But mostly any upfront planning is waste.

Again, from the other side.  Any classic waterfall person or manager who de facto implies that we know almost everything upfront needs to be read a bedtime story and put to bed early.  That person is still a child, and we should get them out of the office.  They are a hazard. We never know enough to predict very accurately.

Will some effort on estimating and planning enable us to improve the probabilities of making some useful business decisions?  Yes.

Do we know enough to enable us as managers, in justice, to 'hold the team accountable for their estimates'?  No, virtually never.  And to try to do so is usually foolish, counter-productive, immoral and rather stupid.

So, again, I think we are in a middle state. We know well more than nothing and well less than everything.  And, in both ways, we must accept the consequences of this level of knowledge of the future.

A purpose for Agile

In general, it is useful to do the most important things.

Not the things we are most sure about.  Not the things that we can do well.  Not the things we can predict well (or better).

But, we should do the most important things.  Usually one at a time.

As though we were human beings. In a sensible world.

Yogi Berra said:

You have to be very careful if you don't know where you are going, because you might not get there.

So, where are we trying to get to with Agile and Scrum? With what I call lean-agile-scrum?

 I think the purpose is to make lives better.

Our own life first.  (I am imagining the ScrumMaster in the Team. My life, she thinks.)  We must first take responsibility for that. And being self-sufficient in this world, maybe even net net contributing.

Then we want the lives of everyone in the Team to be better.  A small group. Almost manageable in size.  People we get to know as people. We don't just 'love humanity', but real, specific, gnarly people.  And we help them make their lives better.  And help them become (better) contributors to other people's lives.

Still, we do not live for ourselves alone.  So, we ask the Team to build something for other people, for the Customers we often say.  This is a good and great thing.  Especially if we can see the Customers as a real person, as a group of real people. (Hence, as one small example, we use the roles in the User Story cards.)

So, we contribute to the lives of others. Perhaps our contribution is small and pleasureable,  Perhaps it is dramatic, maybe even life-saving.  Perhaps it has a spiritual dimension. Or it helps take care of the basic needs of those whose basic needs are not yet assured. I think those different choices, all a matter of personal choice, are not so important. What is essential is that the Team do something for someone else.  Someone outside themselves.  That they love their neighbor, in a small way, as themselves.  They give to others as they would want to be given to.

In economics, we can call it a business transaction. We provide a service and get paid for it. But in real life, we give something of value (X-y) to us, and it has value (X+z) to the Buyer.  And the Buyer only pays X.  So, in real life it is a win-win for everyone.  That is, if people were mostly rational.  Which is an hypothesis we will work with.

So, it is a business transaction, and we are successful, we hope, business-wise.  But some of you will have noted the words I used. Much like the famous words of some great teachers.  It is also a spiritual event, even though at the time we don't make it a 'precious' spiritual event.  We don't become overly conscious of its spiritual dimension.  And even better: we hardly engage the maya of becoming spiritually 'better'.  We just do our duty, have some fun with the Team, and, one hopes, we learn at the same time some important lessons.

If you see Scrum in these terms, does it guide you in adding to it and in playing it in your real situation?  Scrum is very practical; use it well.

They still want us to deliver too much in too little time!

In a class, we had a large group of people from one company.  And the company is doing or getting close to doing mostly Scrum.

But the managers and the Board have not been to a Scrum class.

In any case, 'management' is still asking the Teams to try to deliver too much in too little time. And say both the scope and the date are inflexible. Or at least, that is how it is heard.

What is the solution?

This is one of the hard problems of our work.  There is no easy answer.


I think it boils down to this.

1. Customers want things quickly. And time itself has a value.  For example, we must get to market with our new product before the competition.

2. On the other hand, as they said in Latin some 2,000 years ago, to predict is difficult, particularly of the future.

So, we (the Scrum Team) must make predictions, and try to make them real.  AND, we must get everyone to understand all the changes that will happen. And all the variability in our 'predictions'.  When we think the next release will be done and how much will be in it.

These are not easy conversations. Usually. People mis-understand each other.  Nonetheless, you must have the conversation.

Sometimes we say: we have a fixed time, a fixed scope and a fixed budget.

But always this is never completely true (in my experience).  There is always some proposed small features or featurettes in the work that do not have to be done in the current release.  This gives us an incentive to break down the features smaller, so we can identify small things to move to 'later'.

Always there is a value to delivering fast. Usually, even before the current 'date' would be real good. Often very very good. (Which gives us yet more reason to reduce the number of small features in the release.)

And usually cost is NOT the main driver. There is (or should be) a huge ration between benefit and cost, so that being stingy about cost is silly. Especially when time is so important.

Lastly, in this too brief discussion, we must mention impediments. In Scrum, we want the Team to identify their biggest impediments.  Things that, if we fixed them, we could double (again) our velocity. Not by magic, not by working harder, but by fixing impediments. In the wetware, in the random carbon units, in the automated testing, in the management culture, whatever.  And often fixing impediments costs serious money.  So, we need good judgment to spend wisely.

Finally, we need to use Agile Release Planning to start to identify a reasonable scope and date, initially.  And, as things change (and everything will change at least some), we can revise and revise and revise it, and do the best we can to innovate in the time boxes we ultimately decide to give ourselves.

***

Managers must learn that if they pressure the Team (to deliver more in a given time), the Team usually hears two things: (a) cut quality, and (b) work high overtime.  So, the Team and managers (or customers) must talk about this.

Almost always the managers do not want reduce quality or overtime (bad quality of life). What they want is 'creativity'.  Someone to cleverly see a simpler way to do things. How to remove an impediment. How to do fewer features. Something. Something unexpected that we might not have imagined without a challenge.

So, talk about this. Start to understand each other better. You need each other.

Question: How to order Stories in the PB

Question from a former Class Attendee:

Another question, this time about ranking user stories.

In the recent Scrum Master course, you indicated we should rank stories in the PB (Product Backlog) by Business Value and then do them from the top down. In the Product Owner course back in 2011, you had us calculate R = BVP / SP to arrive at the cost / benefit value and then rank by R. This would have the effect of doing longer items ahead of shorter items that have somewhat less business value.
What do you recommend now? What are your current thoughts?

Reply:

I explain this at a decent length in my small book on Agile Release Planning.

I am still a fan of the R factor.  AKA cost benefit analysis.

And a fan of making stories small.  To me, all epics eventually must be broken down so that we can get 8 items (stories) in a sprint.  So, if we have a velocity of 20 for a 2 week sprint, then items average about 2.5 story points in the sprint.

So, as stories rise to the top and get broken down, we have to re-point (both BVP and SP). And thus the R and the ordering will change.

Not quite sure what your concern is, but I think that resolves it.


The order can never be 'purely' business value.  We have other things to consider (which we might also call business value, but to me it gets too complex to put it that way....).  Things like Risks, Dependencies, Learning, MMFS (minimum marketable feature set), and Other factors.

So, I like to use R factor. Rank by that. And then have the Team discuss changes to that ordering.
Remember that to me the most important thing is that the Team together see the same elephant.

In the long term (or short term??) the PO wants to maximize the business value delivered out of the Team.  I think.  As a contrasting example, the PO is not optimizing the 'efficiency' of the Team (that each hour is used 'productivity').

Help?

The Organzation of our Scrum course

First, is a Scrum Team organized?

Well, a good Scrum is more adaptive, probably, than it is ‘organized’.  Of course, we can debate the meaning of these words, but during a day or during a sprint, or during a release, I usually would rather that the Team be adaptive, than that they follow an organized plan. As one example.
 

This is one small reason that our Scrum course is not organized in a strictly logical way.
Second, why do Scrum Teams fail?

Well, there are many reasons. One key reason that is not true: They were not intellectually able to fight through the complexity of the explicit knowledge around the bare framework of Scrum.

In fact, the bare framework of Scrum is very very simple. (On purpose.)  And understanding the explicit knowledge around that is quite easy.

Hence, we are not worried that we need to organize the course so that the attendees build in their minds ‘complexities upon complexities’ about Scrum.  If Scrum were complex, for example, like the
Calculus, then we would have to organize the course a different way.

Again, Scrum is ‘holistic’ or interdependent.  One cannot understand one part of Scrum without understanding how it works with or plays with another part of Scrum.  ‘No man is an island’ as John Donne famously said.  So, we like to continually weave from one thing to another, so that this weaving starts to be embedded in the ‘back’ minds of the attendees.

So, one of the key problems is tacit knowledge. Getting the tacit knowledge and all that that means into their heads.  But, honestly, not just into their heads, but their hearts, their souls, their guts, their bodies.

And one of the big problems is that the attendees, or many of them, resist intellectually.  So, as in Zen, we have to confuse the intellectual mind in order to enable real learning to happen faster.  Or, as the Army says, we have to break them down in order to build them back up again.

We have to ‘go around’ or ‘get behind’ the intellectual resistance that is common to just about all of us. So, one technique is to do exercises. But not following a highly logical flow to the course is another technique. Surprising the attendees (in small ways) is another technique. Humor is another technique.  Improvisational exercises is another technique. Food is another technqiue.  Addressing them, and getting to know them, as a person is another technique.

For some, our techniques (and there are actually many) are…umm, disconcerting. If one is the organized, intellectually rigorous, ‘it is all about thinking and logic’ type of person, it can feel a bit uncomfortable.  But if one has at least an intellectual understanding or some real experience that says that people and real life do not always follow our pre-conceived intelectual notions, then it is not so uncomfortable.

So, I admit that the course to a new person, or at least a few, can feel uncomfortable. (Actually, my impression is that most people enjoy it. About 95%.  But not all.)

If, in the course you tell me you have that feeling, then I will offer some advice. First, that we will address the topics, or most of them, that are on the one-page (two-sides) outline of ‘Scrum’ we hand out (it is really more than just Scrum).  And we will follow, mostly, the outline on the website. (Except not in that order.)  And that we will follow the slides, pretty much sequentially.  Except that we will cover additional things that are not shown on slides.

We have a strong confidence that most real learning is not logical. It happens in the sub-conscious mind, where a whole bunch of experiences are ‘put together’ by the brain into a ‘logical’ way of looking at the world.  Assembled into a pattern or set of patterns. And we are forcing your brain to break down old patterns, and rebuild all that ‘stuff’ into new patterns. And we have a strong confidence that all of out attendees can do that.

And we also know, sadly, that many are so much ‘controlled’ by what we call ‘waterfall ideas’ that they will not be able, after only 2 or 3 days, to really replace the waterfall patterns with agile/scrum patterns.  So, we are sad when we do not succeed in this way.  It does happen.

Would we succeed better if we presented things in a more organized, more logical way?  Well, in a very small sense with a very few people, they might at least say ‘it was a good logical presentation’.  And they, that small group, would feel better.  But we are completely convinced that, if you look at the overall results, we would be much much lower.

Remember that our goal is not teaching. Nor learning. Nor even action by the attendees. Our goal is to achieve real results with Scrum. For the person, for that person’s team, and for that person’s customers. One never will achieve real results with merely a ‘logical understanding’ of our work.

We are not after explicit knowledge. We are after ‘a sense of urgency’ and the tacit knowledge that leads to successful results.

I wish you every success in having fun in achieving real results.

ROI for Scrum Training

Does Scrum Training give a good ROI?

Well, of course, that depends. Mainly, how whether the Team (the full Team) takes an aggressive attitude toward improvement.

But let's look at the following calculation.

We start with the a Team that, fully loaded, costs about $1 million per year.
Their current Business Value delivered after 1 year's work....taking the NPV (net present value) of all future cash flows, is currently $3 million.  Now, maybe you can follow most of the rest below.

Financial Benefits Estimate
Cost of Team	$1,000,000
Business Value Delivered by Team	$3,000,000
Note: This is NPV from work delivered in one year
Improvement Factor	2
Note: Reasonable improvement factor in 12 months.
Note: Jeff Sutherland is looking for a factor of 5x-10x.
Note: This is usually measured as an improvement in velocity.
BV Run rate after improvement	$6,000,000	per year's work
Gross BV Improvement	$3,000,000	per year's work
Note: Per year!
Note: Thus, this is a LOW (conservative) estimate
Investment required to obtain improvement	$750,000
Note: This includes many things, mainly accumulated cost of impediment removals
Note: This is a HIGH (conservative) estimate
Note: At some point, a lower investment could correlate with a lower improvement factor.
Net Net BV Improvement	$2,250,000
Return on Investment	300%

Let's talk about the investment of $750,000

So, this includes the cost of training.

This includes the cost of travel to the training.

This includes the time lost while training.

This includes the cost of removing impediments for the Team (this is by far the biggest cost).

Removing impediments may require servers, software, training on automated testing, etc, etc, etc.

To be honest, we think $750,000 is a gross over-estimate of the cost of getting a 100% improvement in velocity.  It will cost MUCH less. Maybe $100,000 or $200,000 or $300,000 -- depending on your situation.  If the cost is 'only' $300,000, then the ROI after one year becomes 900%.  Or 9x.  That is huge.

Where would I invest first?

1. Train the whole Team.
2. Get an Agile Coach (I won't debate here how much time the coach should be dedicated to the Team. But a coach for one Team.)
3. Improve the Product Owner
4. Improve the continuous builds
5. Improve automated testing
6. Improve integration and regression testing, so that they are much more robust

Almost always, these are the top areas.

Each Team also has its own specific things, things unique to that Team.

Some Teams are fundamentally dysfunctional.  Some Teams need people skills.  Or facilitation on decision-making. Or training in specific skill sets.  Lots of other possibilities.

The key thing is that you see that we think starting Scrum is the key to releasing all these benefits.

And that it will not be Scrum alone that releases the benefits.  Getting all the benefits will require hard work and further investment.  But it starts with Scrum.

And Scrum, if played professionally, has a huge ROI.  Huge.

Note: Here is a link to a spreadsheet, so that you can do your own calculations.  Use different assumptions, and see what you get as an ROI.

Public Impediment List: "We don't want to see the bad news."

The Scrum Guide does not mention it, by I strongly advocate a public impediment list.

The simple idea is: visual management, and single piece flow off the 'top' of the list.

What's the problem?  There are several, and let's discuss two today.

1. Some impediments are personal and should not be put on the list.

To be this is easy. An example that is fun to talk about is that Sarah and Sunil are having an affair.  And some or all of the team knows about it, and it is 'disrupting' the team in some way.

The issue: listing this specific personal impediment on a public forum will not help. Fair enough.
And the easy solution is that those 'personal' impediments should not listed, at least in the wrong way, on the public impediment list.

2. 'We don't want to see the bad news.'

This is actually a very common and a very hard problem.

Often members of your organization -- while they may never say it this way -- will not want to see the problems in the organization.

Sometimes this will manifest as a denial of some of the impediments. And, to be fair, there should be a healthy discussion about whether all things mentioned are really impediments. But certainly we see people 'defending' things that are really impediments.

A related factor, though, is the wish to feel good about ourselves. And the impediment list makes us, often, see that we are....more imperfect than we sometimes want to admit.  This is hard on the organizational ego.

So, building in some humor, and showing the value of always striving to be better... these are very important things to discuss.

Intermediate CSPO Course

Scrum is, in a way, simple.

But we think that, for many reasons, doing Scrum well requires continuous study.

For one thing, we need to do the practices in harmony with the values and principles behind lean-agile-scrum.  And we are always forgetting the values and principles.

But there is more to it than that.

You may have taken our CSM course. If you have done Scrum for a while, you should seriously consider taking the intermediate CSPO (Product Owner) course + Workshop. If you are a PO, the reason should be obvious; you need to take your skills to the next level.  This will greatly benefit the Team.

If you are a SM, you need to understand the PO role much better, so you can coach your PO to become better.

The course is also good for manager's, business stakeholders, and business analysts.

Our next Intermediate CSPO Course + Workshop is in Charlotte on May 21-23. I hope you can join us. See here!

But my main point here is not this specific course. My main point is the need for continuous education, so that you, your team and your customers realize the full value of scrum.  This course in May or the CSPO course more generally is only examples of that continuous learning.

Empirical Process Control

Ken Schwaber and others talk of Empirical Process Control ideas as being key to understanding Scrum.  I think this makes some sense.

Mr. Schwaber got these ideas from Babtunde Ogunnaike and W. Harmon Ray, who wrote the process bible: Process Dynamics, Modeling and Control.  Big ole book, mainly about chemical processes.

We are talking about how to build new products.  How to get business results, in the form of new products.  Innovation.  Some of you don't even want to call it a 'process'.

But to process geeks, if you build something in a half-way regular way, then that way that you build things, even if fairly irregular, is a process. Of a sort.

Ken Schwaber uses this (to the Scrum world) famous quote from Ogunnaike and Ray's book:

It is typical to adopt the defined (theoretical)
modeling approach when the underlying
mechanisms by which a process operates are
reasonably well understood. When the process
is too complicated for the defined approach, the
empirical approach is the appropriate choice.”

In very simple terms, we Agile folks take 'defined' to mean 'waterfall', roughly as defined by the famous waterfall article by Dr. Winston Royce.

Two things must be said about 'waterfall'.  First, Dr. Royce defines and shows lots of feedback loops, and most people, when they speak of waterfall, do not mean that.  Or they mean that those feedback loops are very weak and work poorly, very poorly.

Two, Dr. Royce calls for builders to 'build it once, throw it away, and build it again (correctly).'  This is virtually never done in real life, and is typically not meant when people say 'waterfall.'

And by 'empirical', we take that to mean, very simply, Scrum, as defined too quickly in the Scrum Guide by Jeff Sutherland and Ken Schwaber.

So, how do we connect the dots?  A later question is: have we connected them fairly, usefully, and with as much rigor as possible.  And a final question: is there any more light that 'process control' ideas can give us, to enable us to do our innovation/new product development work better?
***
Here are what I think of as the basics of process control, as applicable and useful to understanding Scrum better.  And the two basic methods or approaches (defined vs empirical).

This is what I think I have been told over the years.  By several different people. In fact, I may be adding and subtracting what others have said to me.

It is a very simple theory or set of ideas.  Even though it is simple, it is still (IMO) useful.

There is also a lot it does not 'explain.'  Although perhaps one could start to use these basics concepts to discuss these other things or issues.

In the simplest model, process control consists of a flow across three 'elements'.
1. Inputs
2. The black box ('the process')
3. Output

If 1 and 2 are both 'in control' and highly reliable, then 3 is likely to be reliable.  Ceteris paribus.  These are the conditions for a defined (waterfall) approach.

At the other 'end', if both 1 and 2 are 'out of control' or unreliable, then 3 is, by the definition of this model, unreliable. (Unless there is some other element that magically makes it reliable.)  These are the conditions for an empirical approach.

What does empirical approach mean?
a. We inspect 3 often (this is only common sense -- when 3 is unreliable one naturally wants to inspect it more often; one needs to), with the 'best possible' eyes, expecting it often, even usually, not to be what we want.
b. When 3 is not what we want, if we can, we pull it back to the beginning, and run it through again.  And try to 'adapt' either 1 or 2, to make them temporarily more reliable.  And pray that 3 is or becomes -- after we run it through again -- actually what we want.

We are assuming for now 3 (the widget) can be run through again, usefully. Of course, this may not always be the case.  For software, this is true.  For some physical products, this may be a bad option.

Now, the empirical approach is terrible, obviously. If one (the 'God' of the process) had any sense at all, one would change things so that 1 and 2 were both highly reliable.  And then 3 would become reliable.  That is, one would change things so that one could use the defined approach.

But, what we are saying with new product innovation with human beings, is that we never have 1 or 2 in a reliable state.

We are not God, and there is too much stuff hitting the fan.  From every direction.
So, while we might control the inputs and the black box for 3 minutes, after about 3 minutes things are unreliable again.  Sadly.  So we are always stuck using an empirical process.

But at least we understand the process we do have.

***

Personally, I consider human beings highly unreliable inputs to any process. We humans often compare ourselves to machines, but in fact we are highly unreliable.  Now, innovation and creativity are 'the unexpected'.  So, in innovation 'unreliability' is actually a good thing.  So, humans aren't that bad after all.

This very simple theory or paradigm seems very real and accurate to me. It makes sense, to me, of the mess that we are in. The tar pit, as Fred Brooks calls it.
I think this is basically what Tunde Ogunnaike and Harmon Ray meant.  At least for us.  When they compared 'defined' and 'empirical'.  But, I will ask them.

***

Now, some questions that this very simple theory does not answer.
1. What if only 1 or 2 is 'unreliable'?  (And the other one is reliable.)
2. How unreliable do 1 or 2 need to be before one uses an empirical approach (as you call it)?  One imagines that, at very low levels of 'variation' in 1 and 2, ...that the 'defined' approach would still work or be better.
(As a practical matter for software development, I find that 1 and 2 are so 'out of control' that this question becomes moot.)
3. How do you know there is not another 'element'?
4. What if you can't adapt 'enough' (on 1 or 2)?
(Logically, in the simple case, one stops working or trying to produce anything, since 3 is highly likely to be wrong.  Unless one can live with totally random success.)