Retrospective: The driver for change.

Ensuring that the retrospective makes a real difference

This post is only partly in response to an observation made here that retrospectives were found to be generally useless. Having said that the practical tips in the above post are valuable.

I think the feeling (or even reality in some places) that retrospectives are a distraction is due to a few factors

1. The software development organisation, as a rule, doesn’t have a culture of self-improvement (neither developers nor managers)

2. Many teams struggle just to get to the sprint end date meeting the sprint goals and lose stamina by then (an important tip is covering this). Oh, yes I’ll not be in the least surprised if the cause of this is organisational pressure applied onto the development team. Similarly this pressure will lead to ineffective retrospectives.

3. Poor visibility, hence support for getting action items done (also mentioned and dissected here)

Retrospective is the driver for change generated by the team (and possibly anything that the SM points out). So after the lovely discussion how do we actually make the change? Hopefully the Scrum Master and the team have identified action items (not always easy, not always correctly done), but even so, some of them will be straightforward.

It is all a matter of perspective: Encourage all concerned to view the retrospective as a driver of change; and I think it’ll have a very different effect. At some point (ideally the beginning) of the project, or when the team is getting stuck, floundering etc., it has to be said “There are things that are not quite right around here. We need to change, bit by bit; Let’s use the retro”. Wonderbar! but how?

TIP: Suppose these action items were to be put into the Product backlog. This is an extension of suggestion here. A bit radical, eh what!?! as normally the product backlog is reserved to deliver something for the PO or external stakeholder.

caveats: Some action items need a lot of work/effort to be made and teams tend to drop these items from their to-dos, so these could go into the product backlog. Not all action items need go into the product backlog, as some can be done by the Scrum Master, some by the development manager (hopefully) and some are simple to do.

Coming back to the suggestion: What is the effect of putting retrospective items into the product-backlog? It certainly gets attention (visibility), and these items also get budgeted. More importantly a lot of careful thought based on experience WILL get acted upon. But there is something of a dilemma here, as then the PO gets to prioritize the action items! How would a PO react? What does the PO’s reaction tell us? Something of a litmus test, as to how well the whole team, SM and product owner understand the principles and imbibe the ethos of Scrum. This may often mean that the coach/Scrum Master will have some convincing/cajoling to do. However, while it may take time, eventually the team and organisation around it can see the effect of doing/ignoring retrospective action items (as there is greater visibility). Yes, it also gives the Scrum Master a great handle to encourage improvement via these action items.

WHEN NOT TO RETROSPECT: (ideally a separate post, but I’ve the first quality of a programmer)

I’ve as a coach often observed  teams which struggle to meet the sprint end deadline and led and facilitated their retrospectives. A team that takes it’s commitment seriously is at a disadvantage, as they have been working overtime for the last few days (nights!!?!). This usually means that their sprint review meeting, in such circumstances doesn’t go like a bomb, (don’t bother this link if you are from the UK/antipodes). By the time they are done with the review they are tired, somewhat dispirited and possibly uninterested. TIP: Give the team some recovery time. Don’t hurry into a retrospective under such a sepulchral atmosphere on the same day. Don’t hurry the retro to get on with the next sprint. You almost surely will be failing to learn. An ineffective hurried retro is time wasted, a longer one with sensible action points (which get acted upon) is valuable learning and a “velocity+morale” booster. If you must, hold a 30 min debrief. Especially, if the next day is a working day, people will be late, and the retrospective must be conducted well. So the main body of the retrospective should be around lunch the day  and concluded after lunch. This is to give team members time, discuss formally(during meeting) as well as informally (over lunch). In case the sprint ended on Friday evening, we can conduct the retrospective on Monday morning (with a long coffee break in between). But do make sure that at least three action items are identified and people commit to acting upon them (maybe put them into the product backlog). There must be visible progress over time, else each following retrospective would become more useless than the last (ironically, the reverse result for the retrospective itself, opposed it it’s intention for the project).

I’ve generally seen teams benefit from decently conducted retrospectives. As time passes and the retrospective action items become larger and more difficult this improvement slows down. This is where higher visibility (as suggested here) and perseverance will pay dividends. Good luck and God speed.

Advertisement

The VOC, but who is the customer?

A recent post encouraging proactive steps to gather the VOC leaves the issue of “who is the customer?” well alone, since a generic answer would be vague, it is heavily context sensitive. Many good ideas are presented here.

An intriguing point made here is that, these are solutions to a situation “…particularly where there is little time to do any early user research”. You must consider what is underlying such a train of thought carefully.   I hear this sentiment sometimes and it is almost surely leading to building of the wrong product! So lot of activity will ensue, but progress in terms of generating ROI may is certainly not ensured! In other words it is a case of leap before you look. This is not to say what’s written in the above mentioned post is wrong, or that it is recommending ignoring user research. It just is trying to work around a poor decision.

As usual since my blog is about Scrum,  here I’ll try to relate this situation with the product owner’s role. To cut a long story short, it’s the PO’s responsibility to bring the VOC into the project. Identifying and engaging with customer(s) or potential customers and the difficult balancing act that may be needed is in the PO’s court. Again it is difficult to provide general advice that is practical but it certainly helps to make as much activity around this as visible as possible. The transparency of such interactions will mean a very significant reduction in dysfunctional behaviour. So as far as the teams are concerned the PO is the customer, albeit,  one who follows the rules of Scrum.

Strictly left footed right wing striker

Background: The Pune vodQA 2012, where I was involved, but not entirely committed(A frivolous reference to Scrum lore, about commitment, chicken and pigs) in more ways than one, was interesting instructive and illuminating! Read about it here.

Mr Chaitanya Nadkarny, a senior member of Thoughtworks consulting gave an opening talk, covering among other things which factors are going to affect the lives and careers of testers. One point he made was with regard to how testers can be generalizing specialists, or specialising generalists. It took some time for the audience (at least I did take time) to get it’s collective head around this.  Mr Nadkarny’s point here was that

One could be a performance tester (specialist), but also interested in expanding capabilities (therefore generalising) to do general functional testing, manual etc.

OR

A tester (generalist) now wants to be more skilled and focus on an area of testing, for example usability (specializing).

MAIN POST:

Since I see the world through Scrum tinted glasses, I’ve been struck by how Nadkarny has put in a nutshell the possibilities of how people develop, become more valuable and competent. Many years ago a programmer was doing most if not all activities in relation to developing systems/applications, including testing. As the field developed people became more and more specialized. Eventually in the noughties, we have extremes of specialisation. Even being a tester isn’t enough of specialisation, one is now a manual tester or an automation tester, or a performance tester. The side effect of the industry moving in this direction is that people are squeezed into ever narrower roles, even though this isn’t required and actually creates inefficiencies. Finally we have the equivalent of a striker who can only play on the right wing,  kick with his left foot, but will not head the ball into the net, even if the ball is at eye level, straight ahead and the keeper is out of position!

This attitude is so entrenched  and pervasive in some places that even managers have often fallen into the trap. People are type-cast and given no opportunity to learn new things. The Scrum team, essentially being cross-functional, would struggle to succeed if a group of super-specialists have to work together, even if they are initially willing to collaborate. Why so?

Firstly: There is little understanding of what the other person does and needs.

Secondly: Load balancing at different points in a sprint is a nightmare, as work of a given type isn’t evenly loaded throughout the duration of the sprint.

Thirdly:  Majority of work in any organisation is done by generalists, because much of the work doesn’t need a too much of specialization. How often have you visited a general practitioner vs a specialist?

Lastly and very importantly: it encourages the continuation of waterfall habits.

Time to generalize. A cross functional team would need generalizing specialists.

On the other hand, once they get there, they might then need to be specialising generalists, as well… and so it goes.

What can be done when the PO isn’t empowered?

A fairly common situation in organizations is the presence of a PO without authority. It is common for a BA to play this role, and mainly due to the mistaken belief that functional knowledge is  the only criteria for a PO. Also the term ScrumMaster suggesting an all important person in a move to Scrum, contributes to this thinking. As in many things the best course of action is to preempt this situation by educating the rest of the organization (before) the start of Sprint about the roles of SM and PO. Once the realization set in, that the SM is not responsible for the delivery on a sprint to sprint basis, and in terms of delivering the product to the customers, people outside will turn to the PO. The PO will then slowly take on a stronger and more meaningful role, suffused with responsibility.  In my experience, coaching project teams and organizations, this take some time. The first level of understanding come about in 3 to 4 sprints, but deeper understanding typically takes a release or two (maybe 3 to 6  months). What one can do to make this more visible:

1. Re-iterate that Scrum is a radically different system of running software projects.

2. Encourage interaction between team and PO, as well as PO and external customers/client.

3. Explain to all concerned that while the team is largely responsible for “building the system right” every sprint, the PO is reponsible for making available “The right system”. While the team is collectively responsible every sprint, the release to the customers is the responsibility of the PO.

4. In addition for a release the PO is ultimately responsible for delivery of the “right system right”, in other words the appropriate system which is shippable (i.e of high quality)

Sprint Review undone: Part-I

I was recently present at an attempted Sprint Review Meeting for a team I was coaching. They were struggling to get finished by the time of the meeting, and by the appointed time could not get the functionality ready. The problem was that they had picked eight items, and could not even demonstrate one item, and this was their second sprint! They had earlier done a sprint zero, with many of it’s attendant problems which are described here. Now my overall  analysis of how this situation came to pass is at two levels:

1. Poor habits carried over from the waterfall days.

2. Lack of prioritization within the sprint.

Over and above this is the matter of a somewhat diluted DoD (a separate topic in itself) which is bad news being stored for the future. This team was working with a weak DoD, which was seemingly clear to them (For some one who reads tea leaves in his spare time, this is an ominous sign)

In this post I’ll explain the  first point. Within the sprint, each story was done strictly in a waterfall manner, including hand-offs. This not withstanding all the training the team and managers were given. Which meant, that the burndown chart, while warning about bad news, actually did not quite manage to reveal the really bad news. That nothing was actually ‘done’ ( this does not mean the team did not do anything). Suddenly towards the end of the sprint 4  user stories  were 90% done, and as the old saw goes, the remaining 10% takes the other 90% of time. The other 4 items were in even lower levels of completion. However what was the actual completion is moot.  Even more importantly, trying to find out the true completion percentage is missing the point. The team at the time of the sprint review, should try to keep the done/undone as binary as brutally as possible. Try to see it, standing in the PO’s shoes.

So what is to be done? Avoid waterfall. How? Especially given that the team is already facing too much change for comfort (One reason why many teams don’t see successful sprints early on).  Assuming they haven’t changed their technical practices yet (very common for teams starting off on Scrum), the first step is to try to break-down user stories to even smaller units of functionality, whose completion should be staggered across the sprint. Another approach that can be simultaneously used is  to write acceptance tests as the stories are picked up. This means that the team can test the stories (to some extent) at a very early stage, hence testing early and (hopefully) often.  A very good idea.

The above ideas are a reason why teams aiming to succeed have to change their ways of working and learn to breakdown stories and change the approach to testing among other things.

In the next post I’ll explain the other practical step to take (if you haven’t guessed already).

Handling changes in Scrum

Srinivas is the co-author of “Essentials of Scrum practice“- a mini-book
currently in a draft stage. This is an extract from the book.

 

All s/w development projects have greater or fewer changes to be incorporated after the start of development. So how are changes handled in Scrum? and more importantly why are they handled in the manner described/recommended by Scrum?

Basically in Scrum the top few items of a product backlog (the highest priority ones) are chosen for any given sprint. The remaining items, can be modified by the product owner as he/she pleases at any time. As an aside, there has been some recent talk about the backlog not being prioritised, but ordered. I’m less than impressed by that explication.

Since Scrum is supposed to take care of changing functional/non-functional needs over time, the germane question is “How do we handle changes within a sprint/iteration?” in other words “how do we handle changes to the sprint goal?”. This is a recurring question, and the answer lies in correctly understanding the intent of Scrum.  Scrum’s intent is to provide a means, an island of stability where a complex project can be run smoothly, even though the environment around is in flux. This island of stability  is ensured by the “NO CHANGE RULE”. Which means no change in duration, goal or team for any given sprint.

See the following post for the reasoning on keeping the team fixed. Here I’ll essentially cover how and why of keeping the goal fixed.

There are two cases which cover changes to sprint goal

1. Urgent bug-fixes which may turn up at any time, hence cannot be foreseen at the time of the Sprint Planning Meeting (SPM)

2. Change requests to the selected product backlog items

These two cases are to be handled in radically different manner. For the first case the team can inspect and adapt to choose a means of dealing with bug fixes, by either keeping a percentage of available time (say 10 to 15%)  aside as contingency or by rotation having one or two team members on stand-by to handle the bug-fixes as and when they turn up. In other words this is being prepared for the unpredictable. The second case is severely discouraged and is  handled by not taking it up in the current sprint! This may seem extreme, but there is sound reasoning underlying this approach.

First and foremost, if change requests are allowed at any time within a sprint, there is no line drawn as to how much of change is allowed. Given human nature this will result in any amount of change being allowed any number of times with any sprint. What is the implication of this on the usefulness of the SPM? We are very possibly on a slippery slope by allowing a change in sprint goal, mid-sprint. Essentially, by having this rule, we are creating an island of stability.

Secondly, the reason we have sprints with items being pulled into any given sprint, is to accommodate change. This means at a project level change to functionality is handled in an orderly fashion as well as a welcoming manner.

Thirdly, and this will be difficult to stomach for some stakeholders,  a significant amount of change is due to lazy thinking early on. In other words quite a few of the change requests are a result of not thinking through what is needed. While it is recognized that the PO/customer cannot always know everything that is required in detail, up-front, what this rule does is encourage reasonably detailed analysis of the items which are at the top of the backlog. When the PO is confronted, with a situation where he/she/it (How PC am I becoming!?!) has thought of a change and has to wait till the next sprint, this is a stimulus for more careful thinking before and during the following SPMs.

The encompassing principle above is that Scrum is a system that encourages corrective behaviour from the stakeholders, while maintaning a balance between accommodating change and reducing avoidable waste due to lazy change.

Two further points:

1. Scrum does NOT need (in fact discourages)  to have the functionality described in every detail at the starting of the sprint! Then, how in the blue blazes are we supposed to keep change at bay, as clarifications are made during the sprint!?! There is another simple rule in play “If in doubt, it is the team that decides whether it is a change or clarification”. In case of clarification the team simply implements, in case of change, it gets put away for following sprints.

2. What if there is an utterly, unavoidably, compellingly, the-sky-is-falling-on-our-collective-heads, URGENT change of sprint goal, in the middle of the sprint!?! Either a change to an item picked in the current sprint, or an altogether new item added.

Well, indeed if it is THAT important, follow Scrum by the book!

i. Terminate the sprint

ii. Inform all stakeholders

iii. Immediately start a new SPM for a sprint, of what-ever length, that makes sense for the explosive circumstance. Does one need to do a full SPM (of 4 hrs?), not really. Exceptional circumstances can mean that just the new/changed item is to be dealt with in the planning.

Any loss of time/money/delay due to this decision is made in full knowledge by the PO.

Again note, that this is a highly disruptive, but necessary change, and is widely visible. Not to be undertaken lightly. After all if there is a serious earthquake, the sprint goal will change.

Commitment Under Pressure

Srinivas is the co-author of “Essentials of Scrum practice“- a mini-book
currently in a draft stage. This is an extract from the book.

Pressure pushing down on me
Pressing down on you no man ask for
Under pressure that burns a building down
Splits a family in two
Puts people on streets
It's the terror of knowing
What this world is about
Watching some good friends
Screaming let me out

The hidden hobbler of Scrum teams: Pressure

One common problem Scrum teams face is in the difficulty of meeting their sprint goals, followed by increasing pressure, which seems never-ending. Such teams don’t see sustained significant improvement and can quickly fall into a death march mode, or a jaded plodding with longer hours but no end in sight. Such situations are due to their choosing sprint goals (targets) under duress in some form. The ill effects of such commitments are not easy to understand or link to the cause. The inviolate Scrum principle in this regard is that the team chooses the extent of the backlog which it’ll deliver. Team requires to have a fair shot at making an independent, free commitment for their sprint, after all they are the ones who do the work. Mainly managements are worried that the teams will under commit. Hence they may treat the whole affair as a negotiation game.  As entertaining explication of such games is provided by Rob Thomsett. Software development is very commonly done under pressure, and this makes it seem as though this is very normal, and possibly the only way to work. A good anti-dote to such thinking can be found within Tom DeMarco’s entertaining “Slack”.

What do we do when the pressure to finish is so much, that there is no choice left to the team.
Don’t buckle under pressure, and stick to the Scrum principle (i.e make an independent and genuine commitment), however this is much easier said than done. Many projects have deadlines in some form, based on a marketing wish or executive fiat. Since organisations need to make plans, what upper management wishes for, may be necessary. Trouble is in the making, when this wish is widely out of line with reality. What organisations do poorly, is handle the difference between an original high level plan (based on executive wish or even a high level estimate) and the reality as the devil in the details appear. The Scrum approach is to do planning in small bits (2 weeks) and improve continuously by ‘inspect and adapt’. In our experience, far too many projects and managers are worried about the success of any given sprint and don’t give a chance for the team to learn and improve, hence endangering a number of future sprints and ultimately the entire project. A difference is made if a couple of steps were to be taken before the sprint starts:

1. Get an agreement from management that the team be free to choose the extent of work they commit to; offer to be transparent regarding why such a choice is made.
Essentially this means that the details of the sprint planning meeting is made available to the larger organisation. Management also needs to support the Scrum Master in his/her task of protecting the team from any external pressure. Now the Scrum Masters job is easier, if he/she also provides transparency into the Sprint Planning Meeting, by highlighting significant events in a MOM, as well as, circulating the sprint backlog/plan itself after the meeting. Normally the sprint backlog, with estimated hours for each task totaled and compared with available time, will give a realistic view of what is within the realm of possibility for the team to take up.

2. Explain the distinction between a real commitment that the team makes, believes in, as well as is serious about, as opposed to a perfunctory commitment made under duress.
Pressure to finish fast exists in most organisations. However, better the protection a team can be provided, to demonstrate to the team that their honest views are truly valued, better the commitment from the team and the teams attempt to meet this commitment. Fair and transparent planning will mean that team will not knowingly under-commit. Once a true commitment is in place the team, they should be supported by the Scrum Master and the management will do all in their power to meet the sprint goals. An unrealistic deadline imposed may make the team work harder, but they are unlikely to learn and also this will not be sustainable. In fact there is a very good chance that team members will take a fair stab at pretending to work hard. What the Scrum Master and the management can do is replace pressure with motivation. Any pressure must only be peer pressure, generated naturally, not imposed from higher up.

An intangible, but important effect of an open transparent sprint commitment made freely, is that the team sees it’s opinion valued even by management. They are being treated as professionals. I’ve yet to hear of any one, however powerful, saying to a surgeon “You have to do my gall bladder operation in 45 mins, now! and not take 3 hours as you say it’ll take”.

Further more, if after all this discussion we still have a situation where pressure cannot be held off, either due to the unwillingness of people to be open, or the unwillingness of management (briefly glossing over the fact that management also consists of people, of  a sort) to accept bad news, then the Scrum master must present the actual versus the hoped for amount of functionality delivered after the sprint in a neutral communication as a means for the organisation to learn. Many organisations take time to learn so this can be a long drawn out process. Remember Scrum Masters can do this easily, if they are not directly responsible for the success of any and every given sprint, which is typically the case with a project lead. It is well worth noting that a ScrumMaster is not a new glorified label project lead, but an enabler and protector of a team, which alone is responsible for delivery.

Finally, a thought on how breaking the Scrum principle leads to long term undesirable side effects. When the team members realise, that management just wants unrealistic results and is unwilling to listen and act upon information provided in good faith by the team, their motivation drops. Also they have no further incentive to provide bad news ahead of time, and the slide to micro-management is fast. This also means they stop caring for the project as professional pride is buried. Therefore they will go through the motions; not the most inspiring and driven crew, who will be apathetic at the best of times or even hostile.

Pressure pushing down on me
Pressing down on you no man ask for
Under pressure – that burns a building
down
Splits a family in two
Puts people on streets
It’s the terror of knowing
What this world is about
Watching some good friends
Screaming let me out

Manager! Leave the team alone!

We don’t need no disruption
We don’t need no time control
No breathing down the neck in the server room
Manager, leave the team alone.

What do we do if a senior manager is asking for a ‘resource’ from within the team, in the middle of a sprint, to deal with some urgent matter?

As with a lot of things, this is better preempted, by informing as much of the surrounding organisation as possible before the start of a Scrum project, that we have to keep a team constant for at least the duration of a sprint. In other words the organisation should understand that the ‘NO CHANGE RULE’ is critical and central to the Scrum way of working.

However, senior managers may not pay sufficient attention to this or feel that whatever is occupying their minds at the moment is of utmost importance, and they can pull ‘resources’ (actually people) out without too much consideration. It is the Scrum Master’s job to protect the team, and as such does throw the Scrum Masters into a very difficult situation. The organisation culture will be a dominating factor in how people (SM included) behave. In many organisations the hierarchical structure combined usually with a command and control culture means that Scrum Masters are either powerless in face of such disruption or get into a downward spiral of de-motivation. This effect should be highlighted after the sprint end, to the management including any effects on how the team struggled to meet the sprint goal.

The Scrum Master in such circumstances should talk to the team and PO, and if applicable to the PO’s manager (who could be the senior manager in question) about the side-effects of pulling someone out mid-sprint.

Organisations where everyone is well-educated about Scrum, will seldom face such a situation without being able to make corrections and recover fast. This situation is particularly difficult in organisations which are largely ignorant of Scrum and the attendant principles. So it is important to respond to this situation without using jargon in such settings.

Speak in terms of disruption, de-motivation and relative importance of the sprint goal versus the work the pulled out ‘resource’ will do, versus the importance of adhering to the rules of the game. Such a conversation should also guide the persons making this decision to think in terms of ramifications of such requests and induce an inclination for senior managers to talk to the PO and SM before making any such requests/orders to pull out people, in the future.

All this engenders a culture where disruptions are kept to a minimum and only for good, genuine reasons. A relevant point here is the case when in response to a management edict coming from much higher-up, and in essentially a knee-jerk reaction, someone is pulled out even before it is really clear what exactly needs to be done. In other words the person pulled out cannot really begin work, since certain clarifications are still being sought. In such cases the useful practical steps to take consist of the Scrum Master taking a close look, spending sometime to discuss this with relevant team member, so they are prepared, wait till the end of the Sprint, so there is minimal disruption. Then after the sprint ends, the team can be reset, with a freshly updated backlog, which may include this ‘urgent’ piece of work; alternatively the person in question will not be part of the team for the new sprint, and works independently.