Iterative development and branching
Sandy Walsh, a colleague of mine at Rackspace who also works on the OpenStack cloud development project, has written a blog post on how Iterations and Time-boxing are (Mostly) Useless. It’s a well-written, thoughtful post, and I appreciate his perceptions, but some of them are (to my point of view) strikingly incorrect and unsupportable.
His general thesis can be summed up by these sentences:
Personally, I think Distributed Revision Control Systems (DRCS) such as git, bazaar or mercurial are the most significant change in software engineering within that last 10 years. Not because having a non-central repository is so revolutionary, but because branching and merging has become such a low-cost operation that the speed of software development has increased dramatically. Development shops that utilize these tools can see some big improvements in how they deal with the “Rough Edge” at the end of the sprint.
A major point of contention with Sandy is that neither the Agile nor the Scrum methodologies specify a branching or merge process, and thus it’s not the revision control system that dictates whether or not the methodology works. In addition, some of Sandy’s perceptions of the Scrum process appear to be slanted by not having had the experience of working in a smoothly-running, experienced Scrum environment.
There is nothing in Scrum or agile development that requires the use of any specific revision control system, branching, or merge process. In fact, many development teams use Scrum in conjunction with continuous integration, in which there is no branching or merging. When I worked with the Yahoo! media group, all developers worked in the trunk, and branches were only created to save the state of specific releases. To do this requires a definite commitment from the developers; it requires having excellent unit test cases and software designs that permit features to exist in the code but to be “switched off” until they are complete. Continuous Integration is a methodology that has great rewards for the teams that make the commitment to it, but it has nothing to do with Scrum.
Agile (and it’s angry child, Scrum) do not lay claim to be the end point or most perfect method for developing software. (Of course, there may be proponents out there who say this, but I’m speaking for the community in general.) What the methodologies offer, however, is a better way of responding to changes in requirements than more fixed, longer-term methodologies such as waterfall.
Sandy’s post makes some very good points, such as the fact that there is often some “dead time” at the end of an iteration (a sprint cycle in Scrum). The fact remains, however, that such dead time is a very small prices to pay for the predictability of delivery. Moreover, nothing precludes a developer from getting a head start on the following iteration’s work, though doing so might conflict with changes that come up in the next iteration’s planning session, and thus that work might have to be discarded.
Most businesses value predictability over sheer speed of delivery; it’s very rare that “as fast as possible” is a true business requirement. If you’re developing a new product, for example, shaving a day or two off of each iteration might (and it’s a definite maybe) permit the code to be completed earlier, but there are external considerations (marketing collateral, sales and support training, and documentation, for example) that may not be able to adapt as rapidly, and thus the “wasted” days are simply piled together at the end of development rather than spread out in the middle.
Sandy has some rather specific complaints about Scrum:
If you know me, you know I’m not a big fan of Scrum. I don’t like the hand-waving concept of Velocity and I don’t like the fact that it places a priority on process over code.
There are two responses here: first, to the “hand-waving” concept of velocity, and second to the “priority on process over code.” The second one is easier to answer, and can be found on the Agile Manifesto:
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: Individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, [and] responding to change over following a plan.
If Sandy has found Scrum adherents that are placing a priority on process over code, then I would submit that they are most decidedly not following the Scrum methodology.
About the the “hand-waving” concept of velocity, Sandy states that “story points” need to be equated to time. I believe that that’s a very dangerous path to take. One of the reasons for “story points” is that, as Sandy says, “estimation is hard.” Rather than take a “Wild Assed Guess (WAG)” at when something will be completed, story points provide a mechanism for, up front, before you get into the weeds of developing something, to rationally estimate the overall complexity of a feature/requirement/story (like Sandy, I sometimes hate the terminology nuances). The fact is, it’s not (or should not be) a “wild assed guess”; it should be a reasonably well-educated guess, based on the developer’s (and product manager’s) past experiences and expectations.
Velocity, in its best case, simply measures how big is the “stuff” that a team thinks it can complete in a specified period (timebox). Why a specified period? Because without it, there’s no way to relatively measure the “size” of a feature without getting down to the development tasks in hours or minutes, something we want to avoid. In addition, timeboxing forces the development team to reach a stable state regularly, allowing the customers an opportunity to adjust and modify the requirements, and the developers to assess and change their work methods (and hopefully improve them). Sandy’s strategy of “overlapping” development is time-efficient, but could easily result in either features that need to be discarded (wasted work) or features that should be discarded, but aren’t (because so much effort has been put into them already).
There’s no methodology that’s perfect; I have seen year-long software development project using Yourdon’s strict waterfall method delivered on time and under budget, and I’ve seen development teams double their efficiency by using Scrum. One of the keys is to have the developers themselves be thinking about the process and ways to improve it, and Sandy, via his post, is a great example of that.
Well Glen, thanks for taking the time to consider my post. I think there are some grave misunderstandings in your rebuttal and some glaring omissions in what you state as fact. Let’s tackle your gripes one-by-one:
1. ” some of Sandy’s perceptions of the Scrum process appear to be slanted by not having had the experience of working in a smoothly-running, experienced Scrum environment.” … I’ve been using Agile/XP/Scrum techniques consistently since 2001. Just because I’ve used them (and successfully) doesn’t mean I have to like them. If a better way comes along I’ll adopt it. I believe my article expounds a better way. If you look in some of previous blog articles (specifically http://www.sandywalsh.com/2010/03/tip-for-closed-source-software-houses.html) you’ll see that I think the major advances in Agile development came from XP, which is engineering focused, and not Scrum, which is process focused.
I find it quite amusing that Agile pundits are quick to cite the tired horse of “People over Process” but as soon as someone indicates that the Emperor Has No Clothes they jump all over you for not following the process. (See: the ScrumBut nonsense)
2. Nowhere do I talk about Agile recommending the RCS system. What I said was because of new RCS systems, many of the Agile methodologies are moot. I think you completely missed the gist of the article there.
If Yahoo worked in trunk all the time and only branched when it was urgent, that only highlights my argument about old RCS systems. Working in trunk is dangerous and should be avoided. Again, I refer you to the NVIE branching model.
3. “The fact remains, however, that such dead time is a very small prices to pay for the predictability of delivery. Moreover, nothing precludes a developer from getting a head start on the following iteration’s work, though doing so might conflict with changes that come up in the next iteration’s planning session, and thus that work might have to be discarded.”
Why?
The customer still controls the backlog. Developers don’t run down rabbit holes simply because there are no timeboxes. Nothing changes, except the “small price to pay” goes away. What this asks for is *more* customer feedback, more often. Isn’t that what Agile is all about?
“Sandy’s strategy of “overlapping” development is time-efficient, but could easily result in either features that need to be discarded (wasted work) or features that should be discarded, but aren’t (because so much effort has been put into them already).”
How?
What are are doing any different from Scrum other than ditching the timeboxes? Now, *that’s* a hand-waving argument
4. Story points do *nothing* for making better estimates. Of course we assume developers are giving their most professional estimate possible … or, as a colleague called it “a SWAG (Scientific Wild Assed Guess)”. The simple fact is, the further out the estimate, the less accurate it is. No system can foretell the future and requirements change. Story Points are no different than saying “I have 40% effective time in the day for coding.” … I challenge you to prove otherwise.
All the same burn up/down charts are possible without timeboxes and using the, more common, unit of measure: Time. And, if you don’t think Story Points get converted back to time by the managers, you’re foolin’ yourself. Time and meeting Requirements are the only thing that matters. Customers don’t buy Story Points, they buy met requirements delivered on time.
My post was not based solely on the view of a developer, but rather based on the insights of a development manager back in the trenches. Lots have changed in the last few years … lots of really good things. Scrum is past its prime and being superceded by Lean. XP (rightfully) lives on since its focus is on the code, not the process. And, as stated, the light branching capabilities of modern RCS systems are the mechanisms permitting this change to occur, IMHO.
Cheers and thanks again for the valued feedback,
-Sandy
Thanks for the response, Sandy. I admit I made some assumptions in my haste. I’m not really one of the Scrum devotees, but I have seem some remarkable turnarounds in both delivery times and quality from teams that switched to using a rigorous Scrum methodology.
Perhaps I’m not conversant enough with git or bazaar to understand the benefits, but merging is merging, whether distributed or local. Merging introduces risk in the process, whether it’s into the trunk or from the trunk into a branch. The problem with merging from trunk into the branch is that there may be other teams working on the same code base that are also merging into their branches; everyone is perfectly happy until each of those branches go to be merged back into the trunk, at which point collisions wreak havoc with productivity and schedules. Merging introduces uncertainty, unless you are rigorous in merging each branch with every other branch, which is what happens when you continuously merge into trunk.
And, of course you can use tools: burn up, burn down, etc., to measure progress in any methodology you like. The problem lies in reducing uncertainty for the overall project. I’m typically a pragmatist; if your team can reliably predict when you can deliver something, and there’s a sufficient level of quality, then, hey, use it. There’s nothing magical about Scrum, nor is it the solution to all problems. My point is simply that many of the problems you’ve mentioned have little to do with the methodology, at least as I understand it.
Thanks again Glen. I appreciate your feedback and completely understand your perspective.
I was thinking about the merging concern last night and I think I see where the disconnect is:
Merging is a necessary evil. Beyond actual development there are two issues with software development: integration and testing. The XP practice of Continuous Integration goes a long way to making that bearable. Scrum attempts to solve the problem with timeboxes/iterations (you have to check in to meet your deliverable, therefore you have to integrate) … but that’s where the flaw is. CI is exactly that … continuous. I know most devs on our team pull from trunk first thing in the morning in order to keep the deltas small. Trunk is, or should be, always stable. Code reviews and a rigorous “merge-to-trunk” process are imperative here. By pulling from trunk, our branch is kept in sync and our later push back into trunk should be a NOP.
Forcing devs to push to trunk daily has a tendency to relax the merge review process and cause more troubles simply because of the daily deluge of review requests. Instead, we can push to our own branches and only push to trunk when the tests all pass and we are Done.
Also, new DRCS systems treat branches and merges very different from the old systems at a technical level. Changesets are atomic and the basis for the technology. With older systems the code is the basis for comparison and changesets are inferred from the code. Those are very different mentalities. With git/bzr/et al, only the deltas are sent to the server in one blob. Branching is simply creating a new (empty) changeset.
By focusing on the changeset we know all that is required to move from version to version so merging is faster. We don’t have to scan the code over and over again to see what changed. This also means that pulling from trunk is a speedy operation and can be done at low cost to the devs … many times per day if needed.
Yes, there is still a possibility of the monster merge, but *nothing* can prevent that other than keeping developers from working on shared code.
My point is, there *is* a technical basis to my argument. Not all merges are created equal.
And finally, regarding Scrum, you are correct, there is nothing magical about it. But the dogma and mysticism around the terminology and “the dance” seem to infer there is. It was a great system for its time and certainly offered great advancements to teams over waterfall. But, for some of us, it’s outlived its usefulness.
Look forward to continuing the conversation with you over a beer next time! Thanks again for the feedback.
Huh, WordPress let\’s you reply to comments via email. I didn\’t know that.
Anyway, I think some of the \”enthusiasm\” over Scrum comes from trying something new. We need to get people thinking in the correct terms, so management is somewhat rigid about it before we start tinkering with it. I agree that we can always look for improvement, and that the right tools can always help (not necessarily solve) some of the issues.