This is the transcript of an interview that Gil Broza conducted in 2014 for a virtual training on Agile journeys. Though several years have passed, the powerful insights shared here are as applicable today as they were then.
(Prefer to listen? Here is the recording.)
Gil: The whole side of technical execution or technical discipline doesn’t get nearly enough attention — in companies, at conferences, even from a training perspective. Our next guest, Arlo Belshee, will help us redress that. He is truly a trailblazer, working tirelessly to drive mediocrity out of the development world. Hi, Arlo.
Arlo: Hello.
Gil: How was that introduction, huh, “driving mediocrity out?”
Arlo: Yeah, not bad.
Gil: The title of our conversation is “Technical Excellence: Changing the Rules of the Development Game.” Plain and simple, the standards for professional software development have changed. If you like Agile, you must accept some of its principles: being right trumps being fast, change is welcome, and teams share responsibility for their product. When you accept these principles, you must change the rules of the development game. Too few organizations know how to do this or expect technical excellence from their teams; they don’t know how to make it happen. I’m one of a handful of trainers worldwide who teach Agile engineering in public workshops, and let me tell you there is a long way to go for the uptake reaches anything like the level of appetite for Scrum courses.
Arlo: Yeah.
Gil: Yeah. Any of our listeners, if you’ve been hanging around the Agile Conference in the last few years you probably couldn’t miss Arlo. I first met him at Agile 2005 when I and only 14 other people sat in a room where this really young guy (at the time) delivered a deep and revolutionary experience report on what he called “Promiscuous Pairing,” which became A Thing in the industry. A few years later he was one of the forces behind additional waste removal from popular Agile methods with his approach to planning, which he called Naked Planning. His recent focus, if I can even keep up with him, is on driving engineering excellence and maturity. So not only does he share some of my views on Agile engineering, he’s now doing this on a much grander scale – with 10,000 engineers at Microsoft. Did I miss anything our listeners should know about?
Arlo: Oh, there’s a bunch of other stuff along the way but those are a good set of high points. Throughout the whole thing, it’s fundamentally, I drive courage, I drive technical excellence, and I show how when you achieve true excellence you can get rid of all the transaction risks, and that enables making much more intentional risk-taking, so that you can actually run risks that are of value to your business rather than just incidental risks along the way.
Gil: Yes, now I remember hearing this in your presentation with Jim Shore at Agile 2013. We’ll hear about this a bit more because it’s, I think, a couple abstraction levels away from what our listeners were expecting right now.
Let’s start from the basics, let’s talk about technical excellence. So what is it, or how is even different from the technical excellence we were thinking about a decade or two ago?
Arlo: So when I think of it, it comes down to one fundamental question, which is, “Where do bugs come from?” In the old way of thinking, traditional approaches, we think that bugs are a natural side effect of coding. We’re going to the job as fundamentally we’re going to create some code and some bugs, and we’re going to find the bugs, and we’re going to fix some of them, and we’re going to ship the rest. In the new modern way of thinking of it, bugs come from mistakes; they come from developers. Developers create both code and bugs. But developers control the rate at which they create each of those two things independently. And so technical excellence is understanding that and making the choice, the very active of choice, to simply write code and not bugs.
And then once you do that a whole lot of other things change. Because now you’re operating in a different ecosystem. You don’t need to spend all the time looking for bugs. Your bug database is almost always empty. A typical bug database should be size zero. It will get to one bug maybe one day every three weeks, otherwise it’s at size zero. So 90%, 95% of the time it’s zero. And that means that you’ve just got all this freedom to go play with whatever it is that you really need to spend some time and think on.
Gil: Okay. I’m sure we have a few listeners right now who are thinking, “Well, not where I work.” But we’re going to touch on that later on.
Arlo: Absolutely [laughs].
Gil: Yes, you don’t get to zero bugs that quickly.
Arlo: It takes about two years.
Gil: Okay. So the whole technical excellence thing, is it a personal thing, a team thing, or just a matter of culture?
Arlo: Yes.
Gil: That was an “or” question [laughs].
Arlo: Yes it was [laughs].
Gil: Okay.
Arlo: It isn’t a personal thing because at the level of quality and capability that we’re talking here, you need to be learning at a faster rate than individuals can learn, and you need to be maintaining a higher level of discipline that individuals can maintain. However, teams can learn at that rate, and can maintain a discipline. Pair programming, mob programming, these are the practices that are there fundamentally to give the team the ability to learn fast enough and to maintain a high enough level of discipline.
So, at the first level it’s a team thing because to maintain that level of excellence and learning rate and low defect rate, you need the whole team supporting and working as a team, executing as a team. Not just figuring out what we do as a team and then chunking out tasks that we’ll individually execute, but actually collaborating on the execution of the work.
However, teams don’t just spontaneously come to be this way. This is a thing that you learn and grow over time, you don’t just a transition to not writing bugs the way that you can transition to writing your work in stories. Switching to stories, meh, you have a thing to learn, it takes a few hours, you can start doing it, it takes a little while to perfect. Getting rid of writing bugs, you have a few concepts to learn and then a long time of perfecting and finding new ways to do it and so on. And that’s where you need a culture. So the culture becomes the transmission vehicle within a team and between teams of this ability to really excel technically.
Gil: So we really do hope the teams actually stay together over time, right?
Arlo: It helps a heck of a lot!
Now that said, I have seen some good team rotational models that also work well, Menlo Innovations does, Yammer does, several. But when you’re doing that, that works by having a larger workgroup that subsets and does rotations for a period of time, but all of the learning and knowledge is accruing to the larger workgroup. You end up sacrificing some things, gaining some things, it’s an alternative approach.
Gil: Right, yes. Sort of like a superorganism in a way, right?
Arlo: Yes. Now there are a lot of ways to do a rotational model that don’t work, and most people that do a rotational model don’t have the right bounded context outside, they don’t have the right information vehicles for exchanging knowledge between teams, they don’t do the rotations in the right way. You screw up any of those and then you end up without teams at all and you fail.
Gil: I know you like to use the word fluency. Can you tell us what you mean by that?
Arlo: A lot of people when they talked about a still they talk about, “Can I do X?” “Can I refactor this code to do an Extract Method?” is a very specific example, “or can’t I?” That is talking about proficiencies and capabilities, “I know how to do Extract Method, I don’t know how to do Move Method,” say, whatever. The problem with that is what actually correlates with success is not just can I do a thing, but can I do a thing without thinking. Because bugs, again it comes back to bugs, bugs come from mistakes, mistakes come from cognitive load. The more things I have to think about, the higher the cognitive load of the task, and the more likely I am to make mistakes, it will exceed my capacity and I will fail. So if I have to think about doing an Extract Method, then I’m thinking about Extract Method and not about my code. When I achieve true fluency at a proficiency, I can do without thinking. It’s the difference between I extract method and my fingers automatically hit Ctrl+R+M and I type the name and the whole time all I’m thinking about is, “I’d like this code to go over there. Yeah, like that.”
Gil: Right. Yes, I know from personal experience what this feels like, yes.
Arlo: Yes, and it’s just like fluency with a language, fluency with playing an instrument, it’s that same thing of I’m trying to communicate now, I’m not thinking about the sentence, I’m thinking about the idea. Same thing when coding.
Gil: Okay, so let’s circle back to technical excellence and what it means in business terms. So people understand that it’s not just a dev thing.
Arlo: If we look at Lean Startup or other 3-star Agile methods, they talk about hypothesis-driven development and that fundamentally our job is to learn about the environment, learn about our customers, etc. The way you do that is by running experiments. The way you run experiments is you figure out a hypothesis and you take a measure. So great, that’s the way that we should be doing a lot of modern strategic development these days is to actually run some experiments, take some measures, and keep going. There’s just one problem with that and that is in order to run an experiment and take a measure, you need two things. You need a quick cycle time, so that you can go from coming up with a hypothesis to data as quickly as possible so that you can run more of those experiments. And you need a non-noisy system; in order to take a measurement and have it be valid the system has to be in some sense stable. If there’s a cosmetic bug on one path, if you’re doing an A/B test of two different login flows, and one has a cosmetic bug and the other doesn’t, people will subconsciously get triggered by that cosmetic bug and prefer the other approach. And that will mess up your data more than whatever you are trying to test and you won’t even notice it. So in order to be able to actually have reasonable good experiments with good data you need a clean signal. And so technical excellence gives you the ability to go in very fast cycle times, and every single time know for sure that you are measuring what you think you’re measuring.
Gil: So what I’m hearing you say is when we talk about technical excellence and the short feedback loops so our experiments are relevant and valid, it’s not that we’re producing crap and expecting people to give feedback – because crap allows us to go quickly – no, we produce good stuff which is partial, not necessarily the complete thing, it’s just the thing we need for the experiment but it’s well built.
Arlo: Correct, yes. When you think about the old thought of the iron triangle –- scope, quality, and time, go ahead and take your balance and whatever. Teams that get really good at technical excellence find that the higher the quality the faster the delivery. And that’s why they end up doing zero bugs. It’s because if they added bugs, they would slow down.
Gil: Yes.
Arlo: So it’s cheaper to just not have any and to maintain that state. And so when you’re working in that state, now you look at the iron triangle and, well, then of course you just peg quality to the max because the higher the quality, the more money you get to spend on the other two. So then you’re just trading off scope and time. And so now you thin-slice the scope as much as you can to decrease the time. I minimize the scope such that I can get my next answer and I work in chunks of scope always in top quality.
Gil: Okay. So, we’re naturally focusing on defect reduction or regression testing and all that here, right? We’re talking about working with a different perspective, everything we do is now different. What is it exactly and how is it actually specific to Agile – or would this have occurred anyway?
Arlo: So there’s a couple of different parts here. One, if we go back and look at the original lightweight methods before they got re-coined into being called agile, XP was a big dominant player. The reason it was a big dominant player was because it did the technical stuff well and so you could start seeing the other things. In the bad old days, bugs were just assumed to be normal. And when you’re in that state and that mindset, you can’t iterate quickly. And because you can’t iterate quickly there’s no point in slicing in terms of customer value and making stories, you can’t respond to change, all these other things just aren’t possible.
A lot of those ideas became possible once the technical practices started being practiced. And then they actually spread much more quickly than the technical practices did, and now we have this weird inversion where a lot of people are trying to do the things that depend on the tech without doing the tech.
Gil: And they end up in a world of trouble.
Arlo: In a big world of trouble, it’s a very predictable trouble. I can chart on the wall what the releases look like and give me two or three data points and I can tell you the month in which they will collapse on technical debt and no longer be able to release at their cadence, it’s so consistent.
Gil: I know exactly what you mean!
Arlo: Yeah. But yeah, getting back to your question, the culture and mindset that’s specific to Agile is if you’re in software development, then the way that we were doing software development was a very risky, it was very unsafe, full hazards, Anzeneering sort of approach. And we needed to fix that and that’s what the technical excellence and technical practices do. If you’re thinking about Agile or Lean in manufacturing, they’ve already solved many of those problems. An assembly line does a pretty darn good job of producing things as expected. So they didn’t need to go add a whole bunch of technical practices, they’d already adopted them. And so this technical excellence fundamentally permeates Agile in software development. And given that most current Agile thinking came from software development — except for the half that came from manufacturing and Lean before they merged — it comes from people switching to working as teams and working in this new “clean room” way so they can start seeing the benefits and additional stuff on top of it.
Gil: Okay, so we talked about getting in trouble if you don’t do the tech part right. Okay, let’s say I’m a manager, my team and I, we take a shine to Agile and we want to start using it. What should we pay attention to from the technical standpoint?
Arlo: Pair programming and refactoring. And actually there are a whole bunch of other things that you want to do first, want to do as well.
One of the challenges with adopting Agile is that when you look at the lists of practices, like XP’s list of practices, those things that are stated there – refactoring, TDD, pair programming, etc. — those are not a single practice. It’s actually a family of practices and when I look at a specific team, over time they learn to do that practice differently, and they keep always calling it the same thing. So I’ll talk with a team that’s doing TDD and they’re new to it at one year’s Agile conference, and I’ll ask them what they mean by it, and it’s the devs own correctness, they write all the tests, and they’re starting to experiment with test-first, they’re doing a little bit about that. Okay? And then I’ll talk to them six months later and it’s that now they’ve built in refactoring and they’ve built up the ability to do local refactorings and are starting to create units, and so they’ve got unit testing now that they’ve got units, they didn’t before, they still call it TDD. And then later they speed up the inter-dev loop and do red-green-factor and they still call it TDD. And then later they, and so on and so on.
So what you really want to start with is not, “Let’s introduce this one practice, and this is the key thing to focus on.” It’s, “Let’s introduce the mutually reinforcing starter kit set.” For validating my work, devs run automated tests. I don’t care what the test is, write at first, write it last, write integration tests, make it as ugly as you want, I don’t care: devs write tests. And devs can perform local refactorings using a mechanized tool. Rather than doing local, small, within-the-method rewriting and recoding, they understand at least the core five refactorings: Extract Method, Introduce Parameter, Introduce Variable, Introduce Field, and Rename. And they can use those to start reading code by refactoring it to make it readable.
In order to start transmitting the knowledge around the team of the things they already know, you need to have pair programming for the purpose of learning. So pairing for production sake, you’ll need to be pairing four or more hours a day and you’re pairing on all production work. Pairing for the purpose of learning, you might still be coding individually but instead of doing the code review you bring the nearly complete code to a pair session at the end of the day and you pair for one or two hours every day. It’s a lighter weight version of the thing that helps you get started and support what you need. Likewise, retrospectives, you need the core lightweight version of that. The action oriented, it’s about defining one experiment are going to do between each period, and the focus is not on the retro, it’s on actually executing that experiment. In order for that to work we need cross-functional teams, single assignment, reasonably stable, and in particular, every human is in exactly one team.
Once we have those things… so there’s a whole bunch of different things that you need to put into place, but all of them are small micro steps. And then you’ll build on those micro steps to be the next version, and then the next level, and the next level, and the next level of each of the same things.
Gil: Okay, so a lot of people are going to Scrum, right? And Scrum actually doesn’t say a whole lot, as opposed to XP which was very specific about a lot of things, in Scrum there’s not a lot. There’s a lot of principle but not a lot of practice. So how come they still end up missing the technical practices – what’s going on there?
Arlo: So Scrum was originally written to address managers, mid-level management and planning, assuming that – and this was sort of built out about the day that XP was – that of course your teams are doing this XP thing underneath it. There was a really good quote, I think it was Sutherland, I read in Mitch Lacey’s a book on Scrum that was, “Every successful Scrum team that I have ever seen has had technical practices something like those of XP underneath it.”
Gil: I remember sitting down to lunch with Jeff Sutherland at one of the Agile conferences and he told me, “Back in the day when we got into Scrum I wrote to Kent Beck and I said, ‘Send me all you’ve got.’ And they built that stuff on top of that.
Arlo: Yeah.
Gil: Yeah, so he just told this to me over lunch.
Arlo: Yeah, so Scrum assumes XP and it adds on top of it, that was its point.
Gil: Right, but when you read the literature, it doesn’t actually assume XP anymore.
Arlo: [Laughs] No, it doesn’t. And partly that’s because Scrum grew a lot faster than XP, it was marketed extremely well, and it had a better way of transmitting the knowledge from person to person so it could grow faster. XP decided to transmit the knowledge from master to apprentice with apprenticeship-like programs. You would go code with a master for a while, and then a different master, and then after you did that three times you’d be a journeyman and you can start doing it, and that takes forever. Whereas Scrum built a course and then after your couple of days’ course you would go, have concrete experience, try and whatever, and then you’d contact a consultant and they’d help you solve problems, and that’s a lot faster. The same thing can be done with the technical practices it’s just they haven’t been packaged that way and no one has really tried to do that.
Gil: Okay, so what are the consequences? You told us before about charting where they would be. What are the consequences that you see?
Arlo: Okay, so the main one is “tale of two teams.” They both want to have a faster release cadence, that’s the reason that most teams go to Agile, “We need to release more frequently.”
One of those teams does some very quick immediate changes to planning and to releasing, and a few to how we converse about the problems, and that’s about it. And they find that pretty much immediately there able to start hitting on the release cadence. And every iteration they get some stuff done and they have a couple of bugs that don’t quite get it in the sprint but they can punt them, or they close up a couple of things, and they keep going and every iteration it becomes just a little bit harder than the iteration before to complete – actually delivering some stuff in the iteration. And after about 18 to 24 months, depending on a few nuances, they get the point where they first miss an iteration, where they can’t get anything done. And then after that their next iteration they’re able to get done what they were trying in the first one. And then the next one after that they miss and they will not hit an iteration again. They’ve accrued so much technical debt that they no longer have the capability of releasing at the cadence that they’re planning at. So that’s one path.
The other path is the XP one where when they go in, they decide “We need to release more frequently,” they first focus on transaction costs and risks. They focus on working in ways such that we stop writing as many bugs. They focus on refactoring to improve the design so that modifying this code is less likely to cause errors, or confuse people, or take time. And they keep driving the technical debt down. They find that pretty much immediately there’s no real change in release cadence. It might go a little bit faster but it’s not a lot different. But every month it becomes easier, and easier, and easier to release faster and faster. And so at some point they go ahead and switch planning but it doesn’t really matter when that happens because they’ll start releasing faster whether they change. You can do Waterfall and release every week if it’s on top of XP practices. And then it just becomes easier, and easier, and easier until they eventually a couple years down the road reach the point that they might as well just release after every commit and you start seeing Continuous Deployment and all that sort of stuff.
Gil: So you’re talking about organic growth in maturity, as opposed to mortgaging the future for fast moves now.
Arlo: Yep, exactly. And the Scrum teams are mortgaging the future because they aren’t actually addressing the core underlying transaction costs of their system. And so they’re introducing higher and higher transaction costs in order to meet the current goals. Where the XP teams are focused, laser focused, on those transaction costs and risks and then over time that just adds opportunities and options.
Gil: Okay, so what do you mean by transaction costs?
Arlo: So if I talk about a single dev inner loop cycle from: dev picks up a task that he’s got to do or figure out, through design and coding and talking to customers and all the things that you’re going to do until it’s checked in, committed, and dev can set that task down with assurance that it’s done-done, that they won’t have to come back to it and they can pick up the next one. That cycle time, anything that’s on that cycle time, that’s friction that slows it down. So if it takes a while for dev to validate whether something worked or not then that’s a transaction cost. If that validation has a partially nondeterministic result, it’s mostly right, then that’s a transaction risk. At some point some of them you’re going to have bugs slip through and then you have a long-distance feedback cycle where you’re going to have to go pull work you thought was done and start it again.
And a whole bunch of different things fall into transaction costs. Having to go a long distance to talk your customer. Having your customer be only available periodically so you have to go with batch mode conversations to them. Having within the team a bunch of specialties so that a given piece of work requires being handed off between people, devs and testers and UX people, rather than just we all are multi-capable and I take a set of work and as a pair we just execute it. All of those are transaction costs and transaction risks. And the thing that really drives success is being able to do all those inner loops, those tight loops, very, very quickly, and with very high assurance. You drive out all those costs and risks.
Gil: So those costs are really what people would normally call waste, right?
Arlo: Yep, pretty much. Transaction costs and risks are one of the set of wastes, and they’re one of the key wastes, in a Lean system.
Gil: Okay. So when we drive down the transaction cost and thereby drive down waste, this is often taken as a way to increase efficiency. But in Agile we don’t really care so much about efficiency as much as about effectiveness.
Arlo: Yes.
Gil: Okay. Tell us about that.
Arlo: Yeah. Efficiency versus effectiveness. Efficiency is all about utilization, “Am I locally using my devs to mostly do dev-y sorts of things most of the time.” Effectiveness is all about the output, “Is the throughput of the system good?”
Cycle time is a fundamental driver of throughput. The shorter my cycles and the smaller the piece of work that I can push through at one time, and the less work in process that I can have and still maintain that, the higher throughput I can achieve, the higher total volume of work that comes out, and the shorter the lead time to get any particular thing — which, given time value of money, increases its value.
It’s not about efficiency. I might do things in my inner dev loop that decrease the efficiency. For example, I talked about eliminating specializations. It is my normal expectation that on an XP team, a team member, a dev team member, whatever, is able to do testing, design, customer prototyping, UX, dev, ops, handle the calls on tech support, and may even, depending on the circumstance, get into a little bit of sales or marketing, there are teams that do and teams that don’t. It’s a very broad skill set. I am sacrificing a little bit in that those people are going to only be deep specialists in three of those (I use pair programming so I get M-shaped people, they have multiple deep specialties.) But it often means that I’m going to have a pair working on some things that of the nine things that need to be done this task, between the two of them they’ve got four of the specialties covered and the other they’re okay but they’re not specialists. But it is more effective, I get a better output, if those people drive the task to completion. And there are very few tasks that actually require expertise, like true specialist expertise, for those I use the expert but usually you want to spread the knowledge around.
Gil: Just based on the people I’ve worked with in my consulting work, we’re coming close to 2,000 people at this point, I’m guessing that many of our listeners are being very skeptical right now hearing you and saying, “My team is nowhere like this.”
Arlo: Correct.
Gil: Yes. Now, you are doing this right now at Microsoft?
Arlo: Yep.
Gil: Okay, on a grander scale than most human beings ever get to.
Arlo: Exactly [laughs]. Traditionally Microsoft is nowhere like this, right?
Gil: Okay.
Arlo: And we’re working on shifting Microsoft, shifting the fundamental culture, shifting the results from a team level to the whole company level, all towards technical excellence and the things that I’m talking about.
The key to doing that is to recognize that this is a process, this is a process of continuous improvement. This isn’t a thing that you just go get trained on for a day and all I needed was an idea or two, and then poof I get the new result. The whole thing about technical excellence, about 2-Star and 3-Star Agile especially, is it is fundamentally about learning, it’s about lifelong learning and the whole system is set up to continually improve and grow.
The industry appears to grow with an effectiveness doubling time of about two and a half years, or two, two and a half years according to, I forgot who ran some numbers. But they were looking back over time and just… a given problem, how many lines of code could it be solved in, how long would it take a team to solve it. That halved every couple of years which means that each year there’s a 35 percent growth rate in effectiveness and productivity in order to stay average, in order to keep up. You cannot have a one-time change and accomplish that growth rate. Any set of techniques and any set of capabilities that I’m talking about today as typical for XP teams, those were uncommon at some point, they’ve gotten fairly commonplace, and commonplace three years from now is going to blow these out of the water. It’s all about creating a continuously improving, continuous learning culture within the team, and setting up all of the systems such that wherever the team is today, it will be better tomorrow.
When I define working tiny, work tiny is “do everything today in a smaller scope and in a shorter period of time than you did it yesterday.” Achieving excellence is do everything better today than you did it yesterday.
Gil: I’m sorry. No, I’m laughing now because the first session in the series was with a VP of Engineering and when I asked him what he saw success as being, he said, “Doing better today than we did yesterday.”
Arlo: Absolutely! Yeah, and as long as you maintain that and build up good systems for doing that – and there are a number of really good answers and I can get into details if we want to – then wherever you are today you will hit these milestones and have the capabilities that I’m talking about as normal at some point. And those won’t be enough for you and you’ll go beyond them.
Gil: Okay. The subtitle for the session was about changing the rules of the development game and I also heard to touch on 2-Star and 3-Star. So would you share with us what you see as the new rules of the development team?
Arlo: Okay. So, Agile Fluency model mentions the 1-Star, 2-Star, 3-Star thing and it’s a really good model, go read it. It talks about 1-Star Agility is basically being able to talk in terms of customer value and the way I actually think of, starting to formulate a shifting of the model that I want to propose. 1-Star is aligned teams; that we do all our work together, not individually, we fundamentally work together and we are aligned and we’re going. And one of the best ways to do alignment is to focus on customer value, because that can pull all the politics away and we’re focused on something that we all value, it’s a good shared value. But it’s not the only one and is not sufficient and that’s why 1-Star Agile needs to have a good retrospective process, it needs to have cross-functional single-assignment teams, and it needs to have a number of other things that aren’t nominally about customer value, it’s because we need create that team.
Gil: Okay.
Arlo: Once we’ve got that team, then we can do something with it. That team is a learning organization, it is able to change its habits, it is able to grow in response to stuff. And the first thing we do is we drive out technical risk, and this is the 2-Star Agile: executing on the market cadence sustainably, forever. And that’s all these XP things and getting to zero bugs and being able to refactor and change your design on a whim and being able to introduce crosscutting concerns like internationalization in two days and all these great stories that you hear, all of which are fairly normal and repeated events. But they’re because they have really built up the ability to do execution. And then 3-Star Agile, hypothesis-driven testing and all of that.
Now teams should choose what level they need because they don’t necessarily always need to go to 3-Star. In fact if you’re in a fairly stable market, you probably don’t need 3-Star Agility, 2-Star is just fine. If you are selling apps for an app store, especially game apps, they probably have a low shelf life anyway, 2-Star agile is more than you need. You just want to get it out, you get it out once and be done with it, technical debt’s not a problem. Right?
Gil: Right.
Arlo: So you need to pick the level that’s the appropriate for you and then you go to it. Now 2-Star Agile is the one that fundamentally changes the rules of the development game. So often many teams need at least 2-Star Agility because their competitors are doing so. And when your competitor is doing so, it changes the rules. The modern rules of development are:
There basically aren’t bugs, you certainly never worry or think about bugs. When it comes time to do planning, there are no bugs now and there won’t be any bugs in the future, and I don’t need to plan for them.
Also there’s no technical dependencies or ordering to the stories. I can take any arbitrary set of stories, cut them up in various ways, and they will have the same total cost whether I slice them up or leave them big. I can order them however I want and none of the estimates change whether A is before B or B is before A, the size of A and the size of B are independent. If I change a particular implementation of a story such that it hits one section of code versus another, a nuance such that I have to do things, it’s the same cost, it doesn’t depend what part of code is used. All of this is par for the course, normal.
I can change design at any time without introducing bugs. There’s no higher probability of introducing a bug with a design-impacting change than a non-design impacting change. That’s the natural result of refactoring done well and at the high-level.
And when you have these things, now when should I do design? Well, when I have the most information at the end of the project. So I should be doing emergent design and architecture rather than preplanned and it changes things. And because I’m doing design at the end, the verification thing becomes much more of a design verification and a spec’ing process, and it ends up moving to the first. And I end up getting into the simultaneous phases thing, which I can then microslice, where I’m doing all of the things at the same time. In a given one-minute period, a dev is doing planning and customer relations and devving and testing and design and all of it and you can’t really tell when they’re doing which activity because they’re doing them all. And now I can slice that however I want, I can deliver whatever I want whenever I want, I can just imagine arbitrary pieces of scope and execute them.
That’s the new rules that people are playing with and that’s why some people get to do planning and software design from a user perspective fundamentally differently, they’re playing under those rules.
Gil: We could use some examples of what these rules manifest themselves as…
Arlo: Examples, yeah sure.
There was a big refactoring in one of our major projects recently and in the previous version of the project this particular function was refactored from 26,000 lines, in the function, to 17,000 lines of code. It was massive, it was ugly, it was horrible to work with, it was one of those that was a great design when the program was first created — the program is 30 years old — and things have changed [laughs]. It is no longer a great design. So they made very careful small tweaks to it in the previous edition and it took them about a year to be able to make the changes, and then two and a half years to stabilize it, and they still shipped a high priority bug that we had to fix in a service pack.
So then in the next version we needed to change this function to be cross-platform on a set of platforms that had fundamentally different execution constraints. And so basically all the code the way that it worked couldn’t be done that way anymore. So it was effectively a rewrite sort of a design. And breaking up this one function into, I think they ended up with like 70 different classes and very invasive change.
Showed them how to do that with refactoring instead of with local rewrites, the way that they had been doing it. Instead of it taking seven senior devs a year, it took one senior dev plus three junior devs six months. And during that six months they created a grand total of 51 bugs, all of which were found during those six months, there was no stabilization period, no bug tail at the end, they just shipped it and went on.
Gil: I’ve done things like that in my past, yes! Well there’s also the added confidence, right? As you go along, you’re not walking away from the code scared.
Arlo: [laughs] Yeah, right.
Gil: Yeah.
Arlo: Yeah, that ability to refactor it with confidence, check in with confidence and move on, it’s a very fundamentally different way things work. And it really all comes back to at the top of the hour or something you asked, “What are the things that management should focus on if you want to go towards tech excellence?” Refactoring, pair programming. Pair programming is the key to learning, and refactoring is the key to being able to make these changes safely and driving out the transaction risks.
Gil: Exactly, okay. So you just gave as an example from a scary piece of legacy. So what if we have a team that’s not working on a scary piece of legacy? They do want to get to the point where really there are no bugs. What does it mean to believe or operate as if there are no bugs?
Arlo: Okay, so greenfield, I see a greenfield team as typically about two weeks from being a legacy code team [both laugh]. And in fact if you look at XP, XP is designed to deal well with legacy code, that really is the whole point is just to solve the legacy code problem. And so their perspective on a greenfield one is, “We will get the first version out in the market as quickly as possible, preferably in a week. And now it’s a legacy code problem, we’ve reduced it to a previously solved issue.” [laughs]
Gil: Well it’s not that simple, right?
Arlo: It is and it isn’t. It is harder to learn to refactor in greenfield. And in fact at this point when I’m helping a team learn, I try and help them learn on legacy because there are a lot of things that are more easily learned on legacy code than on greenfield. You don’t have as many bad examples sitting there that you can easily make good in a greenfield situation. And so yeah, it is harder to learn, on the other hand it’s easier to do a number of other things once you’ve learned. And when you’re in those greenfield situations, again you need to start with no bugs. When you have no code you also have no defects and that’s a wonderful place to be and your whole job is just to maintain that.
Gil: Okay.
Arlo: You don’t need to burn down the existing bugs, you don’t ever need to work around them, you just need to ensure that you don’t introduce the first one. And you do that by continually adding a little bit of scope, refactoring the code to be simple in its description of the current scope, and going on and on. And you use unit testing to help tell you when and what to refactor. You can maintain good, solid, simple design à la Corey Haines’ or J. B. Rainsberger’s definition of simple design, or Kent Beck’s I guess is the original one, from the get-go because you start with something which being empty meets all the criteria.
Gil: Yes, that’s mathematical induction.
Arlo: Yeah, an induction. It’s a very nice way to work, you still need to learn a lot of the refactoring techniques because fundamentally you’re going to code something and it’s going to be not perfect for the thing that you didn’t think of. Eventually you’re going to think of that thing and then you’re going to need to refactor it. And so it’s that same set of techniques, it’s just that instead of going into a 17,000-line function you need to have your sensitivities tuned high enough that when the function gets to be six lines of code and it’s got like a loop and a conditional in there, that’s starting to look pretty scary and you refactor it because it’s not simple enough. So it’s the same techniques but with a higher level of sensitivity and you get to always be in the clean code.
Gil: I’m thinking back to the example you gave on legacy. I’m sure a lot of our listeners actually work in legacy environments, and most likely in languages like C++ where rapid feedback and change are a bit more challenging than more modern languages. So how can we help them?
Arlo: Legacy languages are a challenge (legacy platforms and languages). Many of the people that I work with work in C++ as well, and it is painful. The refactoring tools aren’t there so you have to do a lot of your refactorings by hand. Instead of being able to use ReSharper, you have to have your copy of the Refactoring book by Fowler and go look at the mechanics for each one and mechanically go through the steps he says every time perfectly. It makes pair programming far more important, because if you’re going to be doing something that mechanical and it depends on high discipline, you better have someone there to help maintain that discipline. And you’re going to be refactoring a lot so you better have them there a lot and regularly available.
Gil: Sorry, say that again — better have what?
Arlo: Have a pair partner regularly available.
Gil: Okay.
Arlo: Because being able to maintain the level of discipline that a computer can maintain on a refactoring algorithm is not easy [laughs]. Individuals will make mistakes, pairs make far fewer on that. And every mistake you make is a bug, so bring in the pair, kill the bugs. So that becomes key when you’re in a legacy language, pair more.
Also when you’re in a legacy language and environment, really focus on dependencies, especially physical dependencies in C++. So a lot of your build times and a lot of your cycle times and verifications are based off of how inter-twisted your code is. And if you can start breaking off pieces… there’s an effort going on in one of our groups call the LIBletization effort.
Gil: What was that?
Arlo: LIBletization. They’re creating LIBlets, little LIBlets.
Gil: Oh, LIBlets, yes.
Arlo: Yes, LIBlets. And so what they do is have this big massive ball of mud, there’s a single DLL, and it’s been shipped as a single DLL for 20 mumble, mumble years, maybe 30 mumble, mumble, a long time [laughs]. And are they go in and they grab little tiny pieces of functionality and shake them loose and snip off the edges so that they can make little static-link lib. Now that thing can be compiled separately, It can be unit-tested, they pull out the part that’s just code and leave behind the mess that’s talking to some external system or anything that’s hard and nasty. And they keep doing that and creating more and more little LIBlets, and over time more and more of the code and the changes you want to make are executed within a LIBlet or maybe the orchestration of two LIBlets together. And it becomes pretty easy, you just need to change and to verify and whatever, and less of it goes into the big snarl.
So it’s those sorts of techniques to start really breaking it apart. The build time of the LIBlets is very fast, and they can build in parallel, and you only need to build the ones you want. And that speeds up your cycle time.
Gil: I’m sure you’re seeing a lot of people who have a pretty long journey ahead of them when it comes to getting to 2-Star or maybe even 1-Star. What sort of advice would you give people to increase their technical excellence, if they can only do so gradually?
Arlo: So if you can only do things gradually, which is fine, then first thing is to create a good team that can improve. So it is critical to have a set of people, that every person in a group identifies that group as their group. It’s small, it can get things done, it can make decisions. If you’ve got that single-assignment, small team, then now you’ve got a group that can decide they want to do something and mutually support each other in doing it. That will dramatically increase the rate that you can make changes, that you can learn, etc. which gives you free time and money to spend on things.
Gil: This really goes back to our session #2, about building a team-friendly environment.
Arlo: Build a team-friendly environment, absolutely. And share work, individual work assignment is a wonderful idea from the ‘70’s that has been replaced [laughs].
Gil: Yeah, deprecated.
Arlo: Yeah, it has been deprecated. Much like the separation of doing the work from deciding how the work is done that is the old way management was defined and was a wonderful idea from the 1800’s that has been deprecated, and management is done differently these days. Same thing.
Once you’ve got those teams, those are centers of growth and improvement, then start bringing on all of the various very lowest-level things at once. So you don’t want to bring in, “We’re going to do TDD with red-green-refactor.” Oh gosh, red-green-refactor has so many prereqs. Choose the ones that have no prereqs or where you already meet all the prereqs but that support each other. So pair program a little bit, know why you’re pair programming, or choose to pair program more but know why. And also learn the basic refactorings, learn the core five, and just get those fluent, get so that I can do them all the time and so that I can start reading code by refactoring it to legibility. And at the same time bring in automated dev testing, and I don’t care whether they’re integration tests or anything, just start doing it. And at the same time start bringing in thinking of scope blocks instead of quality levels and we’re going to get a set of things done and then we’re going to move on to the next set of things. Bring in all of that once, but don’t expect excellence at all of it, get fluent at that lowest level and then start building on it. And then add on things on top of that as they become relevant or make them fun or whatever. And do a mob-programming exercise with a team near you that works in a different way and trade ideas, and so on and so forth. But you’re creating this environment in which learning becomes cheap, and once learning and improvement become cheap then you can do so.
Gil: Yeah, it sounds like this is worth repeating. If you can build an environment where learning is cheap, then a lot of magic will happen.
Arlo: A lot of magic happens there.
Gil: Okay, Arlo, this has been a whirlwind, to say the least [Arlo laughs]. But that’s okay, that’s what I expected, I knew what I was getting myself into.
Arlo: I don’t do just one thing. And Agile success really is the combining of a whole bunch of little details of a whole bunch of different things. It’s not one big idea, it’s all the details.
Gil: To all our listeners, I hope you took good notes here, there was a lot to cover. Some of this is quite advanced but it’s workable, it’s true, it’s been shown to work, and it’s going to take time.
To see what Arlo is up to there days check out ArloBelshee.com.
