• Needed to be a 10x developer so that a company/project works at amazing efficiency.

Quotes

On last minute changes

It’s not a good sign when they’re still attaching parts to the space shuttle at liftoff time.

Key Terms

WIP
Work In Progress
Change
Any task that needs to be done
Types of Work
Types of work available in an IT project
The Three Ways
Ways to eliminate WIP and fully optimize IT operations
Brent
Over-powered resource that’s critical from start to the end

Highlights

When our Marketing people hear this ad, they’ll go ballistic. Because they’re likely art or music majors, not people with a technology background, they’ll publicly promise the impossible and it will have to figure out how to deliver.

Patty nods. “We should have done this a long time ago. We bump up the priorities of things all the time, but we never really know what just got bumped down. That is, until someone screams at us, demanding to know why we haven’t delivered something.”

“As you can see, my assistant has brought in a pile of blank index cards. I want each group to write down every change they’re planning, one change per index card. I want three pieces of information: who is planning the change, the system being changed, and a one-sentence summary. “I’ve drawn a calendar on the whiteboard where we will eventually post approved changes according to their scheduled implementation,” I continue. “Those are the rules. Short and simple.”

A half hour later, we finally write on the whiteboard: “a ‘change’ is any activity that is physical, logical, or virtual to applications, databases, operating systems, networks, or hardware that could impact services being delivered.”

He gestures broadly with both arms outstretched, “In the 1980s, this plant was the beneficiary of three incredible scientifically-grounded management movements. You’ve probably heard of them: the Theory of Constraints, Lean production or the Toyota Production System, and Total Quality Management. Although each movement started in different places, they all agree on one thing: wip is the silent killer. Therefore, one of the most critical mechanisms in the management of any plant is job and materials release. Without it, you can’t control wip.”

Oblivious to my impatience, he says, “Let me tell you a story. Decades ago, there used to be a guy named Mark. He was the supervisor for that first work center, right down there by that desk. Those racks hold the folders for incoming jobs. Isn’t it amazing that those folders look exactly like they did back then? “At any rate,” he continues, “one day I see Mark picking up a folder to start some job. I ask him, ‘On what basis did you choose that job, versus any of the others?’ “And you know what he tells me? He says, ‘It’s a job that requires this work center first. And we’re open.’” He shakes his head incredulously. “I could hardly believe it. I tell him, ‘Your station is just the first of twenty operations. You don’t factor the availability of any of the other nineteen stations in your decision?’ And he replies, ‘Well, no. This is the way I’ve done it for twenty years.’” He laughs. “I suppose to him, it sounds like a reasonable way to pick which job to perform. He’s keeping the first station busy, and it’s similar to first-in, first-out scheduling. But of course, now everyone knows that you don’t release work based on the availability of the first station. Instead, it should be based on the tempo of how quickly the bottleneck resource can consume the work.”

I just stare at him blankly. He continues, “Because of how Mark was releasing work, inventory kept piling up in front of our bottleneck, and jobs were never finished on time. Every day was an emergency. For years, we were awarded Best Customer of the Year from our air freight shipment company, because we were overnighting thousands of pounds of finished goods to angry customers almost every week.”

He pauses and then says emphatically, “Dr. Eliyahu M. Goldratt, who created the Theory of Constraints, showed us how any improvements made anywhere besides the bottleneck are an illusion. Astonishing, but true! Any improvement made after the bottleneck is useless, because it will always remain starved, waiting for work from the bottleneck. And any improvements made before the bottleneck merely results in more inventory piling up at the bottleneck.”

“Rest assured, though,” he says, pointing at the job release desk, “in order to get to where you want to go, eventually you will need to figure out what your equivalent to that desk is. You must figure out how to control the release of work into it Operations and, more importantly, ensure that your most constrained resources are doing only the work that serves the goal of the entire system, not just one silo.

“Once you figure this out, young Bill, you will be well on your way toward understanding the Three Ways,” he says. “The First Way helps us understand how to create fast flow of work as it moves from Development into it Operations, because that’s what’s between the business and the customer. The Second Way shows us how to shorten and amplify feedback loops, so we can fix quality at the source and avoid rework. And the Third Way shows us how to create a culture that simultaneously fosters experimentation, learning from failure, and understanding that repetition and practice are the prerequisites to mastery.”

Erik hands me a slip of paper with a phone number on it. “Remember, there are four types of work. You’ve named business project work as one. When you have the other three, give me a call.” He takes out his car keys from his pocket and asks, “Do you want a lift back to your office?” It’s 5:10 p.m. when I finally get back to my cubicle. I log back into my clunker laptop to reply to e-mails. But I can’t concentrate. The last hour with Erik was like being in a strange parallel universe. Or like being forced to watch a psychedelic movie made in a drug-induced haze.

“We need to focus on the riskiest changes,” I continue. “The 80/20 rule likely applies here: Twenty percent of the changes pose eighty percent of the risk.”

Suddenly I laugh out loud at the absurd number of questions I’ve just asked myself. I feel like a one-man debate club. Or that Erik tricked me into doing some philosophical navel-gazing. Thinking for a moment, I decide there’s value in knowing that changes represent yet another category of work but don’t know why. I’ve now identified three of the four categories of work. For a brief moment, I wonder what the fourth category of work is.

Patty laughs, apparently recalling the story. She says, “I’m not suggesting Brent is doing this deliberately, but I wonder whether Brent views all his knowledge as a sort of power. Maybe some part of him is reluctant to give that up. It does put him in this position where he’s virtually impossible to replace.” “Maybe. Maybe not,” I say. “I’ll tell you what I do know, though. Every time that we let Brent fix something that none of us can replicate, Brent gets a little smarter, and the entire system gets dumber. We’ve got to put an end to that. “Maybe we create a resource pool of level 3 engineers to handle the escalations, but keep Brent out of that pool. The level 3s would be responsible for resolving all incidents to closure, and would be the only people who can get access to Brent—on one condition. “If they want to talk with Brent, they must first get Wes’ or my approval,” I say. “They’d be responsible for documenting what they learned, and Brent would never be allowed to work on the same problem twice. I’d review each of the issues weekly, and if I find out that Brent worked a problem twice, there will be hell to pay. For both the level 3s and Brent.” I add, “Based on Wes’ story, we shouldn’t even let Brent touch the keyboard. He’s allowed to tell people what to type and shoulder-surf, but under no condition will we allow him to do something that we can’t document afterward. Is that clear?” “That’s great,” Patty says. “At the end of each incident, we’ll have one more article in our knowledge base of how to fix a hairy problem and a growing pool of people who can execute the fix

Patty adds, “To prevent another Brent escalation, we should log every keystroke and record the terminal session. Maybe even have someone follow him around with a video camera and turn on audit logging so we know exactly what he changed.” I like it, although it sounds a bit extreme. However, I suspect that it will take extreme measures to get us out of this situation. I venture, “Maybe we take away his production access, so the only way the work can get done is him telling the level 3s what to do.” Wes guffaws. “He might quit if we did that right away.” “So, who do we have that’s available to put into this level 3 resource pool?” I ask.

“I’ll define the new Brent procedures,” Patty says. “I like gating all access to him through you and Wes. But how will we discourage people like that vp of Logistics from going directly to Brent?” I reply immediately, “We’ll collect the names of the people who do, and I’ll call each of their bosses to tell them to cease and desist. And then I’ll let Steve know how they’re disrupting Phoenix.”

“Maybe we send them to whatever conference or training they want. When senior engineers get to the level of Brent, or aspire to be Brent, they want to learn and share what they’ve done. As for Brent, how about we make him take a week off, completely free of any escalation duties?” Wes suggests. “My God,” Wes continues, shaking his head. “I don’t think Brent’s even been able to take a day off without a pager in about three years. You know, he’ll burst into tears when we offer that to him.”

Before I forget, I add, “Wes, I want a timesheet from Brent every day, and I want every escalation Brent works in the ticketing system. We need that documented so we can analyze it later.

“This is amazing,” Patty says. “We’ve gotten more change, incident, and escalation processes going in the last week than we have in the last five years!” “It’s probably just in the nick of time,” Wes says, sounding relieved. “Do me a favor, and don’t tell anyone I said that. I’ve got a reputation to protect.”

Erik called wip, or work in process, the “silent killer,” and that inability to control wip on the plant floor was one of the root causes for chronic due-date problems and quality issues. We just discovered that sixty percent of our changes didn’t complete as scheduled

“We have 240 incomplete changes this week. If we have four hundred new changes coming in next week, we’ll have 640 changes on the schedule next week! “We’re like the Bates Motel of changes,” I say in disbelief. “Changes go in but never come out. Within a month, we’ll have thousands of changes that we’ll be carrying around, all competing to get implemented.”

Looking at the cards piling up, can we afford not to? But when was the question ever asked whether we should accept the work? And on what basis did we ever make that decision? Again, I don’t know the answer. But, worse, I have a feeling that Erik may not be a raving madman. Maybe he’s right. Maybe there is some sort of link between plant floor management and it Operations. Maybe plant floor management and it Operations actually have similar challenges and problems.

“Patty, we need a better understanding of what work is going to be heading Brent’s way. We need to know which change cards involve Brent—maybe we even make that another piece of information required when people submit their cards. Or use a different color card—you figure it out. You need to inventory what changes need anything from Brent, and try to satisfy it instead with the level 3 engineers. Failing that, try to get them prioritized so we can triage them with Brent.”

The entire it Operations team was assembled in preparation for the deployment at 4 p.m. But there was nothing to do because we hadn’t received anything from Chris’ team; they were still making last minute changes. It’s not a good sign when they’re still attaching parts to the space shuttle at liftoff time.

What do you mean we need to open up a firewall port? Why the hell didn’t you tell us this two hours ago?” He then slams the phone down hard, and then pounds the table with his fist, yelling, “Idiots!” Brent looks up from the developer laptop, rubbing his eyes with fatigue. “Let me guess. The front-end can’t talk to the database server because someone didn’t tell us we need to open a firewall port?” The engineer nods with exhausted fury, and says, “I cannot freaking believe this. I was on the phone with that jackass for twenty minutes, and it never occurred to him that it wasn’t a code problem. This is fubar.”

I listen intently. What’s happening with Phoenix is a combination of the need to deliver needed features to market, forcing us to take shortcuts, which are causing ever-worsening deployments. He’s put his finger on a very important downward spiral we need to break out of

So much of what I’ve been trying to do during my short tenure as vp of it Operations is to prevent unplanned work from happening: coordinating changes better so they don’t fail, ensuring the orderly handling of incidents and outages to prevent interrupting key resources, doing whatever it takes so that Brent won’t be escalated to…

Very good,” he says. “You’ve put together tools to help with the visual management of work and pulling work through the system. This is a critical part of the First Way, which is creating fast flow of work through Development and it Operations. Index cards on a kanban board is one of the best mechanisms to do this, because everyone can see wip. Now you must continually eradicate your largest sources of unplanned work, per the Second Way

He replies, “Ah, well if we’re going to talk about your next steps, you definitely need to know about constraints because you need to increase flow. Right now, nothing is more important.” Erik assumes a lecturing voice as he starts, “You say you learned about plant operations management when you were in business school. I hope as part of your curriculum, you read The Goal by Dr. Eli Goldratt. If you don’t have a copy anymore, get another one. You’re going to need it.”

He continues, “You’ve identified this Brent person as a constraint to restore service. Trust me, you’ll find that he constrains many other important flows of work, as well.”

There are five focusing steps which Sensei Goldratt describes in The Goal: Step 1 is to identify the constraint. You’ve done that, so congratulations. Keep challenging yourself to really make sure that’s your organizational constraint, because if you’re wrong, nothing you do will matter. Remember, any improvement not made at the constraint is just an illusion, yes? “Step 2 is to exploit the constraint,” he continues. “In other words, make sure that the constraint is not allowed to waste any time. Ever. It should never be waiting on any other resource for anything, and it should always be working on the highest priority commitment the it Operations organization has made to the rest of the enterprise. Always.” I hear him say encouragingly, “You’ve done a good job exploiting the constraint on several fronts. You’ve reduced reliance on Brent for unplanned work and outages. You’ve even started to figure out how to exploit Brent better for the three other types of work: business and it projects and changes. Remember, unplanned work kills your ability to do planned work, so you must always do whatever it takes to eradicate it. Murphy does exist, so you’ll always have unplanned work, but it must be handled efficiently. You’ve still got a long way to go

In a more stern voice, he says, “But you’re ready to start thinking about Step 3, which is to subordinate the constraint. In the Theory of Constraints, this is typically implemented by something called Drum-Buffer-Rope. In The Goal, the main character, Alex, learns about this when he discovers that Herbie, the slowest Boy Scout in the troop, actually dictates the entire group’s marching pace. Alex moved Herbie to the front of the line to prevent kids from going on too far ahead. Later at Alex’s plant, he started to release all work in accordance to the rate it could be consumed by the heat treat ovens, which was his plant’s bottleneck. That was his real-life Herbie.”

“Fully two decades after The Goal was published,” he continues, “Sensei David J. Anderson developed techniques of using a kanban board to release work and control wip for Development and it Operations. You may find that of interest. You and Penelope are close with your change board to a kanban board that can manage flow.” “So, here’s your homework,” he says. “Figure out how to set the tempo of work according to Brent. Once you make the appropriate mapping of it Operations to work on the plant floor, it will be obvious. Call me when you’ve figured it out.”

“Wait, wait,” I say, hurriedly before he hangs up. “I’ll do the homework, but aren’t we missing the entire point here? What caused all the unplanned work is Phoenix. Why are we focusing on Brent right now? Don’t we need to address all the issues with Phoenix inside of Development, where all the unplanned work actually came from?” “Now you sound just like Jimmy, complaining about things you can’t control,” he sighs. “Of course Phoenix is causing all the problems. You get what you design for. Chester, your peer in Development, is spending all his cycles on features, instead of stability, security, scalability, manageability, operability, continuity, and all those other beautiful ’itties. “On the other end of the assembly line, Jimmy keeps trying to retrofit production controls after the toothpaste is out of the tube,” he says, scoffing. “Hopeless! Futile! It’ll never work! You need to design these things, what some call ‘nonfunctional requirements,’ into the product. But your problem is that the person who knows the most about where your technical debt is and how to actually build code that is designed for Operations is too busy. You know who that person is, don’t you?”

I groan. “Brent.” “Yep,” he says. “Without solving your Brent problem, you’ll just be inviting him to design and architecture meetings with Development, but he’ll never show up because…” Being prompted again, I respond, “Unplanned work.”

“Good!” he says. “You’re getting better at this. But before you get a big head, I’ll tell you that there’s still a big piece of the First Way that you’re missing. Jimmy’s problem with the auditors shows that he can’t distinguish what work matters to the business versus what doesn’t. And incidentally, you have the same problem, too. Remember, it goes beyond reducing wip. Being able to take needless work out of the system is more important than being able to put more work into the system. To do that, you need to know what matters to the achievement of the business objectives, whether it’s projects, operations, strategy, compliance with laws and regulations, security, or whatever.”

He continues, “Remember, outcomes are what matter—not the process, not controls, or, for that matter, what work you complete.”

He says, “I know that right now, nothing, absolutely nothing, is more important to the company’s success than how this leadership team performs. “The second thing I’ve learned is that my actions have made almost all our it problems worse. I turned down Chris and Bill’s requests for more budget, Bill’s request for more time to do Phoenix right, and micromanaged things when I wasn’t getting the results I wanted.”

Steve then looks at me. “The person I wronged the most was Bill. He told me things that I didn’t want to hear, and I shut him down. In hindsight, he was completely right, and I was completely wrong. And for that, Bill, I’m very sorry.”

“A great team doesn’t mean that they had the smartest people. What made those teams great is that everyone trusted one another. It can be a powerful thing when that magic dynamic exists.

Steve continues, “One of my favorite books about team dynamics is Five Dysfunctions of a Team, by Patrick Lencioni. He writes that in order to have mutual trust, you need to be vulnerable. So, I’m going to tell you a little about myself and what makes me tick.

“It may make you uncomfortable, but it’s part of what I need from you as leaders. If you can’t do it for yourself, do it for the livelihood of the nearly four thousand Parts Unlimited employees and their families. I don’t take that responsibility lightly, and you shouldn’t either.” Oh, shit. That’s another part of “management off-sites” I forgot about. Touchy-feely crap.

“He says that although I’ve gotten consistently high ratings over the years, without exception, none of the people who served under me would want to work with me again. He tells me that if there were an Asshole of the Decade Award, I’d win by a wide margin. And that if I want to make something of myself, I need to get this fixed.”

“Over the next three decades, I became a constant student of building great teams that really trust one another. I did this first as a materials manager, then later as a plant manager, as head of Marketing, and later, as head of Sales Operations. Then twelve years ago, Bob Strauss, our ceo at the time, hired me to become the new coo.” Steve exhales slowly, rubbing his face, suddenly looking very tired and old. “Somehow, I’ve made the wrong turn again, just like I did in the Army. I’ve become that person I promised myself I’d never be again.”

“First, I’d like to thank all of you for giving of yourself and doing that exercise with me,” he says. “Although it’s nice to get to know each of you better, I wouldn’t do this if I didn’t think it was important. Solving any complex business problem requires teamwork, and teamwork requires trust. Lencioni teaches that showing vulnerability helps create a foundation for that.

Steve puts both hands in front of him, and says, “Just to get the ball rolling, I’d like to propose that one of our main problems is that we blow every commitment and schedule that we make. People outside of it are always grumbling that we miss whatever expectations we set. By a mile. “Which makes me think,” he says, looking around the room, “that we’re probably not good at making internal commitments to one another here within it. Thoughts?”

Chris turns red, raising both hands in front of him. “That’s not what I meant.” He thinks for a moment, adding, “It was a total disaster. But, technically, we did hit the date.” Interesting. “If that’s true,” I say, digging in, “there’s something really wrong with our definition of what a ‘completed project’ is. If it means ‘Did Chris get all his Phoenix tasks done?’ then it was a success. But if we wanted Phoenix in production that fulfilled the business goals, without setting the entire business on fire, we should call it a total failure.”

Thinking for a moment, I finally say, “I don’t know. But this is a recurring pattern. Chris’ group never factors in all the work that Operations needs to do. And even when they do, they use up all the time in the schedule, leaving none for us. And we’re always left cleaning up the mess, long afterward.” Chris nods understandingly. “Well, you and I are fixing some of this. Part of it is a planning and architecture issue, which you and I have talked about fixing. But you’re underestimating how much of a bottleneck your group is. We’ve got a bunch of other applications that need to be deployed, but because your team is tied up, all the other deployments waiting in line get delayed as well.”

He adds, “On any given week, we’ve got five or six application groups waiting in line for your group to deploy something or another. And when anything goes wrong, everything gets stacked up. No offense, but when you guys are late, it’s like an airport that closes down. Before you know it, you have a bunch of airplanes circling, all waiting to land.”

Wes grumbles loudly, “Yeah, well, that’s what happens when the airplane you’ve built crash lands, totally destroying the runway.” Then Wes raises a placating hand. “Look, I’m not blaming you, Chris. I’m just stating a well-known fact. When deployments don’t go as planned, whether the plan was written by your group or mine, it affects everybody else. That’s all I’m saying.”

I turn to look at Steve, saying, “I showed you our project list. On top of the thirty-five business projects, we’ve got another seventy-five or so Ops projects we’re working. We’ve got a backlog of thousands of changes that apparently all need to execute for some reason or another. On top of that, we have an ever increasing amount of unplanned work, mostly caused by all our fragile applications breaking, which includes Phoenix.” I say flatly, “We are way over capacity, given the amount of work in front of us. And we haven’t even counted properly the big audit finding remediation project yet, which Steve says is still top-priority.”

“He asked on what basis do we decide whether we can accept a new project. When I said that I didn’t know, he took me on another tour of mrp-8 manufacturing plant. He took me to Allie, the Manufacturing Resource Planning Coordinator, and asked her how she decides on whether to accept a new order.” I flip back to my notes. “She said that she would first look at the order and then look at the bill of materials and routings. Based on that, she would look at the loadings of the relevant work centers in the plant and then decide whether accepting the order would jeopardize any existing commitments. “Erik asked me how we made the same type of decision in it,” I recall. “I told him then, and I’ll tell you now, I don’t know. I’m pretty sure we don’t do any sort of analysis of capacity and demand before we accept work. Which means we’re always scrambling, having to take shortcuts, which means more fragile applications in production. Which means more unplanned work and firefighting in the future. So, around and around we go.” To my surprise, Erik interrupts. “Well put, Bill. You’ve just described ‘technical debt’ that is not being paid down. It comes from taking shortcuts, which may make sense in the short-term. But like financial debt, the compounding interest costs grow over time. If an organization doesn’t pay down its technical debt, every calorie in the organization can be spent just paying interest, in the form of unplanned work.” “As you know, unplanned work is not free,” he continues. “Quite the opposite. It’s very expensive, because unplanned work comes at the expense of…”

“Unplanned work has another side effect. When you spend all your time firefighting, there’s little time or energy left for planning. When all you do is react, there’s not enough time to do the hard mental work of figuring out whether you can accept new work. So, more projects are crammed onto the plate, with fewer cycles available to each one, which means more bad multitasking, more escalations from poor code, which mean more shortcuts. As Bill said, ‘around and around we go.’ It’s the it capacity death spiral.” I smile to myself at Erik mangling Wes’ name. I’m not sure what kind of mental game he’s playing, but it’s amusing to watch. Uncertain, I ask Steve, “Are we even allowed to say no? Every time I’ve asked you to prioritize or defer work on a project, you’ve bitten my head off. When everyone is conditioned to believe that no isn’t an acceptable answer, we all just became compliant order takers, blindly marching down a doomed path. I wonder if this is what happened to my predecessors, too.”

“Of course you can say no!” Steve replies heatedly, with a look of genuine irritation on his face. He then takes a deep breath before saying, “Let me be clear. I need you to say no! We cannot afford to have this leadership team be order takers. We pay you to think, not just do!”

He looks right at me. “If you, or for that matter, anyone knows that a project will fail, I need you to say so. And I need it backed up with data. Give me data like that plant coordinator showed you, so we can understand why. Sorry, Bill, I like you a lot, but saying no just based on your gut is not enough.”

Erik snorts and mutters, “That’s some pretty nice, soaring rhetoric, Steve. Very moving. But you know what your problem is? You guys in the business are punch drunk on projects, taking on new work that doesn’t have a prayer of succeeding. Why? Because you have no idea what capacity you actually have. You’re like the guy who is always writing checks that bounce, because you don’t know how much money you have and never bother opening your mail.

“Let me tell you a story,” he says. “Let me tell you about what that mrp-8 plant was like before I arrived. Those poor bastards would get these manila envelopes that would just show up, containing all sorts of crazy orders. The business would make absurd commitments to ship something at some impossible date, oblivious to all the work already in the system.” He continues, “It was a nightmare everyday. They had inventory piled up to the ceiling. And was there a systematic way to get wip through the plant? Hell, no! What got worked on was based on who yelled the loudest or most often, who could engineer the best side deals with the expediters, or who could get the ear of the highest ranking executive.”

Erik is as animated as I’ve ever seen him. “We started restoring sanity when we figured out where our constraint was. Then we protected it, making sure that time on the constraint was never wasted. And we did everything to make sure work flowed through it.” Erik then grows still and merely says, “To fix your problem, you need to do a lot more than just learning how to say no. That’s the tip of the iceberg.”

Steve is the first to speak. “You must be out of your mind,” Steve says, his initial disbelief turning into exasperation. “You want to just stop doing work? Who do you think we are? Subsidized potato farmers paid not to grow crops?”

He starts rattling off the points on his fingers, “We’ve got the audit finding that has board visibility, the high-visibility project that can’t fail, and an operational failure that can’t happen again, either. We should pour on the gas and put in the security controls we need, once and for all.”

Wes interjects, chortling to John, “I’m stunned! I thought you would love Bill’s idea. I mean, you love stopping things from getting done and saying no, right? This should be like a dream come true for you!”

Looking at John, he continues, “Remember, Jimmy, the goal is to increase the throughput of the entire system, not just increase the number of tasks being done. And if you don’t have a trustworthy system of work, why should I trust your system of security controls? Bah. A total waste of time.” John looks back at Erik, puzzled. “What?”

“You’ve been a plant manager. Think of it as freezing materials release until enough wip completes and leaves the plant. In order to control this system, we need to reduce the number of moving parts.”

The idea is for it Operations and Development to not accept any new projects for two weeks and to stop all work in it Operations except for work related to Phoenix. I look around. “If we single-task on the most important project for two weeks and still aren’t able to make a big dent, then I think we should all find new day jobs.”

John crosses his arms. “I’m not sure if I can support this insane proposal. First, I’ve never seen any organization do anything even remotely like this before. Second, I’m very concerned that if we do this, we’ll lose our shot at getting all the audit issues fixed. As Steve has already said, those audit findings could kill the company, too.” “You know what your problem is?” Erik says, pointing a finger at John. “You never see the end-to-end business process, so I guarantee you that many of the controls you want to put in aren’t even necessary.” John says, “What?” Again, Erik waves John’s question away. “Don’t worry about it for now. Let the inevitable happen, and we’ll see what we can learn from it.”

“My guys are routinely strong-armed into doing pet projects by almost every manager in this company. I think we need an e-mail from you to the entire company, not only explaining why you’re doing this, but what the consequences will be if someone tries to put unauthorized work into the system.” Erik makes an encouraging noise. “No problem,” Steve quickly replies. “I’ll send you all a draft after this meeting. Revise it and I’ll send it out to all the company managers. Good enough for you?”

Patty looks over at him, appearing surprised. “You actually doubted it? Come on, we’ve both been talking about how we’ve never seen people so focused before. It’s amazing how the project freeze has reduced the priority conflicts and bad multitasking. We know it’s made a huge difference in productivity.”

“When we have multiple streams of work going on simultaneously, how does anyone decide what needs to get worked on at any given time?” “Well,” Wes says, “we trust them to make the right decision, based on the data they have. That’s why we hire smart people.” This is not good.

“Let’s be honest,” Patty says. “Priority 1 is whoever is yelling the loudest, with the tie-breaker being who can escalate to the most senior executive. Except when they’re more subtle. I’ve seen a bunch of my staff always prioritizing a certain manager’s requests, because he takes them out to lunch once a month.”

More sternly, he asks, “So now, tell me again which work centers you’ve determined to be the constraints in your organization?” I smile, answering easily, “Brent. We talked about that before.” He scoffs, turning back to look at the plant floor. “What?” I nearly shout. “How can it not be Brent? You even congratulated me when I told you it was Brent a couple of weeks ago!” “Suddenly Brent is a robotic heat treat oven? You’re telling me your equivalent of that paint curing booth down there is Brent?” he says with mock disbelief. “You know, that might be the dumbest thing I’ve ever heard.”

He continues, “So, where would that leave your two managers, Chester and Penelope? Let me guess. Maybe they’re equivalent to that drill press station and that stamping machine over there? Or maybe it’s that metal grinder?” Erik looks sternly at me, “Get serious. I asked you what work centers are your constraints. Think.”

“Oh,” I say, thinking out loud. “The heat treat oven is a work center, which has workers associated with it. You asked what work centers are our constraints, and I told you that it was Brent, which can’t be right, because Brent isn’t a work center. “Brent is a worker, not a work center,” I say again. “And I’m betting that Brent is probably a worker supporting way too many work centers. Which is why he’s a constraint.”

“Now we’re getting somewhere!” Erik says, smiling. Gesturing broadly at the plant floor below, he says, “Imagine if twenty-five percent of all the work centers down there could only be operated by one person named Brent. What would happen to the flow of work?” I close my eyes to think. “Work wouldn’t complete on time, because Brent can only be at one work center at a time,” I say. Enthusiastically, I continue, “That’s exactly what’s happening with us. I know that for a bunch of our planned changes, work can’t even start if Brent isn’t on hand. When that happens, we’ll escalate to Brent, telling him to drop whatever he’s doing, so some other work center can get going. We’ll be lucky if he can stay there long enough for the change to be completely implemented before he’s interrupted by someone else.”

“Obviously,” he continues, “every work center is made up of four things: the machine, the man, the method, and the measures. Suppose for the machine, we select the heat treat oven. The men are the two people required to execute the predefined steps, and we obviously will need measures based on the outcomes of executing the steps in the method.” I frown. These factory terms are vaguely familiar from my mba years. But I never thought they’d be relevant in the it domain.

“Good,” he says. “You’re standardizing Brent’s work so that other people can execute it. And because you’re finally getting those steps documented, you’re able to enforce some level of consistency and quality, as well. You’re not only reducing the number of work centers where Brent is required, you’re generating documentation that will enable you to automate some of them.” He continues, “Incidentally, until you do this, no matter how many more Brents you hire, Brent will always remain your constraint. Anyone you hire will just end up standing around.”

“You’re asking about how to lift the project freeze. Your problem is that you keep confusing two things. Until you can separate them in your head, you’ll just walk around in circles.”

commission, and the maintenance team is working on getting it back online. What would happen if every piece of equipment down there needs Brent to fix it?” I laugh. “Every outage escalated immediately to Brent.” “Yes.” He continues, “Let’s start with your first question. Which projects are safe to release when the project freeze is lifted? Knowing how work flows through certain work centers and how some work centers require Brent and some do not, what do you think the answer is?” I slowly repeat what Erik just recited, trying to piece together the answer “I got it,” I say, smiling. “The candidate projects which are safe to release are those that don’t require Brent.” I smile even wider when he says, “Bingo. Pretty simple, yes

“Wait, how do I know which projects don’t require Brent? We never think we actually need Brent until we’re halfway through the work!”

“What you’re building is the bill of materials for all the work that you do in it Operations. But instead of a list of parts and subassemblies, like moldings, screws, and casters, you’re cataloging all the prerequisites of what you need before you can complete the work—like laptop model numbers, specifications of user information, the software and licenses needed, their configurations, version information, the security and capacity and continuity requirements, yada yada…” He interrupts himself, saying, “Well, to be more accurate, you’re actually building a bill of resources. That’s the bill of materials along with the list of the required work centers and the routing. Once you have that, along with the work orders and your resources, you’ll finally be able to get a handle on what your capacity and demand is. This is what will enable you to finally know whether you can accept new work and then actually be able to schedule the work.”

“Your second question was whether it was safe to start your monitoring project. You already established it doesn’t require Brent. Furthermore, you say that the goal of this project is to prevent outages, which prevents Brent escalations. More than that, when outages do occur, you’ll need less of Brent’s time to troubleshoot and fix. You’ve already identified the constraint, exploited it to squeeze the most out of it, and then you’ve subordinated the flow of work to the constraint. So, how important is this monitoring project?”

“Properly elevating preventive work is at the heart of programs like Total Productive Maintenance, which has been embraced by the Lean Community. tpm insists that we do whatever it takes to assure machine availability by elevating maintenance. As one of my senseis would say, ‘Improving daily work is even more important than doing daily work.’ The Third Way is all about ensuring that we’re continually putting tension into the system, so that we’re continually reinforcing habits and improving something. Resilience engineering tells us that we should routinely inject faults into the system, doing them frequently, to make them less painful. “Sensei Mike Rother says that it almost doesn’t matter what you improve, as long as you’re improving something. Why? Because if you are not improving, entropy guarantees that you are actually getting worse, which ensures that there is no path to zero errors, zero work-related accidents, and zero loss.”

“Sensei Rother calls this the Improvement Kata,” he continues. “He used the word kata, because he understood that repetition creates habits, and habits are what enable mastery. Whether you’re talking about sports training, learning a musical instrument, or training in the Special Forces, nothing is more to mastery than practice and drills. Studies have shown that practicing five minutes daily is better than practicing once a week for three hours. And if you want to create a genuine culture of improvement, you must create those habits.”

attention from the work centers to all the space between the work centers. Just as important as throttling the release of work is managing the handoffs. The wait time for a given resource is the percentage that resource is busy, divided by the percentage that resource is idle. So, if a resource is fifty percent utilized, the wait time is 50/50, or 1 unit. If the resource is ninety percent utilized, the wait time is 90/10, or nine times longer. And if the resource is ninety-nine percent utilized?” Although I’m not quite understanding the relevance, I do the math in my head: 99/1. I say, “Ninety-nine.”

“When a resource is ninety-nine percent utilized, you have to wait ninety-nine times as long as if that resource is fifty percent utilized.” He gestures expansively, “A critical part of the Second Way is making wait times visible, so you know when your work spends days sitting in someone’s queue—or worse, when work has to go backward, because it doesn’t have all the parts or requires rework.

Remember that our goal is to maximize flow. Here at mrp-8, we had a situation many years ago where certain components were never showing up at final assembly on time. Was it because we didn’t have enough resources or because certain tasks were taking too long? “No! When we actually followed the parts around on the plant floor, we found that for the majority of time, the parts were just sitting in queues. In other words, the ‘touch time’ was a tiny fraction of ‘total process time.’ Our expediters had to search through mountains of work to find the parts and push them through the work center,”

When I catch up to him, I say, “Wait. Are you saying that audit issues aren’t important enough to fix?” “I never said that,” he says, stopping in his tracks and turning to face me. “You screw up something that jeopardizes the business’ ability to maintain compliance with relevant laws and regulations? You better fix it—or you should be fired.”

“Tell me. All those projects that Jimmy your ciso is pushing. Do they increase the flow of project work through the it organization?” “No,” I quickly answer, rushing to catch up again. “Do they increase operational stability or decrease the time required to detect and recover from outages or security breaches?” I think a bit longer. “Probably not. A lot of it is just more busywork, and in most cases, the work they’re asking for is risky and actually could cause outages.” “Do these projects increase Brent’s capacity?” I laugh humorlessly. “No, the opposite. The audit issues alone could tie up Brent for the next year.” “And what would doing all of Jimmy’s projects do to wip levels?” he asks, opening the door that takes us back into the stairwell. Exasperated, I say as we descend the two sets of stairs, “It would go through the roof. Again.” When we reach the bottom, Erik suddenly stops and asks, “Okay. These ‘security’ projects decrease your project throughput, which is the constraint for the entire business. And swamp the most constrained resource in your organization. And they don’t do squat for scalability, availability, survivability, sustainability, security, supportability, or the defensibility of the organization.” He asks deadpan, “So, genius: Do Jimmy’s projects sound like a good use of time to you?”

I look up at the sound of the audit partner closing his briefcase. He says to Dick, “Given this new data, it does appear that for the two potential material weaknesses, the it controls may indeed be out of scope and thus can be resolved very quickly. Thank you in advance for making yourselves available to get us the documentation we need to close out these issues as expeditiously as possible. “We will take all this under advisement and send you something in the next day or two,” he continues. “Most likely, we’ll want to schedule further testing of these newly documented downstream controls to make sure they were in place and operating—to support the financial statement assertions you’re making.” As he stands up, I stare in disbelief at the audit partner. We really dodged the bullet. Looking around the table, the Parts Unlimited team looks equally surprised.

One exception is Erik, who just nods approvingly, obviously irritated that it took so long to finally have the auditors on the run.

I’m about to get up to check on John when the audit partner shakes Dick’s hand and, to my surprise, Erik gets up to give him a hug. “Erik, it’s been a long time since gait and Orlando,” the audit partner says warmly. “I was sure our paths would cross again, but I never would have guessed it would be at a client engagement! What have you been up to lately?” Erik laughs and says, “Mostly, happily sailing on my boat. A friend asked me to join the Parts Unlimited board, partly due to their external auditors making trouble with a bunch of young, bottom-up auditors who strayed off the reservation. I should have known you’d be involved.” The audit partner looks genuinely embarrassed, and they huddle together, whispering. For the past five hours, John, Wes, and I sat on the sidelines while the business managers walked the auditors through a precise discussion about how the it control issues simply couldn’t lead to an undetected financial reporting error. They pulled out something called the “gait Principles” document and cited some of the enclosed flowcharts.

“You know what your problem is, Jimmy?” Erik says, pointing his finger at him. “You are like the political commissar who walks onto the plant floor, proudly flashing your badge at all the line workers, sadistically poking your nose in everybody’s business and intimidating them into doing your bidding, just to increase your own puny sense of self-worth. Half the time, you break more than you fix. Worse, you screw up the work schedules of everyone who’s actually doing important work.”

“Why, thank you for nothing, Mr. ciso,” Erik says, interrupting him. “As you just observed, the organization can keep the auditors away without you having to do anything at all. You are like the plumber who doesn’t even realize that you’re servicing an airplane, let alone the route you’re flying, or the business condition of the airline.”

I’m about to intervene on his behalf, when Erik stands up and shouts to John, “I don’t have anything further to say to you until you prove to me that you understand what just happened in this room. The business managed to dodge the sox-404 audit bullet, without any help from your team. Until you figure out how and why, you don’t have any business interfering with the daily operations of this organization. This should be your guiding principle: You win when you protect the organization without putting meaningless work into the it system. And you win even more when you can take meaningless work out of the it system.”

Erik says to me, pointing his thumb at John, “This guy is like the qa manager who has his group writing millions of new tests for a product we don’t even ship anymore and then files millions of bug reports for features that no longer exist. Obviously, he is making what you and I would call a ‘scoping error.’” John is shaking with outrage. He says, “How dare you! As a potential board director, I can’t believe you’re telling us to put our customer data and financial statements at risk!” Erik looks calmly back at John. “You really don’t get it, do you? The biggest risk to Parts Unlimited is going out of business. And you seem hell-bent on making it go out of business even faster, with all your ill-conceived, irrelevant technical minutia. No wonder you’ve been marginalized! Everyone else is at least trying to help the business survive. If this were an episode of Survivor, you’d have been voted off a long time ago!”

By now, Erik is standing over John. “Jimmy, Parts Unlimited has at least four of my family’s credit card numbers in your systems. I need you to protect that data. But you’ll never adequately protect it when the work product is already in production. You need to protect it in the processes that create the work product.”

Putting his hands in his pockets, he says more softly, “You want a clue? Go to mrp-8 plant and find the plant safety officer. Go talk to her, find out what she’s trying to accomplish and how she does it.” Erik’s expression brightens slightly and he adds, “And please convey my regards to her. I’ll be ready to talk with you again when Dick says he actually wants you around.”

carelessly. He’s been carrying it around for over two years. In front of where he was sitting is a single piece of paper, almost blank with a few lines scribbled on it. Wondering if it’s a suicide note or a resignation letter, I sneak a quick peek at what appears to be a poem. A haiku? Here I sit, hands tied Room angry, I could save them If only they knew

Patty wrinkles her forehead. “We know that it work can be projects or changes. And in many of the projects, there are many tasks or subprojects that show up over and over again. Like setting up a server. It’s recurring work. I guess you could call that a subassembly.”

Maybe the machine,” I speculate, “is the tools necessary to do the work? The virtualization management consoles, terminal sessions, and maybe the virtual disk space that we attach to

Patty shakes her head. “Maybe. The consoles and terminals sound like they could be the machine. And I think disk space, the applications, license keys, and so forth are all actually inputs or the raw materials needed to create the outputs.”

Next thing I know, I’m standing in Patty’s Change Coordination Room. I immediately spot on the back wall a new board. On it: index cards arranged in four rows. The rows are labeled “Move worker office,” “Add/change/delete account,” “Provision new desktop/laptop,” and “Reset password.” Each row has been divided up into three columns, labeled “Ready,” “Doing,” and “Done.” Interesting. This looks vaguely familiar. “What is this? Another change board?”

Patty breaks out into a grin and says, “It’s a kanban board. After our last meeting, I went to mrp-8 myself. I was so curious about this work center notion that I had to see it in action. I managed to find one of the supervisors that I’ve worked with before, and he spent an hour with me showing how they managed the flow of work.”

“If it’s not on the kanban board, it won’t get done,” she says. “And more importantly, if it is on the kanban board, it will get done quickly. You’d be amazed at how fast work is getting completed, because we’re limiting the work in process. Based on our experiments so far, I think we’re going to be able to predict lead times for work and get faster throughput than ever.”

I’m apparently fourteenth in line, with my laptop projected to arrive four days from now. “You actually believe this schedule?” I say, trying to be skeptical. However, it really would be fantastic if we could actually publish this to everyone, and be able to hit those dates.

She shakes her head. “You know, I did a quick poll of people we’ve issued laptops to. It usually takes fifteen turns to finally get them configured correctly. I’m tracking that now, and trying to drive this down to three. We’re putting in checklists everywhere, especially when we do handoffs within the team. It’s really making a difference. Error rates are way down.”

smile and say, “This is important. Getting executives and workers the tools they need to do their jobs is one of our primary responsibilities. I’m not saying I don’t believe you, but let’s keep these time estimates to ourselves for now. If you can generate a week’s track record of hitting the dates, then let’s start publishing this to all the requesters and their managers, okay?

“My plant supervisor friend also told me about the Improvement Kata they’ve adopted. Believe it or not, Erik helped them institute it many years ago. They have continual two-week improvement cycles, each requiring them to implement one small Plan-Do-Check-Act project to keep them marching toward the goal. You don’t mind that I’ve taken the liberty of adopting this practice in our group to keep us moving toward our own goals, right?”

She looks at the board and says, “Oh, I haven’t told you? We’re color-coding the cards to help us get ready for when we lift the project freeze. We’ve got to have some way to make sure we’re working on the most important things. So, the purple cards are the changes supporting one of the top five business projects, otherwise, they’re yellow. The green cards are for internal it improvement projects, and we’re experimenting with allocating twenty percent of our cycles just for those, as Erik recommended we do. At a glance, we can confirm that there’s the right balance of purple and green cards in work.” She continues, “The pink sticky notes indicate the cards that are blocked somehow, which we’re therefore reviewing twice a day. We’re also putting all these cards back into our change tracking tool, so we’re putting the change ids on each of the cards, too. It’s a bit tedious, but at least now part of the tracking is automated.”

She flips to the second page. “The projects seem to fall into the following categories: replacing fragile infrastructure, vendor upgrades, or supporting some internal business requirement. The rest are a hodgepodge of audit and security work, data center upgrade work, and so forth.” I look at the second list, scratching my head. Patty is right. How does one objectively decide whether “consolidating and upgrading e-mail server” is more or less important than “upgrading thirty-five instances of sql databases”?

“I don’t care how important everyone thinks their project is. We need to know whether it increases our capacity at our constraint, which is still Brent. Unless the project reduces his workload or enables someone else to take it over, maybe we shouldn’t even be doing it. On the other hand, if a project doesn’t even require Brent, there’s no reason we shouldn’t just do it.”

“We should make sure we have a project to replace or stabilize each one of these. And maybe we suspend indefinitely any infrastructure refresh project for anything that’s not fragile.” “Now hang on a minute,” Wes says. “Bill, you said it yourself. Preventive work is important, but it always gets deferred. We’ve been trying to do some of these projects for years! This is our chance to get caught up.” Patty says quickly, “Didn’t you hear what Erik told Bill? Improving something anywhere not at the constraint is an illusion. You know, no offense, but you sort of sound like John right now.”

While Wes and Patty are scribbling notes, I scan the list of internal projects again. “Hey, why is there a project for upgrading the bart database even though it’s going to be decommissioned next year?” Patty peers down at her list and then looks embarrassed. “Oh, jeez. I didn’t see that because we never reconciled the business and it projects with each other. We’re going to have to scrub the lists one more time to find dependencies like this. I’m sure there are others.” Patty thinks for a moment, “It’s strange. Even though we have so much data on projects, changes, and tickets, we’ve never organized and linked them all together this way before.

Two days later, I’m surprised to see a new laptop in my office. My old laptop has been disconnected and moved to the side. I look at my clipboard, flipping back to the laptop/desktop replacement schedule that Patty gave me earlier this week. Holy crap. Patty had promised laptop delivery for Friday, and I’m receiving it two days early. I log on to make sure it’s been configured properly. All the applications seem to be there, all my data have been transferred, e-mail is working, the network drives show up like before, and I can install new applications.

Patty grins. “Yep. Every single one of them. A couple of the early ones we delivered had a few configuration errors or were missing something. We’ve corrected it in the work instructions, and we seem to be batting one hundred percent delivering correct systems for the past two days.”

sheets of paper. “The task that Kirsten called about is delivering a test environment to qa. As she said, Brent estimated that it would take only forty-five minutes.” “Sounds about right,” Wes says. “You just need to create a new virtualized server and then install the os and a couple of packages on it. He probably even doubled the time estimate to be safe.” “That’s what I thought, too,” Patty said, but she’s shaking her head. “Except it’s not just one task. What Brent signed up for is more like a small project—there’s over twenty steps involving at least six different teams! You need the os and all the software packages, license keys, dedicated ip address, special user accounts set up, mount points configured, and then you need the ip addresses to be added to an acl list on some file server. In this particular case, the requirements say that we need a physical server, so we also need a router port, cabling, and a server rack where we have enough space.” “Oh…,” Wes says, sounding exasperated, reading what Patty is pointing at. He mumbles, “Physical servers are such a pain in the ass.”

“You’re missing the point. This would still be happening, even if it were virtualized,” Patty says. “First, Brent’s ‘task’ turns out to be considerably more than just a task. Second, we’re finding that it’s multiple tasks spanning multiple people, each of whom have their own urgent work to do. We’re losing days at each handoff. At this rate, without some dramatic intervention, it’ll be weeks before qa gets what they need.”

I nod, knowing exactly what Wes is referring to. The lead time for firewall changes has become legendary. Wait. Didn’t Erik mention something like this? For a firewall change, even though the work only required thirty seconds of touch time, it still took four weeks of clock time. That’s just a microcosm of what’s happening with Brent. But what’s happening to us right now is much, much worse, because there are handoffs.

I tell them what Erik told me at mrp-8, about how wait times depend upon resource utilization. “The wait time is the ‘percentage of time busy’ divided by the ‘percentage of time idle.’ In other words, if a resource is fifty percent busy, then it’s fifty percent idle. The wait time is fifty percent divided by fifty percent, so one unit of time. Let’s call it one hour. So, on average, our task would wait in the queue for one hour before it gets worked. “On the other hand, if a resource is ninety percent busy, the wait time is ‘ninety percent divided by ten percent’, or nine hours. In other words, our task would wait in queue nine times longer than if the resource were fifty percent idle.”

I conclude, “So, for the Phoenix task, assuming we have seven handoffs, and that each of those resources is busy ninety percent of the time, the tasks would spend in queue a total of nine hours times the seven steps…” “What? Sixty-three hours, just in queue time?” Wes says, incredulously. “That’s impossible!” Patty says with a smirk, “Oh, of course. Because it’s only thirty seconds of typing, right?”

Creating and prioritizing work inside a department is hard. Managing work among departments must be at least ten times more difficult. Patty says, “What that graph says is that everyone needs idle time, or slack time. If no one has slack time, wip gets stuck in the system. Or more specifically, stuck in queues, just waiting.”

As we digest this, Patty continues. “Each of those sheets of paper on the board is like this Phoenix ‘task,’” she says, making air quotes with her hands. “It looks like a single person task, but it’s not. It’s actually multiple steps with multiple handoffs among multiple people. No wonder Kirsten’s project estimates are off. “We need to correct this on Kirsten’s schedule and her work breakdown structure, or wbs. Based on what I’ve seen, fully one-third of our commitments to Kirsten fall into this category.”

Patty says, “I wonder if we could create a kanban lane for each of these ‘tasks?’” “Yes, that’s it,” I say. “Erik was right. You’ve just found a big pile of recurring work! If we can document and standardize this recurring work, and gain some mastery over it, just like you did with laptop replacement, I’m sure we can improve flow!” I add, “You know, if we can standardize all our recurring deployment work, we’ll finally be able to enforce uniformity of our production configurations. That would be our infrastructure snowflake problem—you know—no two alike. How Brent turned into Brent is that we allowed him to build infrastructure only he can understand. We can’t let that happen again.”

Over the next forty-five minutes, we create our plan. Patty is going to work with Wes’ team to assemble the top twenty most frequently recurring tasks. She will also figure out how to better manage and control tasks when they are queued. Patty proposes a new role, a combination of a project manager and expediter. Instead of day-by-day oversight, they would provide minute-by-minute control. She says, “We need fast and effective handoffs of any completed work to the next work center. If necessary, this person will wait at the work center until the work is completed and carry to the next work center. We’ll never let critical work get lost in a pile of tickets again.” “What? Someone assigned to carry around tasks from person to person, like a waiter?” Wes asks in disbelief.

“At mrp-8, they have a ‘water spider’ role that does exactly that,” she counters. “Almost all of this latest Phoenix delay was due to tasks waiting in queues or handoffs. This will make sure it doesn’t happen again. “Eventually,” she adds, “I’ll want to move all the kanbans, so that we don’t need a person acting as the signaling mechanism for work handoffs. Don’t worry. I’ll have it figured out in a couple of days.”

take a deep breath, trying to decide what to tell him. A friend told me years ago, “To tell the truth is an act of love. To withhold the truth is an act of hate. Or worse, apathy

Finally, I say, “Look, John. You’re a good guy, and I know your heart is in the right place, but up until you helped hide us from the pci auditors during the Phoenix meltdown, I would have said no. I know that’s not what you want to hear, but…I wanted to make sure that I wasn’t feeding you a line of bullshit.”

With a very small smile, he adds, “Want to hear something funny? People say that I’m more approachable than Steve! Steve’s incredibly charismatic, and let’s face it, I’m an asshole. But when people have concerns, they don’t want to have their minds changed. They want someone to listen to them and help make sure Steve gets the message.”

A good day feels like my staff panicking about the size of the commission checks we’re going to be writing them.

A bad day is like the one we had two weeks ago,” he says, now looking frustrated and even angry. “We can’t close the quarter because of some it failure, we can’t seem to execute on our most critical project to close the gap with the competition, we keep losing customers, the auditors are making noises about some restatement project, and the board of directors is debating whether to fire all of us because we’re such screwups

“What are your goals, objectives, and measurements for this year?”

CFO GOALS Health of company Revenue Market share Average order size Profitability Return on assets Health of Finance Order to cash cycle Accounts receivable Accurate and timely financial reporting Borrowing costs

After all, the best accounts receivables team on the planet can’t save us if we’re in the wrong market with the wrong product strategy with an r&d team that can’t deliver.”

As I’m pondering this, Dick then points to the second slide, saying, “So that’s what’s on the second slide, which shows what I believe are the more important company goals. I look at this slide every day.” Are we competitive? Understanding customer needs and wants: Do we know what to build? Product portfolio: Do we have the right products? r&d effectiveness: Can we build it effectively? Time to market: Can we ship it soon enough to matter? Sales pipeline: Can we convert products to interested prospects? Are we effective? Customer on-time delivery: Are customers getting what we promised them? Customer retention: Are we gaining or losing customers? Sales forecast accuracy: Can we factor this into our sales planning process?

As the elevator doors open, John says to me, “You know, I think there’s something very similar about us dodging the sox-404 audit bullet and what’s on Dick’s second slide,” he says. “I can’t quite put my finger on it, but I think there’s something here that we need to understand better.”

“Are you available later today to meet? Let’s see if we can connect all these dots. I suspect that we’re sitting on the missing link that might explain both why the company keeps missing its goals as well as why it keeps being undervalued.” “Absolutely,” he says excitedly.

No wonder Dick just has a vague sense that it is screwing up—it’s a dull, throbbing ache that he can’t localize. Our next step is obvious: We must make those pains very specific and visible to convince Dick that it is capable of not just screwing up less often but helping all of the business win. This is too important and urgent to struggle blindly in the dark, and I need to call Erik for advice. Standing in the lobby of Building 2, I speed-dial him.

“As part of the First Way, you must gain a true understanding of the business system that it operates in. W. Edwards Deming called this ‘appreciation for the system.’ When it comes to it, you face two difficulties: On the one hand, in Dick’s second slide, you now see that there are organizational commitments that it is responsible for helping uphold and protect that no one has verbalized precisely yet. On the other hand, John has discovered that some it controls he holds near and dear aren’t needed, because other parts of the organization are adequately mitigating those risks.

“This is all about scoping what really matters inside of it. And like when Mr. Sphere told everyone in Flatland, you must leave the realm of it to discover where the business relies on it to achieve its goals.” I hear him continue, “Your mission is twofold: You must find where you’ve under-scoped it—where certain portions of the processes and technology you manage actively jeopardizes the achievement of business goals—as codified by Dick’s measurements. And secondly, John must find where he’s over-scoped it, such as all those sox-404 it controls that weren’t necessary to detect material errors in the financial statements.

“You may think that we’re mixing apples and oranges, but I assure you that we are not,” he continues. “Some of the wisest auditors say that there are only three internal control objectives: to gain assurance for reliability of financial reporting, compliance with laws and regulations, and efficiency and effectiveness of operations. That’s it. What you and John are talking about are just different slides of what is called the ‘coso Cube.’”

“Here’s what you and John need to do: Go talk to the business process owners for the objectives on Dick’s second slide. Find out what their exact roles are, what business processes underpin their goals, and then get from them the top list of things that jeopardize those goals.

“You must understand the value chains required to achieve each of Dick’s goals, including the ones that aren’t so visible, like those in it. For instance, if you were a cross-country freight shipping company that delivers packages using a fleet of one hundred trucks, one of your corporate goals would be customer satisfaction and on-time delivery.” I hear him continue, “Everybody knows that one factor jeopardizing on-time delivery is vehicle breakdowns. A key causal factor for vehicle breakdowns is failure to change the oil. So, to mitigate that risk, you’d create an sla for vehicle operations to change the oil every five thousand miles.” Obviously enjoying himself, he keeps explaining, “Our organizational key performance indicator (kpi) is on-time delivery. So to achieve it, you would create a new forward-looking kpi of, say, the percentage of vehicles that have had their required oil changes performed.

“After all, if only fifty percent of our vehicles are complying with the required maintenance policies, it’s a good bet that in the near future, our on-time delivery kpis are going to take a dive, when trucks start getting stranded on the side of the road, along with all the packages they’re carrying. “People think that just because it doesn’t use motor oil and carry physical packages that it doesn’t need preventive maintenance,” Erik says, chuckling to himself. “That somehow, because the work and the cargo that it carries are invisible, you just need to sprinkle more magic dust on the computers to get them running again.

“Metaphors like oil changes help people make that connection. Preventive oil changes and vehicle maintenance policies are like preventive vendor patches and change management policies. By showing how it risks jeopardize business performance measures, you can start making better business decisions.

“Make sure John fulfills his mission. He must talk with the finance side of the sox-404 audit team. He must learn exactly how the business managed to dodge the last audit bullet and what the actual control environment looks like and where reliance is really placed. And he must then explain it to you. “You’ll be ready for your meeting with Dick when you’ve built out the value chains, linking his objectives to how it jeopardizes it. Assembling concrete examples of how it issues have jeopardized those goals in the past. Make sure you’re prepared.” And with that, he says, “In fact, feel free to invite me to that meeting. I want to see Dick’s face when you present what you learn,”

When everyone is here, I quickly share what I had learned from Erik. We decide that Patty and I will start the business process-owner interviews for “understanding customer needs and wants,” “product portfolio,” “time to market,” and “sales pipeline,” while John will research the business sox-404 control environment, as directed by Erik.

There’s at least a predictable funnel that comes from marketing campaigns, generating prospects, leads, qualified leads, and sales opportunities that leads to a sales pipeline. One sales person missing their number rarely jeopardizes the entire department.

On the other hand, any of my engineers can get me fired by making a seemingly small, harmless change that results in a crippling, enterprise-wide outage.

“You’re listed as the owner of the ‘sales pipeline’ and ‘sales forecast accuracy’ measures on Dick’s spreadsheet. What can you tell me about the challenges of hitting those numbers?” “Look, I don’t know much about it. Someone on my staff might be better for you to talk to,” he responds. “Don’t worry, I’m not asking about anything it-related. Let’s just talk about your measures,” I assure him.

“Okay, it’s your nickel…” he says. “If you want to talk about sales forecast accuracy, you first need to know why it’s so inaccurate. It starts when Steve and Dick hand me a crazy revenue target, leaving me to figure out how to deliver on it. For years, I’ve had to assign way too much quota capacity to my team, so of course we keep missing our numbers! I tell Steve and Dick this, year after year, but they don’t listen. Probably because they’re having some arbitrary revenue target jammed down their throats by the board.

“It’s a crappy way to run a company. It demoralizes my team, and my top performers are quitting in droves. Of course, we’ll replace them, but it takes at least a year for replacements to perform at full quota capacity. Even in this lousy economy, it takes too long to find qualified sales people. “You know what chaps my hide?” he continues. “Sarah promised that acquiring the retail stores would accelerate our sales. And has it happened? Hell, no! “We’re completely screwing up the execution. This morning, a district manager was screaming that they need truckloads of our new fuel injector kits because all his stores are completely stocked out. We’re losing the easiest sales we can make! Our customers want to buy, but they’re walking out empty-handed, probably buying something crappier from one of our competitors.” Ron says angrily, “We are clueless about what our customers want! We have too much product that will never sell and never enough of the ones that do.”

“A bad day?” he repeats, staring at me disapprovingly. “Why, Bill, it’s positively catastrophic when the mrp and phone systems that you manage go down like they did a few weeks ago. For just the mrp outage, we had customers screaming about delayed orders, and two of them canceled a quarter-million dollars of orders outright. We’re scrambling to keep some of our best customers from putting $1.5 million of contracts up for rebid.” He leans across his desk. “And when the phones went down in the last few days of the quarter, and customers couldn’t give us orders or make last-minute changes! That’s delayed another $1.5 million of orders, and ten customers are reevaluating their contracts, putting another $5 million of contracts at risk.” “You’re making my job much, much harder, pal,” he says. “A lot of my sales people missed their quota by the tiniest fractions, due to things totally outside of their control. To keep morale up, I’m demanding Steve give quota credit for any order that was delayed because of our screwups.”

The trouble is, we need to expand the licensing beyond what Information Security has deployed, and emergency capital is hard to get these days. Especially for it Operations.” Ron turns bright red. “Why? What are they saving it for? Probably another harebrained acquisition that Sarah is dreaming up.” He laughs humorlessly. “How much money are we talking about?” When I tell him, he looks disgusted. “We spend more money watering the lawns at the manufacturing plants every week! Dick is going to hear from me about this. If he’s not willing to spend money, we may lose orders—even if your project is just insurance so we can collect on all the hard work my sales team does—it’s a no-brainer!” “We sure think so. Thanks for the support,” I say. “We’re about out of time. Are there any other challenges or impediments that we can help with?”

In describing her responsibilities, she summarizes, “Ultimately, the way I measure our understanding of customer needs and wants is whether customers would recommend us to their friends. Any way you cut it, our metrics aren’t very good.” When I ask why, she sighs. “Most of the time, we’re flying blind. Ideally, our sales data would tell us what customers want. You’d think that with all the data in our order entry and inventory management systems, we could do this. But we can’t, because the data are almost always wrong.”

“Our data quality is so bad that we can’t rely on it to do any sort of forecasting. The best data that we have right now comes from interviewing our store managers every two months and the customer focus groups we do twice a year. You can’t run a billion-dollar business this way and expect to succeed!

“At my last job, we received sales and stockout reports every day,” she continues. “Here, we get them once a month from Finance, but they’re full of errors. What do you expect? They’re done by a bunch of college interns, copying and pasting numbers between a million spreadsheets.” “If you could wave a magic wand, what would you do instead?” I ask. “How big of a magic wand?” she asks. “It can do anything you want,” I reply, smiling. “That’s a big magic wand,” she says, laughing. “I want accurate and timely order information from our stores and online channels. I want to press a button and get it, instead of running it through the circus we’ve created. I’d use that data to create marketing campaigns that continually do a/b testing of offers, finding the ones that our customers jump at. When we find out what works, we’d replicate it across our entire customer list. By doing this, we’d be creating a huge and predictable sales funnel for Ron. “I’d use that information to drive our production schedule, so we can manage our supply and demand curves. We’d keep the right products on the right store shelves and keep them stocked. Our revenue per customer would go through the roof. Our average order sizes would go up. We’d finally increase our market share and start beating the competition again.” As she’s telling us this, she looks animated and excited. Then her exuberance disappears. Sounding defeated, she says, “But we’re stuck with the systems we have, unfortunately.”

“All we’ve gotten from Phoenix is a bunch of promises. It was supposed to do a lot of this reporting, but there’s so much political pressure to ship something, they keep dropping features. And guess which feature they’re delaying until sometime next year?” She rolls her eyes in disbelief. “Reporting?” I guess, fearing the worst. When Maggie nods, I try to stay upbeat. “For now, let’s assume that the magic wand worked. We now have great data coming out of the stores. You’re keeping the right products in the stores, and the campaigns you’re dreaming up are succeeding beyond your wildest dreams. What then?” “Life gets exciting, that’s what!” she says, her eyes lighting up. “Last year, we shipped a custom fuel injection system for a new upcoming sports car. We had just six months to get this to market, from the drawing board to the shelves. We nailed it! The designers, r&d folks, and Marketing all kicked ass. We had the right product, right place, right brand, right price, and right quality. It was one of the best-selling products of the year.

“We took a risk and won big,” she says. “If we had better visibility into our retail operations, with our amazing r&d and manufacturing capabilities, we could make fifty of those bets per year. I’d bet four of those would be runaway hits! We’d not only be profitable, we’d be insanely profitable.”

Patty interjects, “What’s an acceptable time to market for your products?” She responds quickly, “These days? Products need to ship in six months. Nine months, tops. Otherwise, some Chinese company will steal our idea, have them on our competitors store shelves, and take the majority of the market. “In these competitive times, the name of the game is quick time to market and to fail fast. We just can’t have multiyear product development timelines, waiting until the end to figure out whether we have a winner or loser on our hands. We need short and quick cycle times to continually integrate feedback from the marketplace.

But that’s just half the picture,” she continues. “The longer the product development cycle, the longer the company capital is locked up and not giving us a return. Dick expects that on average, our r&d investments return more than ten percent. That’s the internal hurdle rate. If we don’t beat the hurdle rate, the company capital would have been better spent being invested in the stock market or gambled on racehorses. “When r&d capital is locked up as wip for more than a year, not returning cash back to the business, it becomes almost impossible to pay back the business,”

Holy crap. Maggie is starting to sound eerily like Erik, too. The need to continually reduce cycle times is part of the First Way. The need for amplification of feedback loops, ideally from the customer, is part of the Second Way.

But nine months to return cash back to the company, tops? We’ve been at Phoenix for nearly three years and it still hasn’t created the desired business value.

“Our planning horizon is six to twelve months. How does anyone know what projects they should be working on three years from now?” she says angrily, suddenly reminding me of Ron.

She shares a bunch of ideas about hiring more it people, dedicating it people to her group, putting more scrutiny on the projects that are clogging the it project queue, and so forth. Most of the ideas aren’t new, and I merely raise my eyebrows at the notion of a larger it budget. Steve and Dick will never go for that.

“Incredible!” Patty exclaims, as we leave Maggie’s office. “I can’t believe how frustrated Maggie and Ron are. Can you believe that unreliable data in the order entry and inventory management systems came up again? And I can’t believe that Phoenix, as it’s currently designed, won’t actually fix the data quality problems!” I nod and say decisively, “Get a meeting together with John and Wes. We’ll show them what we’ve learned so far. Invite Chris, as well. This goes beyond just it Operations. This may change how we prioritize and develop our applications, too.”

We’ve spent over $20 million on Phoenix over three years. With all that wip and capital locked inside the project, it will likely never clear the ten percent internal hurdle rate. In other words, Phoenix should not have been approved.

Pointing at the whiteboard, I say, “The first column names the business capabilities and processes needed to achieve Dick’s desired outcomes; the second column lists the it systems that those business processes rely upon; the third column lists what can go wrong with either the it systems or data; and in the fourth column, we’ll write down the countermeasures to prevent those bad things from happening, or at a minimum, detect and respond.”

“But what’s the solution?” I say, “Anyone have any ideas?” Surprisingly, John speaks up first. “Seems pretty obvious to me. We need to come with the controls to mitigate the risks in your third column. We then show this table to Ron and Maggie, and make sure they believe that our countermeasures help them achieve their objectives. If they buy it, we work with them to integrate it into their performance measures… “That example Erik gave you is perfect. They integrated ‘compliance with vehicle maintenance procedures’ as a leading indicator for ‘on-time delivery’ and ‘customer retention.’ We need to do the same.”

“So, what’s causing the data integrity issues?” Chris snorts in disgust. “Phoenix fixes a bunch of them, but we still have issues. Most of them are caused upstream, because the Marketing people keep putting in malformed inventory skus. Marketing needs to get their crap together, too.” So for ‘marketing needs and wants,’ our proposed measurements include ability for Phoenix to support weekly and eventually daily reporting, percentage of valid skus created by Marketing, and so forth.

Over the next day, Patty and I get great feedback from Ron and Maggie, and they commit to supporting our proposal with Dick. When Ron learns that we still haven’t been granted budget for our monitoring project, he calls up Dick right in front of us, leaving him a heated voicemail, demanding to know why he’s dragging his feet. With all this enthusiastic support, I figure our Thursday meeting with Dick will be a slam dunk.

“All you’ve told me is that you’re completely asleep at the wheel!” Dick says sternly, obviously unimpressed by what I’ve presented. Suddenly, I’m reminded of how Steve didn’t even look at the spreadsheets I had prepared for him when I asked him to prioritize Phoenix and the audit finding work. But Dick isn’t being dismissive. He’s actually angry. “You’re telling me something a nutless monkey could have figured out. You didn’t know these measurements were important? At every town hall meeting, Steve repeats them over and over. It’s in our company newsletters, it’s what Sarah talks about in every one of her strategy briefings. How could you all miss something so important?”

Erik clears his throat, and says to Dick, “I agree a nutless monkey should have figured this out. So, Dick, explain why on that little measurement spreadsheet of yours, you list four levels of management for each of your measurements but nowhere are there any it managers listed. Why?”

Not waiting for Dick to respond, he continues, “Every week, it people get dragged into fire drills at the last minute by managers trying to achieve those measurements—just like Brent was pulled in to help launch Sarah’s latest sales promotion.” Erik pauses and says, “Quite frankly, I think you’re just as much of a nutless monkey as Bill.”

Dick turns back to me. “Bill, you’re telling me that everyone could do everything right in the business, but because of these it issues, we would all still miss these objectives?” “Yes, sir,” I say. “The operational risks posed by it need to be managed just like any other business risk. In other words, they’re not it risks. They’re business risks.” Again, Dick grunts. He slumps in his chair, rubbing his eyes. “Shit. How the hell are we supposed to write an it outsourcing contract if we don’t even know what the business needs?” he says, slamming his hand on the table. He then asks, “Well, what’s your proposal? You’ve got one, I presume?” I sit upright and begin the pitch that I’ve rehearsed so many times with the team. “I’d like three weeks with each of the business process owners on that spreadsheet. We need to get the business risks posed by it better defined and agreed upon and then propose to you a way to integrate those risks into leading indicators of performance. Our goal is not just to improve business performance but to get earlier indicators of whether we’re going to achieve them or not, so we can take appropriate action. “Furthermore,” I continue. “I’d like to schedule a single topic meeting with you and Chris about Phoenix,” then explaining my concerns how Phoenix as defined should not even have been approved.

I continue, “We’re going way too slowly, with too much wip and too many features in flight. We need to make our releases smaller and shorter and deliver cash back faster, so we can beat the internal hurdle rate. Chris and I have some ideas, but it will look very different than our current plan of record.” He remains silent. Then decisively, he declares, “Yes on both of your proposals. I’ll assign Ann to help. You need the best talent in the company.” Out of the corner of my eye, I see Chris and Patty smile. “Thank you, sir. We’ll make it so,” I say, standing up and pushing everyone out of the room, before Dick changes his mind. As we walk out of his office, Erik claps his hand on my shoulder. “Not bad, kid. Congratulations on being well on your way to mastering the First Way. Now help John get there, because you’re going to have your hands full tackling the Second Way.” Confused, I ask, “Why? What’s going to happen?” “You’ll find out soon enough,”

He turns toward Wes and Patty. “I needed to figure out how we escaped all the findings from the internal and external auditors. At first, I thought it was just the audit partner bending over backward to retain us as a client. But that wasn’t it at all… “I got in front of everyone from the Parts Unlimited team who was at that meeting, trying to figure out who had the magic bullet. To my surprise, it wasn’t Dick or our corporate counsel. Ten meetings later, I finally found Faye, a Financial Analyst who works for Ann in Finance. “Faye has a technical background. She spent four years in it,” he says, as he hands out papers to each of us. “She created these sox-404 control documents for the finance team. It shows the end-to-end information flow for the main business processes in each financially significant account. She documented where money or assets entered the system and traced it all the way to the general ledger. “This is pretty standard, but she took it one step further: She didn’t look at any of the it systems until she understood exactly where in the process material errors could occur and where they would be detected. She found that most of the time, we would detect it in a manual reconciliation step where account balances and values from one source were compared to another, usually on a weekly basis. “When this happens,” he says, with awe and wonder in his voice, “she knew the upstream it systems should be out of scope of the audit.” “Here’s what she showed the auditors,” John says, excitedly flipping to the second page. “Quote: ‘The control being relied upon to detect material errors is the manual reconciliation step, not in the upstream it systems.’ I went through all of Faye’s papers, and in every case, the auditors agreed, withdrawing their it finding. “That’s why Erik called the pile of audit findings a ‘scoping error.’ He’s right. If the audit test plan was scoped correctly in the beginning, there wouldn’t have been any it findings!”

“I’m rebuilding our compliance program from scratch, based upon our new understanding of precisely where we’re relying our controls,” John says. “That dictates what matters. It’s like having a magic set of glasses that can differentiate what controls are earth-shatteringly important versus those that have no value at all.” “Yes!” I say. “Those ‘magic glasses’ helped us finally see what matters to Dick for company operations. It was right in front of us for years, but we never saw it.” John nods and smiles broadly. He flips to the last page of the handout. “I’m proposing five things that could reduce our security-related workload by seventy-five percent.”

When he verbalizes so precisely why it’s safe to do, I realize that John too is also mastering the First Way, having truly achieved a “profound appreciation of the system.”

His second proposal requires that we find out how production vulnerabilities got there in the first place and that we ensure that they don’t happen again by modifying our deployment processes. His third proposal requires that we flag all the systems in the scope for compliance audits in Patty’s change management process—so we can avoid changes that could jeopardize our audits—and that we create the on-going documentation that the auditors will ask for. John looks around, seeing all of us staring at him in shocked silence. “Did I say something wrong?” “No offense, John…” Wes says slowly. “But…uh… You feeling okay?” I say, “John, I don’t think you’ll get any objections from my team on your proposals. I think they’re great ideas.” Wes and Patty vehemently nod in agreement.

“My fourth proposal is to reduce the size of our pci compliance program by getting rid of anything that stores or processes cardholder data, which is like toxic waste. Losing or mishandling it can be lethal, and it costs too much to protect. “Let’s start with the asinine cafeteria point of sale system. I never want to do another security review of that piece of crap. Frankly, I don’t care who takes it, even if it’s Sarah’s cousin Vinnie. It’s gotta go.”

“Okay, John, let’s do it. You’re batting four out of four,” I say, decisively. “But do you really think we can get rid of it in time to make a difference?” “Yep,” John says, smiling confidently. “I’ve already talked with Dick and the legal team. We just need to find a suitable outsourcer and convince ourselves that they can be trusted to maintain and secure the systems and data. We can outsource the work but not the responsibility.”

“My fifth and last proposal is that we pay down all the technical debt in Phoenix, using all the time we’ve saved from my previous proposals. We know there’s a huge amount of risk in Phoenix: strategic risk, operational risk, huge security and compliance risk. Almost all of Dick’s key measures hinge on it.

We quickly agree to pair up people in Wes’ and Chris’ group with John’s team, so that we can increase the bench of security expertise. By doing this, we will start integrating security into all of our daily work, no longer securing things after they’re deployed.

The insight we’ve gained from that first strange meeting with Dick and John tells me that we’re hot on the trail of understanding how we can really help the business win. Opening up my e-mail, I see a note from Kirsten. All her project managers are gushing about how projects are flowing so much faster. The number of tasks waiting for Brent and the rest of it Operations is way down. In fact, if I’m reading the report correctly, Brent is almost caught up.

When the inevitable outages and incidents do occur, we’re operating like a well-oiled machine. We’re building a body of tribal knowledge that’s helping us fix things faster than ever, and, when we do need to escalate, it’s controlled and orderly.

Because of our ever-improving production monitoring of the infrastructure and applications, more often than not, we know about the incidents before the business does.

We’ve cut a bunch of unneeded security projects from our audit preparation and remediation work, replacing them with preventive security projects that my entire team is helping with.

Our change management meetings are going more smoothly and regularly than ever. We not only have visibility into what our teams are doing, but work is really flowing.

More than ever, people know exactly what they should be working on. People are getting satisfaction out of fixing things. I’m hearing that people are feeling happier and more upbeat, because they can actually do their jobs.

Patty’s experiments with establishing kanbans around Brent are a success. We’re also finding instances of work going backward to Brent, because we didn’t understand or didn’t sufficiently specify some task or outcome, requiring Brent to translate or fix it. When this happens now, we quickly jump on it to make sure that it doesn’t happen again.

Against my staff’s expectations, we keep bumping up the number of projects we think we can handle concurrently. Because we have a better idea of what our flows of work are, and managing carefully which ones are allowed to go to Brent, we’re finding that we can keep releasing more projects without impacting our existing commitments.

Erik says that we are starting to master the First Way: We’re curbing the handoffs of defects to downstream work centers, managing the flow of work, setting the tempo by our constraints, and, based on our results from audit and from Dick, we’re understanding better than we ever have what is important versus what is not.

As Erik keeps reminding me, a great team performs best when they practice. Practice creates habits, and habits create mastery of any process or skill. Whether it’s calisthenics, sports training, playing a musical instrument, or in my experience, the endless drilling we did in the Marines. Repetition, especially for things that require teamwork, creates trust and transparency.

Last week, as I sat through our latest biweekly outage drill, I was very impressed. We were getting very good at this. I feel certain that if the payroll failure that happened on my first day of the job happened now, we could complete the entire payroll run—not just the salaried staff, but the hourly staff, as well.

John quickly got the approval from Dick and Steve to have an outsourcer to take over the cafeteria pos systems and replace it with something commercially supported.

It seems to me that if anyone is managing it without talking about the Three Ways, they are managing it on dangerously faulty assumptions.

Some people are going around the project freeze. You have time to talk?” “Let’s meet in ten minutes,” I say. “Don’t leave me hanging. Who’s trying to backdoor the system?” I hear John laugh on the other end of the line. “Sarah. Who else?”

John starts presenting what he found. Sarah’s group has four instances of using outside vendors and online services. Two are relatively innocuous but the others are more serious: she has contracted a vendor for a $200,000 project to do customer data mining and another vendor to plug into all our pos systems to get sales data for customer analytics. “The first problem is that both projects violate the data privacy policy that we’ve given our customers,” John says. “We repeatedly promise that we will not share data with partners. Whether we change that policy or not is, of course, a business decision. But make no mistake, if we go ahead with the customer data mining initiative, we’re out of compliance with our own privacy policy. We may even be breaking several state privacy regulations that expose us to some liability.”

This doesn’t sound good, but John’s tone of voice suggests there’s worse to come. “The second problem is that Sarah’s vendor uses the same database technology that we used for our cafeteria pos system, which we know is virtually impossible to secure and maintain support for in production, if and when it becomes a part of daily operations.” I feel my face get red hot. It’s not just about another cafeteria pos system that we’ll need to retrofit for production. It’s because applications like this contribute to our inaccurate sales order entry and inventory management data. We have too many cooks in the kitchen and no one accountable for maintaining the integrity of the data.

“Look, I don’t care about Sarah’s project management and invoicing tools—if it makes them more productive, let them use it,” I say. “It’s probably safe as long as it doesn’t interface with an existing business system, store confidential data, affect financial reporting, or whatever. But if it does, then we need to be involved and at least confirm that it doesn’t impact any of our existing commitments.” “I agree,” John says, “Want me to take the first stab at that outsourced it service policy document?”

“Perfect,” I say. But with less certainty, I continue, “Although, what’s the right way to handle Sarah? I feel completely out of my league. Steve constantly protects her. How do we convey to him the potential mayhem she’s causing with her unauthorized projects?” Making sure John’s office door is closed, I say to John and Patty “Guys, help me out. What does Steve see in her? How does she get away with so much crap? Over the past couple of weeks, I see how hard-nosed Steve can be, but Sarah routinely gets away with murder. Why?”

“Steve prides himself on being an operations guy, and he’s admitted several times in company meetings that he doesn’t have a flair for strategy. I think that’s why he loved working with his old boss and our new chairman Bob so much. For a decade, Bob was the strategy guy, and all Steve had to do was execute the vision. “For years, Steve searched for a strategy person to be his right-hand man. He went through quite a few people, even setting a couple executives against each other in this awful, drawn out competition. Pretty Machiavellian,” she continues. “And Sarah won. The word on the street was that there was a lot of backstabbing and underhanded tactics, but I suppose that’s what it takes to come out on top. Evidently, she has mastered how to whisper the right things in his ear, reinforcing his paranoia and aspirations.”

She continues, “Steve has a blind spot for Sarah’s shortcomings, because she has something he needs and admires, which is the ability to come up with creative strategies, regardless of whether the strategy is good or bad. On the other hand, because Sarah is so insecure, she’ll do whatever it takes to not look bad.

“She simply doesn’t care about the body count she leaves in her wake, because she wants to be the next ceo of Parts Unlimited.” Patty says. “And apparently, Steve does too. He’s been grooming her as his successor for years.”

“When we’re out of the woods,” I say, “we’ve got to figure out how to prevent this from happening again. There should be absolutely no way that the Dev and qa environments don’t match the production environment.”

To me, the answer to your problem is obvious. The First Way is all about controlling the flow of work from Development to it Operations. You’ve improved flow by freezing and throttling the project releases, but your batch sizes are still way too large. The deployment failure on Friday is proof. You also have way too much wip still trapped inside the plant, and the worst kind, too. Your deployments are causing unplanned recovery work downstream.”

“Now you must prove that you can master the Second Way, creating constant feedback loops from it Operations back into Development, designing quality into the product at the earliest stages. To do that, you can’t have nine-month-long releases. You need much faster feedback.

“You’ll never hit the target you’re aiming at if you can fire the cannon only once every nine months. Stop thinking about Civil War era cannons. Think antiaircraft guns.”

“In any system of work, the theoretical ideal is single-piece flow, which maximizes throughput and minimizes variance. You get there by continually reducing batch sizes. “You’re doing the exact opposite by lengthening the Phoenix release intervals and increasing the number of features in each release. You’ve even lost the ability to control variance from one release to the next.”

“That’s ridiculous, given all the investments you’ve made virtualizing your production systems. You still do deployments like they’re physical servers. As Sensei Goldratt would say, you’ve deployed an amazing technology, but because you haven’t changed the way you work, you haven’t actually diminished a limitation.”

“Think like a plant manager. When you see work going upstream, what does it mean to you?” He quickly responds, “The flow of work should ideally go in one direction only: forward. When I see work going backward, I think ‘waste.’ It might be because of defects, lack of specification, or rework… Regardless, it’s something we should fix.”

“The flow of work goes in one direction only: forward. Create a system of work in it that does that. Remember, the goal is single-piece flow.”

“Incidentally, this will also solve the problem that you’ve been fretting about with Dick. An inevitable consequence of long release cycles is that you’ll never hit the internal rate of return targets, once you factor in the cost of labor. You must have faster cycle times. If Phoenix is preventing you from doing that, then figure out how to deliver the features some other way.

“To do this, you’ll need to put Brent at the very front of the line, just like Herbie in The Goal. Brent needs to be working at the earliest stages of the development process. Bill, you of all people should be able to figure this out.

“It may be ‘impossible’ to you, but look at it from my perspective,” Chris says levelly. “I need my developers building new features. They can’t be constantly tied up with Bill’s team, dealing with deployment issues.”

“But we still need to figure out how to achieve what Steve needs,” I continue. “As Erik suggested, if we can’t do that inside the Phoenix framework, maybe we can do it outside of Phoenix. I propose we form a swat team by detaching a small squad from the main Phoenix team, telling them to figure out what features can help us hit our revenue goals as soon as possible. There’s not a lot of time, so we’ll need to select the features carefully. We’ll tell them that they’re allowed to break whatever rules required to get the job done.”

“A manufacturing plant is a system. The raw materials start on one side, and a million things need to go just right in order for it to leave as finished goods as scheduled out the other side. Everything works together. If any work center is warring with the other work centers, especially if Manufacturing is at war with Engineering, every inch of progress will be a struggle.”

“You’ve got to stop thinking like a work center supervisor. You need to think bigger, like a plant manager. Or better yet, think like the person who designed this manufacturing plant and all of the processes it relies upon. They look at the entire flow of work, identify where the constraints are, and use every possible technology and bit of process knowledge they have to ensure work is performed effectively and efficiently. They harness their ‘inner-Allspaw.’”

In manufacturing, we have a measure called takt time, which is the cycle time needed in order to keep up with customer demand. If any operation in the flow of work takes longer than the takt time, you will not be able to keep up with customer demand.

“So when you run around screaming, ‘Oh no! We don’t have environments for Phoenix ready! Help, help! Oh, no! We can’t deploy, because someone broke the Phoenix environments again!’” he says in a high, girlish voice, “That means the cycle time of some critical operation in your area of responsibility is greater than the takt time. That is the reason you can’t keep up with customer demand.

“As part of the Second Way, you need to create a feedback loop that goes all the way back to the earliest parts of product definition, design, and development,” he says. “Given the conversations you’re having with Dick, you may even be able to go earlier in the process.”

“Look down at the long lane of equipment between the orange tape on the floor. That lane makes some of the highest profit items we have. But as fate would have it, that particular flow of work involves two operations that have the longest setup and process times: application of a paint powder coating and baking it in the heat treat oven.” He looks up, arms spread outward. “Back in the day, the cycle time for those two operations was so much larger than takt time, we were never able to keep up with customer demand. How can life be so unfair? The most profitable items used both of our constraints: the heat treat oven and the paint booths! What do we do? “Customers were even offering to throw money at us, begging us for more of these widgets, but we had to turn them away. The setup time for each job took hours or even days. We had to use enormous batch sizes to meet demand. We had these huge trays to paint and would bake as many units at a time as possible. We knew we had to reduce batch sizes to improve throughput, but everyone said that it couldn’t be done.”

“How Toyota solved this problem is legendary,” he says. “During the 1950s, they had a hood stamping process that had a change-over time of almost three days. It required moving huge, heavy dies that weighed many tons. Like us, the setup times were so long that they needed to use large batch sizes, which prevented them from using one stamping machine to manufacture multiple different car models simultaneously.

“They closely observed all the steps required to do the changeover, and then put in a series of preparations and improvements that brought the changeover time down to under ten minutes. And that, of course, is where the legendary ‘single-minute exchange of die’ term comes from. “We studied all the works of Sensei Taiichi Ohno, Steven Spear, and Mike Rother. We knew that we had to decrease our batch size, but we weren’t dealing with hood stamping dies. We were dealing with painting and curing,” he continues. “After weeks of brainstorming, investigation, and experimentation with Engineering, we had a crazy idea: Maybe we could do the painting and curing in a single machine. We cobbled together an oven that also applied the paint powder onto the parts, which were pulled through on a chain and gear that we took from a bicycle.

“We combined four work centers into one, eliminating over thirty manual, error-prone steps, completely automating the work cycle, achieving single-piece flow, and eliminating all that setup time. Throughput went through the roof.

“The benefits were enormous,” he says with pride. “First, when defects were found, we fixed them immediately and we didn’t have to scrap all the other parts in that batch. Second, wip was brought down because each work center never overproduced product, only to sit in the queue of the next work center. But the most important benefit was that order lead times were cut from one month to less than a week. We could build and deliver whatever and however many the customer wanted and never had a warehouse full of crap that we’d need to liquidate at fire-sale prices.

“So, now it’s your turn,” he says sternly, poking a finger into my chest. “You’ve got to figure out how to decrease your changeover time and enable faster deployment cycle time. “I think your target should be…” he says, pausing for a moment. “Ten deploys a day. Why not?” My jaw drops. “That’s impossible.”

Allspaw and Hammond ran the it Operations and Engineering groups at Flickr. Instead of fighting like cats and dogs, they talked about how they were working together to routinely do ten deploys a day!

“Let me tell you,” he continues, “we all thought that this engineer had lost his marbles. But I learned that the practices that Allspaw and Hammond espoused are the inevitable outcome of applying the Three Ways to the it value stream. It totally changed how we managed it and it saved our company.

“Allspaw taught us that Dev and Ops working together, along with qa and the business, are a super-tribe that can achieve amazing things. They also knew that until code is in production, no value is actually being generated, because it’s merely wip stuck in the system. He kept reducing the batch size, enabling fast feature flow. In part, he did this by ensuring environments were always available when they were needed. He automated the build and deployment process, recognizing that infrastructure could be treated as code, just like the application that Development ships. That enabled him to create a one-step environment creation and deploy procedure, just like we figured out a way to do one-step painting and curing.

Jez Humble and Dave Farley independently came to the same conclusions, and then codified the practices and principles that enable multiple deployments per day in their seminal book Continuous Delivery. Eric Ries then showed us how this capability can help the business learn and win in his Lean Startup work.”

“you need to create what Humble and Farley called a deployment pipeline. That’s your entire value stream from code check-in to production. That’s not an art. That’s production. You need to get everything in version control. Everything. Not just the code, but everything required to build the environment. Then you need to automate the entire environment creation process. You need a deployment pipeline where you can create test and production environments, and then deploy code into them, entirely on-demand. That’s how you reduce your setup times and eliminate errors, so you can finally match whatever rate of change Development sets the tempo at.” “Hold on,” I say. “What is it exactly that I’m supposed to automate?”

“Go ask Brent. Get him assigned to that new team, and make sure that he doesn’t get distracted. Now more than ever, until you get your build process automated, he is your bottleneck. Get the things that are in his head encoded into the build procedures. Get humans out of the deployment business. Figure out how to get to ten deploys a day.”

“Stop focusing on the deployment target rate. Business agility is not just about raw speed. It’s about how good you are at detecting and responding to changes in the market and being able to take larger and more calculated risks. It’s about continual experimentation, like Scott Cook did at Intuit, where they did over forty experiments during the peak tax filing season to figure out how to maximize customer conversion rates. During the peak tax filing season!

“If you can’t out-experiment and beat your competitors in time to market and agility, you are sunk. Features are always a gamble. If you’re lucky, ten percent will get the desired benefits. So the faster you can get those features to market and test them, the better off you’ll be. Incidentally, you also pay back the business faster for the use of capital, which means the business starts making money faster, too. “Steve is betting his entire survival on your ability to execute and deploy capabilities faster. So, get to work with Chris to figure out how at every stage of the agile development process, you not only have shippable code, but a working environment it can deploy into!”

“You think it Operations is rocket-science compared to manufacturing. What absolute baloney,” he says dismissively. “From where I’m sitting, the people in this building have been far more creative and courageous than anything I’ve seen come from you it guys so far.”

“Well, he gave us a couple of clues. He said that the problems we need to focus on are the deployment process and the way we’re building the environments. He seems to think we must be doing something fundamentally wrong because of all the chaos resulting from each Phoenix deployment.”

“What? Why in the world would we need to do ten deploys a day? Our sprint intervals are three weeks long. We don’t have anything to deploy ten times a day!”

“Are you sure? What about bug fixes? What about performance enhancements when the site grinds to a halt, like what’s happened during the last two major launches? Wouldn’t you love to do these types of changes in production routinely, without having to break all the rules to do some sort of emergency change?” Chris thinks for a couple of moments before responding. “Interesting. I would normally call those types of fixes a patch or a minor release. But you’re right—those are deployments, too. It would be great if we could roll out fixes more quickly, but come on, ten deploys a day?”

“How about enabling Marketing to make their own changes to content or business rules or enabling faster experimentation and a/b split testing, to see what offers work best?”

Wes scratches his head for a moment before he says, “What do you think, Brent? I would have thought about a hundred steps…” “Really?” Brent responds. “I would have thought it was more like twenty steps.”

interjects, “I’m not sure where you’re starting to count, but if we begin at the point where Development commits code and we label it as a ‘release candidate,’ I can probably come up with a hundred steps, even before we hand it to it Operations.” Uh-oh. Wes interrupts, “No, no, no. Bill said ‘deployment steps.’ Let’s not open up a can of—” As Wes talks, I think about Erik challenging me to think like a plant manager as opposed to a work center supervisor. I suddenly realize that he probably meant that I needed to span the departmental boundaries of Development and it Operations. “You guys are both correct,” I say, interrupting Wes and William. “William, would you mind writing down all the steps on the whiteboard? I’d suggest starting at ‘code committed,’ and keep going until the handoff to our group.

When William is finished, there are thirty boxes on the board. Looking over at Wes, I see that rather than looking irritated, he actually appears deep in thought, rubbing his chin while looking at the diagram. I indicate to Brent and Wes that one of them should continue where William left off.

Brent gets up and starts drawing boxes to indicate the packaging of the code for deployment; preparing new server instances; loading and configuring the operating system, databases, and applications; making all the changes to the networks, firewalls, and load balancers; and then testing to make sure the deployment completed successfully.

Patty says, “You know, this reminds me of something that I’ve seen the plant floor guys use all the time. If one of them walked in, I’m guessing that they’d think we’re building a ‘value stream map.’ Mind if I add a couple of elements?” I pass the whiteboard marker to her and sit down. For each of the boxes, she asks how long each of these operations typically takes then jots the number on top of the box. Next, she asks whether this step is typically where work has to wait then draws a triangle before the box, indicating work in process.

Brent beats him, saying, “You’ve nailed it. William can probably relate to these problems: the release instructions are never up-to-date, so we’re always scrambling, trying to futz with it, having to rewrite the installer scripts and install it over and over again…”

Brent says, “Maybe William and I can work together to build a deployment run book, to capture all the lessons learned from our mistakes?”

“Having each group cobble an environment together obviously isn’t working. Whatever we do must take us a big step toward this ‘ten deploys a day’ target,” I say. “This implies that we need a significant amount of automation. Brent, what would it take for us to be able to create a common environment creation process, so we can simultaneously build the Dev, qa, and Production environments at the same time, and keep them synchronized?”

“Interesting idea,” Brent says, looking at the board. He stands up and draws three boxes labeled “Dev,” “qa,” and “Production.” And then underneath them, he draws another box labeled “Build Procedure” with arrows into each of the boxes above. “That’s actually pretty brilliant, Bill,” he says. “If we had a common build procedure, and everyone used these tools to create their environments, the developers would actually be writing code in an environment that at least resembles the Production environment. That alone would be a huge improvement.”

He takes the marker cap out of his mouth. “To build the Phoenix environment, we use a bunch of scripts that we’ve written. With a bit of documentation and cleanup, I bet we could cobble together something usable in a couple of days.”

Chris seems excited. “Brent, if it’s okay with you and everyone else, I’d like to invite you to our team sprints, so that we can get environment creation integrated into the development process as early as possible. Right now, we focus mostly on having deployable code at the end of the project. I propose we change that requirement. At each three-week sprint interval, we not only need to have deployable code but also the exact environment that the code deploys into, and have that checked into version control, too.”

“On the manufacturing floor, whenever we see work go backward, that’s rework. When that happens, you can bet that the amount of documentation and information flow is going to be pretty poor, which means nothing is reproducible and that it’s going to get worse over time as we try to go faster. They call this ‘non-value-add’ activity or ‘waste.’”

“If we redesign the process, we need to have the right people involved upfront. This is like the manufacturing engineering group ensuring that all parts are designed so that they are optimized for manufacturing and that the manufacturing lines are optimized for the parts, ideally in single-piece flow.”

William walks to the whiteboard and points at a box called “code commit.” “If I could wave this magic wand, I would change this step. Instead of getting source code or compiled code from Dev through source control, I want packaged code that’s ready to be deployed.”

“And you know,” he continues, “I want this so much, I’d happily volunteer to take over responsibility for package creation. I know exactly the person I’d assign, too. She would be responsible for the Dev handoff. When code is labeled ‘ready to test,’ we would then generate and commit the packaged code, which would trigger an automated deployment into the qa environment. And later, maybe even the Production environment, too.”

“Wow. You’d really do that?” Wes asks. “That would be really great. Let’s do it—unless Brent really wants to keep doing the packaging?” “Are you kidding?” Brent asks, bursting out laughing. “I’ll buy whoever this person is drinks for the rest of the year! I love this idea. And I want to help build the new deployment tools. Like I said, I’ve got a bunch of tools that I’ve written that we can use as a starting point.”

The first challenge: to name the swat team project. We couldn’t keep calling it “mini-Phoenix,” so we eventually had to spend an hour debating names. My guys wanted to call it “Cujo” or “Stiletto.” But the developers wanted to call it “Unicorn.”

Regardless of my distaste for the name, Project Unicorn was shaping up amazingly well. With the objective of doing whatever it takes to deliver effective customer recommendations and promotions, we started with a clean code base that was completely decoupled from the Phoenix behemoth.

It was amazing to see how this team tackled obstacles. One of the first challenges was to start analyzing the customer purchase data, which was the first brick wall. Even touching the production databases meant linking to their libraries, and any changes to them would require convincing the architecture team to approve it. Since the entire company could be out of business by that time, the developers and Brent decided to create a completely new database, using open source tools, with data copied from not only Phoenix but also the order entry and inventory management systems. By doing this, we could develop, test, and even run in operations without impacting Phoenix or other business critical applications

However, a small part of me wondered how we’re going to manage the inevitable sprawl, if every project could spawn a new database on a whim. I remind myself to ensure that we standardize what types of databases we can put into production to ensure that we have the right skills to support these long-term.

In the meantime, Brent worked with William’s team to create the build procedures and automated mechanisms that could simultaneously create the Dev, qa, and Production environments. We were all astonished that within the three-week sprint, perhaps for the first time in memory, all the developers were using exactly the same operating system, library versions, databases, database settings, and so forth. “This is unbelievable,” one of the developers said at the sprint retrospective, held at the end of each sprint. “For Phoenix, it takes us three or four weeks for new developers to get builds running on their machine, because we’ve never assembled the complete list of the gazillion things you need installed in order for it to compile and run. But now all we have to do is check out the virtual machine that Brent and team built, and they’re all ready to go.

Similarly, we were all amazed that we had a qa environment available that matched Dev so early in the project. That, too, was unprecedented. We needed to make a bunch of adjustments to reflect that the Dev systems had considerably less memory and storage than qa, and qa had less than those in Production.

Automated code deployments weren’t quite working yet, nor was the migration of code among the environments, but William’s team had demoed enough of those capabilities that we all had confidence that they’d have it nailed down soon. On top of that, the developers had hit their feature sprint goals ahead of schedule. They generated reports showing “customers who bought this product bought these other products.” The reports were taking hundreds of times longer than expected, but they promised that they could improve performance.

Because of our rapid progress, we decided to shrink the sprint interval to two weeks. By doing this, we could reduce our planning horizon, to make and execute decisions more frequently, as opposed to sticking to a plan made almost a month ago.

“Unicorn is the one last hope we have of hitting our quarterly number. One more blown quarter, and the board will surely split the company apart, and you’ll be able to help the task force then. But if we hit our numbers, we have a shot at keeping the company together. That’s why Unicorn is our absolute highest priority. Steve was very clear on this.”

I nod. “I’m amazed at the what we can do with the fast Unicorn cycle times, too. With Phoenix, we only rehearsed and practiced doing the deployments once per quarter. Just in the last five weeks, we’ve done over twenty Unicorn code and environment deployments. It almost feels routine. As you said, it’s the opposite of Phoenix.”

John says, “Most of the reservations I had about Unicorn don’t seem valid anymore. We’ve put in regular checks to make sure that the developers who have daily access to production only have read-only access, and we’re making good progress on integrating our security tests into the build procedures. I’m pretty confident that any changes that could affect data security or the authentication modules will get caught quickly.”

But the notion of having to keep up with ten deploys a day?” he continues. “Complete lunacy! But after being forced to automate our security testing, and integrating it into the same process that William uses for his automated qa testing, we’re testing every time a developer commits code. In many ways, we now have better visibility and code coverage than any of the other applications in the company

He adds, “You should know that we just closed out the last of the sox-404 issues. We were able to prove to the auditors, thanks in large part to the new change control processes you put in, that all the current controls are sufficient, closing out the three-year repeat audit finding.” With a smile, he adds, “Congratulations, Bill. You’ve done what none of your predecessors have been able to do, which is to finally get the auditors off our back!”

Much to my surprise, the short week goes smoothly. Before everyone leaves on Wednesday for the Thanksgiving holiday, the big Unicorn campaign is ready. Code performance is still ten times slower than we need, but we’re okay for now because we can just spin up hundreds of compute instances in the cloud. We had a genuine showstopper when qa discovered that we were recommending items that were out of stock.

Ironically, one of the developers suggested turning off all the real-time recommendations, which we had worked so hard to build. Why recommend more products to buy, he argued, if customers can’t even complete a transaction? Maggie quickly agreed, but it still took the developers two hours to change and deploy. Now, this feature can be disabled with a configuration setting, so we can do it in minutes next time around, instead of requiring a full code rollout. Now that’s what I call designing for it Operations! It’s getting easier and easier to manage the code in production.

“This morning, our store managers opened up their locations for Black Friday. From the moment they opened the doors, people poured in, waving around printouts of their Unicorn promotion e-mails. In-store traffic today is at record levels. The problem is that the promoted items are now almost completely gone. Our store managers started panicking because customers were leaving angry and empty-handed.

“When store managers try to issue rain checks or get an item shipped to the customer, they’re having to manually key in the order from our warehouse. It’s taking them at least fifteen minutes per order, which is resulting in long lines at the stores and more frustrated customers.”

quickly continue, “Chris believes that once this mrp application is back in-house, we could build an interface to Unicorn. We would then start building the manufacturing capability to move us from ‘build to inventory’ to ‘build to order,’ which would enable us to provide the custom kits as Sarah requested. If we execute everything flawlessly, and the integration with the order entry and inventory management systems goes as planned, we could match what our competitors are doing in about ninety days.”

Chris takes this one. “The outsourcer may have made big changes to the code base that we don’t know about, which would slow down the development schedule. But my personal belief is that this risk is minimal. Based on their behavior, I don’t think they made any significant changes to functionality. “I’m not worried about the technical challenges,” he continues. “The mrp was not designed for large batch sizes and certainly not batch sizes of one that we’re talking about here. But I’m sure we can make something work short-term and figure out a long-term strategy as we go.” When Chris finishes, Patty adds, “The outsourcer could also decide to make the transition back to us difficult, and there could be animosity from the affected engineers. There were a lot of hard feelings when we announced the contract—among other things, their pay was cut the instant they switched from being Parts Unlimited employees to being a vendor.”

Steve is listening attentively. He then turns to Sarah and asks, “Your thoughts on the team’s proposal?” She says nothing for several moments but eventually says imperiously, “I think we need to check with Bob Strauss and get full board approval before we undertake a project this big and risky. Given the previous performance of it, this could jeopardize all of our manufacturing operations, which is more risk than I think we should take on. In short, I personally do not support this proposal.”

Steve continues, “On the contrary, thanks to it, we may no longer need to consider all the onerous strategic options that you and Bob are preparing, but your point is well taken.” To the rest of us, Steve says, “I’m assigning you one of Dick’s best people and our corporate counsel. They’ll help you flawlessly execute this project and make sure we use every trick in the book to get what we need from the outsourcer. I’ll make sure Dick gives this project his personal attention.” Sarah’s eyes widen even further. “That’s an excellent idea, Steve. That would significantly reduce our risk here. I think Bob will really like it.”

As I’m driving, I contemplate how quiet it’s been. I keep expecting someone to call in another Sev 1 incident. Instead, my phone just sits in the cup holder, completely silent—like yesterday—and the day before that.

Thankfully, I’m now coaching all my managers through our two-week improvement cycles, according to the Improvement Kata, which keeps me from feeling totally useless. I’m especially proud that for an entire month, my group hit our target of spending fifteen percent of our time on preventive infrastructure projects. And it shows.

Against my expectations, everyone jumped enthusiastically on Project Narwhal otherwise known as the “Simian Army Chaos Monkey” project. Like the legendary stories of the original Apple Mac os and Netflix cloud delivery infrastructure, we deployed code that routinely created large-scale faults, thus randomly killing processes or entire servers. Of course, the result was all hell breaking loose for an entire week as our test, and occasionally, production infrastructure crashed like a house of cards. But, over the following weeks, as Development and it Operations worked together to make our code and infrastructure more resilient to failures, we truly had it services that were resilient, rugged, and durable.

John loved this, and started a new project called “Evil Chaos Monkey.” Instead of generating operational faults in production, it would constantly try to exploit security holes, fuzz our applications with storms of malformed packets, try to install backdoors, gain access to confidential data, and all sorts of other nefarious attacks. Of course, Wes tried to stop this. He insisted that we schedule penetration tests into predefined time frames. However, I convinced him this is the fastest means to institutionalize Erik’s Third Way. We need to create a culture that reinforces the value of taking risks and learning from failure and the need for repetition and practice to create mastery. I don’t want posters about quality and security. I want improvement of our daily work showing up where it needs to be: in our daily work.

Steve smiles. “Thanks for coming over early. We’re going to have a record-breaking quarter. We couldn’t have done it without you and Chris. For the first time in years, our market share is up! You know, I wish I could see the looks on our competitors’ faces. They’re probably scrambling, trying to figure out how we did it.” Steve is smiling broadly. “I actually saw Dick crack a smile the other day. Well, he showed his teeth, at least. Project Unicorn and that new project, Narwhal, are helping us understand what are customers actually want. Our average order size just hit a record last week, and Dick said that Unicorn had the fastest payback of any project we’ve done in recent memory.”

I laugh. “No one is more upset about it than Maggie. She’s convinced that all her product managers are laughing at her. She’s told her husband that if the next project is called ‘Hello Kitty,’ she’ll quit.”

“I agree with you, Bill. Which is why I’ve decided that we should fill this position internally. You have any suggestions on who we should promote?” I go through the possible candidates in my mind. It’s not a long list. “I think Chris is the obvious choice. He was the driving force behind Unicorn, as well as Narwhal. If it weren’t for his leadership, I’m pretty sure we’d still be stuck dead in the water.” He smiles. “You know, it’s funny. Everyone thought you’d say that. However, I won’t be following your recommendations.” He continues, “This is going to take a while to explain. You were everyone’s unanimous choice to become cio. But to be brutally candid, I don’t want you there.”

Reacting to my obvious distress, he says, “Hey, relax. Let me explain. My board holds me responsible for making the best use of company resources to achieve the goals that maximize shareholder value. My primary job is to lead my management team to make that happen.”

“Now I want to see you grow and learn, and build new skills to best help all of Parts Unlimited. If you’re up to it, I am prepared to invest in you. I want to put you on a fast track, two-year plan. You’ll do rotations in sales and marketing, manage a plant, get international experience, manage the relationships of our most critical suppliers, and manage our supply chain. Trust me, this will not be a vacation. You’re going to need help—a lot of it. Erik has kindly agreed to mentor you, because we both believe this will be the most difficult thing you’ve ever done.

“But,” he continues, “if you achieve each of fifteen specific performance targets we’ve laid out for you, we’ll move you into a provisional Chief Operating Officer role in two years, where you’ll work closely with Dick while he gets ready to retire. If you work hard, get results and play your cards right, you’ll be the next coo of the company in three years.”

“You don’t have to answer now,” he says, obviously satisfied that his pitch is having the desired effect. “Half my board thinks I’m crazy. Maybe they’re right, but I trust my instincts. I don’t know how this is going to shape up, but I have confidence that this is what’s best for the company. My gut tells me that in ten years, when we’re mopping the last pieces of our competition off the floor, this is the gamble that will have made that possible.

“While we’re dreaming big dreams here, let me say this,” he continues. “In ten years, I’m certain every coo worth their salt will have come from it. Any coo who doesn’t intimately understand the it systems that actually run the business is just an empty suit, relying on someone else to do their job.”

Suddenly, I think of how one of Erik’s higher ups decided to mustang him from a senior nco to a lowly lieutenant, forcing him to climb the ladder again from the very bottom of the officer ranks. Obviously, Erik had the courage to do it, and the rewards for him (and his family, if he has one) seem pretty evident. He’s living a life that seems to have transcended our mortal plane of existence.

As if Steve knows what I’m thinking, he says, “You know, when Erik and I first met, many months ago, he said that the relationship between it and the business is like a dysfunctional marriage—both feel powerless and held hostage by the other. I’ve thought about this for months, and I finally figured something out. “A dysfunctional marriage assumes that the business and it are two separate entities. it should either be embedded into business operations or into the business. Voilà! There you go. No tension. No marriage, and maybe no it Department, either

Dick approaches me, holding a glass of scotch. “Congratulations, Bill. I’m looking forward to working closely with you in the years to come.” Shortly, I find myself laughing with a bunch of other people, accepting their congratulations, and trading stories about the amazing journey we’ve been on.

As he reaches into the box at his feet, he says, “We argued for a long time about what it should be. But in the end, it was obvious…” When I see what he pulls out from the box, I burst out laughing. “Your old craptop!” he exclaims, holding it high in the air. “It was a shame to make it unusable by bronzing it, but you’ve got to admit, it’s beautiful, isn’t it?”

“After what I’ve seen, I don’t want any part of it!” Erik says, laughing loudly. “I think Parts Unlimited is going to make a lot of money. We’ll see how good your competition really is, but my suspicion is that they’ll have no idea what hit them. For me, this isn’t just idle theory. If all goes according to plan, within a couple of weeks, I’ll likely be one of the largest investors in this company. The last thing I want is a bunch of insider information that restricts my ability to buy and sell!”

I’ve long believed that to effectively manage it is not only a critical competency but a significant predictor of company performance,” he explains. “One of these days, I’d like to create a hedge fund that invests in companies, taking long positions on companies with great it organizations that help the business win, and short the companies where it lets everyone down. I think we’d make a killing. What better way is there to force the next generation of ceos to give a shit about it?” He continues. “I can’t do that if I’m tied up as a board director in all these companies. Bad optics. Too much potential jeopardy with the sec, auditors, and all that.”

“Give me a break. You’re bored in your current role. You’re going to become a lot less bored. Fast. And remember that there are a lot of experienced people around you who’ve been on similar journeys, so don’t be the idiot that fails because he didn’t ask for help.”

I need you to help me elevate the state of the practice of how organizations manage technology. Let’s face it. Life in it is pretty shitty when it’s so misunderstood and mismanaged. It becomes thankless and frustrating as people realize that they are powerless to change the outcome, like an endlessly repeating horror movie. If that’s not damaging to our self-worth as human beings, I don’t know what is. That’s got to change,” he says passionately. “I want to improve the lives of one million it workers in the next five years. As someone wise once told me, ‘Messiahs are good, but scripture is better.

He says, “I want you to write a book, describing the Three Ways and how other people can replicate the transformation you’ve made here at Parts Unlimited. Call it The DevOps Handbook and show how it can regain the trust of the business and end decades of intertribal warfare. Can you do that for me?”

Maybe everyone attending this party is a form of DevOps, but I suspect it’s something much more than that. It’s Product Management, Development, it Operations, and even Information Security all working together and supporting one another. Even Steve is a part of this super-tribe.

When this book was first published in January 2013, DevOps was very much in its early years, less than four years after the famous “10+ Deploys Per Day: Dev and Ops Cooperation at Flickr” was presented by John Allspaw and Paul Hammond, and a little over two years after the first DevOpsDays conference in the United States.

So much about DevOps is counter intuitive, contrary to common practice, and even controversial. If production deployments are problematic, how on earth can deploying more frequently be a good idea? How can reducing the number of controls actually increase the security of our applications and environments? And can technology really learn anything from manufacturing? There are countless more examples of these difficult to believe claims.

I would recommend to any one who has interest in Dr. Goldratt’s work to listen to his audiobook Beyond the Goal, which was released twenty-one years after The Goal. It brilliantly captures in one place his own lifetime of learnings, and synthesizes those learnings into a comprehensible and comprehensive whole

I’ve also come across otherwise smart [people] who are of the mistaken belief that if they hold on to a task, something only they know how to do, it’ll ensure job security. These people are knowledge Hoarders. This doesn’t work. Everyone is replaceable. No matter how talented they are. Sure it may take longer at first to find out how to do that special task, but it will happen without them.

After finishing the book, I took a moment to look back on my career. Thinking of all of the people like John who I’d run into and worked with over the year I realized, with a little bit of terror, why I hated him so much. Before I studied Visible Ops and DevOps, I was John.

As Dr. W. Edwards Deming is famously paraphrased, “Learning is not compulsory…neither is survival.”

After seeing the 2009 Velocity Conference presentation “10 Deploys per Day” by John Allspaw and Paul Hammond, I was convinced others were thinking in a similar way. So I decided to organize the first DevOpsDays, accidently coining the term DevOps.

After his session I had an opportunity to sit down and have coffee with him. I was totally sold on what we now call infrastructure as code. However, while we met for coffee, Luke started going even further, explaining his ideas. He started telling me he believed that operations was going to have to start behaving like software developers. They were going to have to keep their configurations in source control and adopt CI/CD delivery patterns for their workflow. Being the old IT Operations person at the time, I think I replied to him with something like, “That idea is going to sink like Led Zeppelin with Ops folk.” (I was clearly wrong.)

Myth—DevOps Replaces Agile: DevOps principles and practices are compatible with Agile, with many observing that DevOps is a logical continuation of the Agile journey that started in 2001. Agile often serves as an effective enabler of DevOps, because of its focus on small teams continually delivering high quality code to customers.