3 Reasons why DevOps isn’t changing IT faster

iStock_RobotRaceThe recent article from Luke Kanies (from Puppet Labs) on Wired.com really got me thinking. Similar to Luke, I have had an interesting vantage point to observe the changing nature of systems administration – spending time as one myself – over the last decade or so. From my graduate physics department, to Loudcloud, EDS, BladeLogic, BMC, and now Sumo Logic, I have seen the best and not-so-great IT shops and how they operate. Not all of those IT teams I saw in action were, or are now, adopting best practices like automation and DevOps. The trend that Luke points out means that operations teams that continue to languish in constant firefighting mode, relying on ad-hoc scripts and the sweat off the admin’s brow, are becoming more obvious for being clearly out of step with the direction of the industry.

So, why aren’t all operations teams, and the techies themselves, falling over each other to embrace tools like Puppet, SumoLogic, and other DevOps/Automation tools? Clearly some organizations are embracing them. Why not all of them? I have few of my own ideas here, and I would like to hear yours as well.

1. The move from “Artist” to “Manager” is not natural

Back when I started in IT, most IT admins “owned” a small collection of devices or applications. Server admins owned a handful of servers, database admins a few databases, network admins a few switches and firewalls, etc. They controlled access to their systems jealously, and took personal pride in their operation. They were artists, and their systems, and the way those systems were managed, were an art form. As IT budgets have shrunk, and the load on IT increased, this level of care is impossible.

Yet, you still find many admins jealously guarding their root access privileges, instead of moving to a shared responsibility model with other admins. Why? I think it is the same reason why I feel such satisfaction after cooking a meal, building new shelves in my garage, or fixing a leaky faucet. I did it myself, and it feels good to start and finish something. Participating in automated processes can be deeply unsatisfying. That is why admins need to learn new skills, and find new pride in the quality of their automation or find satisfaction in steadily improving quality.

2. Using Automation may seem like losing control

One conversation from my IT past sticks in my mind more than any other. I was on site with a customer trying to explain the benefits of automation to a group of systems administrators. One system admin floored me by insisting that she could more accurately, and more quickly, make changes to 20 UNIX servers than I could ever do with automation. It was like some modern version of John Henry calling me out to some man vs. machine contest, and subtly decrying the inhumanity of my automation tool. I can’t even remember my answer now, but this perspective is at the root of much of the push-back against automation and DevOps. Instead of looking at the business outcome – better experience and value for the customer – some frustrated system admins see these new ideas as a direct affront to the quality of their work. This is precisely why I think the fundamental shift in DevOps is from a internal IT focus to an external customer focus. That way admins can measure their success by customer impact. Not an easy change to make, but it is essential for IT’s continued relevance.

3. Change seems hard/bad/unnatural/unneeded

Isn’t the root of the resistance here really the natural tendency to resist change? On the other hand, how many times have operations teams been assured that the latest IT fad will reduce their workload and improve quality, only to see the opposite happen? So what’s different about DevOps? I think I could write a whole blog entry just on that, but a few things come to mind. First, the focus is on customer value, which greatly simplifies priorities. Second, it’s all about outcomes, not process for the sake of process. Finally, it is all about continuous improvement driven by the experience of the people on the frontline. This means that admins must be rewarded going forward for doing things that increase customer value, rather than putting out fires or pleasing angry executives.

So, it all comes back to culture – surprise, surprise. I think this will be the primary challenge of DevOps going forward. How do we overcome the IT culture so resistant to change, while providing an attractive way for all of those systems administrators to breathe easily in their new roles?


Kaizen and the Art of DevOps Automation Maintenance

And now we come to the most “boring” part. Right? Maintenance. The death of joy for the innovator. Or is it? I don’t think so. Continuous innovation is at the core of DevOps and Lean methodology. Maintenance is essential to keeping the spirit of DevOps strong, and automation that isn’t improving will grow stale and useless.

So, let’s review the IT Automation Curator’s job description on last time:

  • Collect existing automation, and then Catalog it where others can find it (See Part 2)
  • Develop new automation based on requirements from IT (See Part 3)
  • Train others on how to use the automated processes (See Part 4)
  • Maintain the existing automation

Going back to Lean methodology, we can look to the idea of Continual Improvement or Kaizen. There are 3 main areas from Masaaki Imai‘s 1986 book Kaizen: The Key to Japan’s Competitive Success.

  • Reflection of processes. (Feedback)
  • Identification, reduction, and elimination of suboptimal processes. (Efficiency)
  • Incremental, continual steps rather than giant leaps. (Evolution)

Use Metrics and Reporting for Feedback

How can you improve if you don’t know how far you’ve come and how well you are doing? That’s like going on a diet without ever weighing yourself or even looking in the mirror. Healthy weight loss involves such small changes every week that it would discouraged if you didn’t look at changes over the long term (I speak from experience). So, why do so few companies and automation vendors include metrics and reporting on automation efficiency?! The whole point of implementing automation is to reduce waste, reduce costs, and increase velocity. But you have no context for understand if you have succeeded if you don’t have metrics and reporting (I talked about this in a previous post).

Ruthlessly eliminate sub-optimal processes

The whole point of this process is get rid of wasted effort and time – muda. The hardpart here is that once you agree to continually improve your automated processes, you have to be ruthless in your evaluation of their efficiency. That means there are no sacred cows. Just because somebody smart and dedicated invested hours of their life into creating something, doesn’t mean it can’t be improved or scrapped entirely. The whole team has to be dedicated to, and incentivized towards, efficiency and continuous improvement. This point is important – these kind of improvements come from the grassroots – not the top. If the people in the trenches aren’t bought into continual improvement, it won’t work.

Baby Steps, not Leaps of Faith

I know the hardest part of this approach for me is the gradualism. I like to grandiloquently solve grandiose problems with lofty and visionary solutions. The problem is that most of those involve large amounts of kool-aid, and they are never finished. The truly mature IT organization has to keep their eye on the goals of the business, and relentlessly reduce muda – step by tortuous step. We can refer back to the weight loss analogy. You lose weight through all of the small victories – Do I really need that donut? One serving is good enough. But bringing us back to our first point – small victories only show up as victories when you can measure your long term progress. Otherwise it looks like tentative, timid, risk-adverse behavior.

And so, we reach the last chapter of my IT Automation Curator series. It has been a lot of fun writing it, and I hope that you enjoyed it as well. I am looking forward to continuing to explore how the proven methods of lean and agile can be applied to DevOps and Operations overall.

Invite the Suits to the DevOps party? Why DevOps needs business people.

One of the best things about DevOps is the potential to harness technology more completely to the goals of the business. In particular, as Damon Edwards put it almost two years ago “DevOps is not a technology problem”, but rather a business problem. To put it differently – DevOps is a business solution, since the average corporation spends 50% of their budget on IT. Shouldn’t they expect more out of their IT organizations?

As Gene Kim, another great DevOps thinker, said in a BMC interview, even before DevOps was the latest trend, he found that high performing IT organizations were 5-7 times as productive as than their low-performing peers. The ground breaking application of agile and lean methodologies to the DevOps area are blowing that number out of the water as well, making John Allspaw’s 10+ deploys a day in 2009 look positively quaint for some businesses.

So, if the spirit of DevOps is made reality, then it has the potential to make IT a profit engine rather than an unfortunate and annoying cost. Bottom line, DevOps is the best opportunity in decades, maybe ever,  for Operations to be relevant to the business, and a valued growth engine.

So, with all that said, why is most of the DevOps discussion still revolve around technology, and not the business and organizational obstacles to success? We in IT start with what we know – technology. How do I take this cool technology I love – like configuration automation, and apply it to the DevOps problem? Also, much of the discussion is also being driven by software companies that have a vested interest in selling technology and services rather than advocating change per se (As an aside, that is why we have tried so hard to keep the DevOps leadership series at BMC informational, rather than hawking product). In my opinion, this is the natural evolution of any technology trend. So, how do we move it forward, help DevOps continue to mature?

I would suggest that an earlier paper that I co-wrote with Tim Fessenden could provide a clue – building a center of excellence. I will spend some time in future posts working out the details of what that means, but I am convinced that a similar approach as we advocated is essential to implementing DevOps for the average enterprise IT shop (rather than extraordinary ones like Amazon or Etsy). The essential element of the argument is that most IT projects fail because they are incapable of proving their value to the business – usually because of a complete lack of established business metrics. Any why no business metrics – because nobody is held responsible for creating and reporting on business value. So, in short, here is one way to start fixing that:

1. Create a virtual “Center of Excellence” focusing on implementing DevOps at your company

2. Make sure to include business analysts on that team who will be responsible for business metrics

3. Agree on set of “success metrics” that will reported on something like a monthly basis to executives

4. Implement tools to gather and report on those metrics

5. Rinse and repeat for each new project


Over time, the hope is that you will have business expertise embedded in your operations team, and senior leadership will have a very clear idea of what value you are providing. Now, this does mean inviting the “suits” to the party, which might seem to be the end of the party. My suggestion is, ask them to lose the tie and jacket, make them pay for lunch, and you might find that you can work together pretty well after all.