Training others in the dark arts of DevOps Automation

If you are the Automation hero, why would you EVER share that stage? You are basically reducing your value to the organization by sharing your secrets. Right? Wrong! You are actually doing yourself a lot of harm, as I discussed in the the first blog post. How can you move on to other exciting challenges if you have to maintain your work of automation genius?

That is why the the IT Automation Curator’s job description has training as a core requirement:

  • Collect existing automation, and then Catalog it where others can find it (See Part 2)
  • Develop new automation based on requirements from IT (See Part 3)
  • Train others on how to use the automated processes
  • Maintain the existing automation

I had a great comment from jamesmarcus in the first blog post. Here is what he said:

“As a Director of IT I look to tools that promote easy automation, documentation, andbest practices. I try to design networks and setups with the “if I disappear” rule in mind. Meaning another sys admin of lesser knowledge should be able to look at my work and understand how why we did something in a certain way”.

I think this is a great perspective at the core of why I included training in my job description. Very few programmers, sysadmins, and other IT techies enjoy documenting their work. I don’t either – when it is after the fact. It is so mind-numbing to document your automation after you are already done and want to move on. So, that brings us to our first post.

Build your Automation to be well-documented and re-usuable

While performing amazing feats of scripting judo can impress your colleagues and get you kudos online, it is not a good long-term objective. One thing I learned early as a programmer is that creating incredibly efficient and elegant code seemed great, but it was really bad if even I couldn’t figure out what I had done a year later. That all comes down to great comments while you are writing the code. I know this may seem basic, but I have seen too many IT organizations with automation scripts, packages, etc. that no-one understands anymore. This is essentially a guarantee that the automation in question will be left alone to become outdated, brittle, and even “dangerous”. And if you are the only one that understands it, then it is your burden to bear.

So, basic to our train function of the IT Automation curator. How can you possible train people if your automation is over-complicated, un-documented, and impenetrable to mere mortals? Only with difficulty, and no one (especially you) will enjoy the experience. By documenting your automation very well as you write it, and building it to be as straightforward and simple as possible, you increase your chance of handing it off successfully. Writing well-documented and straight-forward automation has to be part of your process – bottom line.

Find automation disciples, and train them in the dark arts

While I see no reason why you can’t “teach” a class on automation as part of this role, I don’t think that is optimal, or even desirable for most people (stage fright anyone?). I have always envisioned a much more personal approach to automation training. Not every sysadmin, administrator, or IT techie extraordinaire will have an aptitude for, or interest in, designing automation. The right person has a somewhat rare combination of programming know-how, patience, troubleshooting skills, and IT systems knowledge. Obviously, everyone will use the automation, but only a few will write it.

The IT Automation Curator should be a mature, senior IT operator that has an eye for spotting talent. Like Mr. Miyagi in Karate Kid, you can watch for the young IT admin with lots of promise and fire in their belly, but unable to conquer the IT problems with their lousy karate skills. In all seriousness, I think mentoring promising candidates on automation best practices is more enjoyable and effective than the typical shotgun approach. The best part is that you can let the young upstart take care of the boring automation bits, while you save the best for yourself!

So, in summary:

  1. You can’t pass on automation that impenetrable to anyone but yourself
  2. One-on-one mentoring is a much more effective way to pass on your automation skills and knowledge

So, here is a parting challenge for all of you out there that actually remember Karate Kid. What might the IT Automation equivalent be of Mr. Myagi’s “catch a fly with chopsticks” trick?


Developing Automation with Lean Methodology

So, now we are on what I consider the most fun part of the IT Automation Curator’s job description ( see the first blog post ) – developing new automation. Let’s first review the job description as I outlined it:

  • Collect existing automation, and then Catalog it where others can find it (See Part 2)
  • Develop new automation based on requirements from IT
  • Train others on how to use the automated processes
  • Maintain the existing automation

There is lots of great technical material out there about creating automation, and I won’t try to duplicate it here. And I also won’t push a particular toolset, though I have my opinions, of course. I am convinced that most automation project fail, not because of problems with the toolset, but rather due to problems with the approach. That doesn’t mean that one tool isn’t better than another, quite the contrary. The difference is just how successful can you be when you do it right.

To Automate or not to Automate – that is the question.

I think choosing what to automate is much more important to how you choose to achieve the automation. The best model we have for that, in my opinion, is Lean Methodology, and Lean Software Development, in particular. The Poppendiecks, creators of Lean Software Development, have their first principle as Optimize the Whole. The whole point of this step is to eliminate waste, known as muda in Lean methodology. Muda is “anything which does not provide customer value”. This is such a simple, yet revolutionary, concept. What if every sysadmin asked himself whether that script he was working on added customer value? Would they even know how to answer the question.

So, how do you answer that question? You look at the whole, end-to-end, process. This means no silos, no team-centric thinking, no “that isn’t my job”. What does it take to deliver a service to the end user/customer, and where are we wasting time and resources? In Lean Methodology you figure this out by drawing a value stream map. In the context of manufacturing, the source of lean methodology, it meant going from factory to factory, supplier to supplier, and putting together the complete picture of how a product is built and delivered.

So, what does that mean to our Automation Curator? Typically lean, or agile, software development stops at the delivering of the product to IT Operations. In the spirit of DevOps, that needs to stop. IT organizations need to start looking at the delivery of a service to a customer from requirements all the way through to production. That is the only context where it makes any sense to uncover muda. If you view IT Operations in isolation, you could very well create a highly efficient IT operation that doesn’t good services.

The IT Automation Curator has to focus on the automation projects that provide value to the customer – delivering new applications and features faster, restores services faster, provides better customer experience, etc.

What does success look like?

Once you have automated a process, how do you even know that you have achieved your goal of eliminating waste? That where measurement and reporting come in. If you have no way of measuring the effects of your hard work, how do you even know if you have been successful? You don’t. That means part of developing any automation has to be about measuring the impact on the customer. For example, you just automated the full-stack build of an application from scratch. How much more quickly can you deliver updated applications, or recover from problems? So, you have automated the application release process. Did that reduce downtime and error during new releases?

Once you have those reports – advertise them. Don’t be shy about showing how you are increasing value for the end customer. So many IT departments have been forced to show value by how much they can cut out of their budget. How much more would the business appreciate an IT department that demonstrably increases value for the customer?

I think this approach is so much more effective for determining what to automate and how. This singular customer and business focus is bound to make the IT Automation Curator a valuable member of the team. A lot more valuable than the “IT Hero” that corrects a preventable problem with heroic effort. I am also hoping to see this result-focused approach take over from the tools-focused approach so prevalent in DevOps today. If DevOps is to be relevant for more than just a few years, it must encourage behavior that adds value to the customer. Why? Because the customer is King!

IT Automation Curator for DevOps – Part 2 – Collect and Catalog

This topic is far too interesting and deep to cover in just one blog post. So, I am going to split the discussion into a few sections. I’ll use my proposed “job description” for an IT Automation Curator as a starting point:

  • Collect existing automation, and then Catalog it where others can find it
  • Develop new automation based on requirements from IT
  • Train others on how to use the automated processes
  • Maintain the existing automation

This first step of collect and catalog is where I have seen many automation efforts stumble. The natural inclination of most techies (myself included) is to jump right into developing automation, no matter what is in place. As I learned the hard way, that is a bad idea. So, I will give a few reasons why this step is important:

Reason #1: If you don’t know about all the automation in place, you don’t really understand how your data center is operating

It’s great that you developed that new automated process that auto-magically deploys a set of configurations for you. Are you sure that other scripts or tools won’t change it or corrupt it? Most IT teams have scripts strewn all over the place – some well known , some the detritus of sysadmins past. They may have been scheduled centrally or on individual servers. This is very hard to get a grip on. There are a few tools out there, but it is hard to ensure that you have found all automation spread over all the systems. This is just another reason why you need to control access and even re-build some servers from scratch (hopefully in an automated way).

Most IT operations teams also have multiple automation tools in play. Each silo-ed team has their preferred tool, which they guard jealously. Overall, this is not a good approach. The more tools you have, the harder it is to standardize automation and create efficient end-to-end processes. At a minimum, all of these tools need to documented and managed centrally.

Reason #2: Don’t duplicate work and ignore experience

A lot of the automation in place may not be optimal, but it was most likely built to solve the same problems you will need to solve later. Tossing it out, or just ignoring it, is essentially disregarding the combined experience of the IT team. Even if you rebuild it in a better tool, and in a more efficient way – the lessons learned will be valuable.

There is also an important less here about prioritization. Just because you can make an automated process more elegant or more efficient, doesn’t mean you should. More often than not you will have no end of automation projects to look at. Why spend your time on what already works? What is important is to apply automation judiciously, where it provides the most value for the business.

Reason #3: More sharing will always lead to better results

Fostering a culture of sharing automation, essentially an open-source culture, will ensure that everyone has access to the best work on offer, that they don’t re-invent the wheel, and it will allow for continual improvement. That last point is crucial. The idea is not for the automation curator to control all the automation per se. They should be catalysts for making better automation, whether they do it or not. So, it is important to leave one’s ego at the door, and admit that your automation becomes better when you let others critique it and improve on it.

Bottom line, having a central place to share and continually improve automation is essential. This will most likely affect your choice of automation platforms as well. If you can’t share and improve, then you will be hobbling yourselves.

So, how do you do this in your own environment? Do you have ideas about the best way to go about it? Any success stories?

IT Automation Curator – Good for techies, good for business, good for DevOps

Recently my thoughts have been going back to a concept I like in the seminal IT operations book, The Visible Ops Handbook (By Gene Kim, Kevin Behr, and George Spafford). I have been doing a lot of thinking about how Lean, DevOps, Agile, etc. are changing IT culture, or at least pressing for change. Properly leveraged automation is a big part of that change process – which makes me think of the passage in Visible Ops where the authors discuss changing the behavior of senior IT staff:

“Their mastery of configurations continually increases while they integrate it into documented and repeatable processes. We jokingly refer to this phenomenon as ‘turning firefighters into curators’ […]”*

As a former IT techie myself, I get the need to challenge oneself in the often routine and monotonous world of IT. Personally, I think that is a lot of the grass-roots impetus behind the DevOps movement, and the adoption of open-source automation tools. Creating automation is a way of turning the mind-numblingly mundane into something exciting and intellectually challenging. So far so good. Boredom leads to sinking morale and productivity – poor morale is bad for business.

So, what’s not to like? In short, it goes back to focus and sustainability. No, I’m not talking green-energy windmills. How do you sustain and focus the efforts of these budding automation aficionados? Left to their own devices, they will likely create lots of useful, but narrowly directed scripts, packages, etc. All of these will be focused on the problems they face on a daily basis. For the problems outside of the automation guru’s gaze – those problems will most likely remain unsolved.

So, this is where the idea from Visible Ops comes to the rescue. The answer is that we pull these gurus out of their day-to-day grind in the IT trenches,and make them automation curators. Now, I know that many of you hear curator and think of a older man in a tweed jacket, peering over horn rimmed glasses, waxing rhapsodic about the various manufacturer stamps of 18th American chamber pots. So, as interesting as early american port-a-potties may be, let’s look at the definition of curator:

curator – one who has the care and superintendence of something (Marriam-Webster Dictionary)

Clearly tweed is not mentioned. In all seriousness, museum curators do much more than merely talk about old things. Considering the Smithsonian’s own description, curators:

  • Acquire new items for the collection
  • Research the collection
  • Display the collection
  • Maintain the collection

So, if we work off the Smithsonian’s “model”, I suggest that an IT Automation Curators would:

  • Collect existing automation, and then Catalog it where others can find it
  • Develop new automation based on requirements from IT
  • Train others on how to use the automated processes
  • Maintain the existing automation

This kind of role is exactly what I missed someone had offered me early in my career. I would have jumped at it. It would have been a great new challenge for me, I would have been creating value for the business, and IT would have been more efficient. And this isn’t really a new idea. Software developers have long needed to share code snippets and concepts with each other, and they defined the interfaces between code as well. The trick here is that Automation Curator needs to take an active role in both building the best automation and also in promoting the proper use of automation in IT.

One last comment. We might ask if this would be better classified as an Automation Librarian. I think it is good question. At the end of the day, I think having the existence of the position is more important than what you call it. However, in my mind the concept of curator leans more towards the acquisition, development, and training part. The words Library and Librarian in IT seem to lean more towards the maintenance and storage part of the equation (notwithstanding what traditional librarians actually do). Curator is also a cool word.

So, why aren’t more IT shops doing this? What do you think?

This is the first part of a multi-part series. Check out the other parts:

* Kim, Gene; George Spafford; Kevin Behr (2005-06-15). The Visible Ops Handbook: Implementing ITIL in 4 Practical and Auditable Steps (Kindle Locations 917-919). IT Process Institute, Inc.. Kindle Edition.

Let’s talk about maturity in automation

I encourage all of you to check out John Allspaw’s new blog series on automation. I am excited about changing the conversation in our industry from tool-happy discussions to how IT can mature in its use of automation. It doesn’t matter if you are pushing DevOps, Lean, ITIL, whatever. The point is that the IT industry needs to grow up and sit at the adults’ table. Too much of our economy rides upon the success of what we do for a living to let ourselves get distracted by the coolness of what we do, and forget the seriousness of what it means.

Push on Mr. Allspaw. Now, I will delve into those massive papers you referred to in your blog. I feel like I am in graduate school again. Too bad there isn’t any Jolt Cola in the house.

Can you do DevOps without automation?

I was reading Tom Parish’s interview Lori MacVittie on the DevOps Leadership Series, and I came across her statement that DevOps and automation are not, in fact, tied at the hip. I think that asking if DevOps needs automation to be successful is a really good question. In fact, it may be one of the essential questions – which comes first? Process or Automation? Lori clearly lands on the process side, and I tend to agree.

I always make the analogy to a hammer and nails. After repairing my fence last year, I discovered what a wonderful thing a nail gun is compared to a hammer and nails. What would have taken me 20-30 minutes, took me 5 minutes – bam, bam, bam. But nail guns are also dangerous. You can’t shoot yourself in the foot with a hammer… I think automation is a lot like that. Just automating something with no firm process in mind is like handing a nail gun to someone who immediately attached their foot to the floor – in record time. And I am not innocent of this. At one point in my career I was fascinated with the power of JScript on Windows when enabled with Active Directory (yes – I am geeking out – bear with me). I could write one script that would execute across 10s of servers in a way that was more difficult in UNIX at the time. But that was the problem. I tested a script on a live environment, and almost took out a live site (we won’t discuss how close I came to doing just that…).

Now, the counter point is that well-tested automation following a good process can’t beaten. I remember back in my BladeLogic days when a sysadmin was intent on convincing me that she could make the same change to 20 UNIX servers, one after another, and be more accurate than an automated tool. Doubtful. But say she could. That only makes sense on the smallest of scales. But if she had automated that, she could have spent her on something actually useful. Imagine that. And isn’t that the point, to make ourselves more productive, so we can make the business (and ourselves) more successful?

That is why applying a methodology like KanBan to DevOps is so promising. Instead of blindingly applying automation with cool tools, you look at your processes holistically, and apply automation systematically where it provides the most value. Which is exactly what manufacturing has done over the last few years.

So, yes. You can do DevOps without automation. But that’s the wrong question. Can you do DevOps without well-understood process and culture change? No. And don’t apply automation unless you tackle that as well.

Invite the Suits to the DevOps party? Why DevOps needs business people.

One of the best things about DevOps is the potential to harness technology more completely to the goals of the business. In particular, as Damon Edwards put it almost two years ago “DevOps is not a technology problem”, but rather a business problem. To put it differently – DevOps is a business solution, since the average corporation spends 50% of their budget on IT. Shouldn’t they expect more out of their IT organizations?

As Gene Kim, another great DevOps thinker, said in a BMC interview, even before DevOps was the latest trend, he found that high performing IT organizations were 5-7 times as productive as than their low-performing peers. The ground breaking application of agile and lean methodologies to the DevOps area are blowing that number out of the water as well, making John Allspaw’s 10+ deploys a day in 2009 look positively quaint for some businesses.

So, if the spirit of DevOps is made reality, then it has the potential to make IT a profit engine rather than an unfortunate and annoying cost. Bottom line, DevOps is the best opportunity in decades, maybe ever,  for Operations to be relevant to the business, and a valued growth engine.

So, with all that said, why is most of the DevOps discussion still revolve around technology, and not the business and organizational obstacles to success? We in IT start with what we know – technology. How do I take this cool technology I love – like configuration automation, and apply it to the DevOps problem? Also, much of the discussion is also being driven by software companies that have a vested interest in selling technology and services rather than advocating change per se (As an aside, that is why we have tried so hard to keep the DevOps leadership series at BMC informational, rather than hawking product). In my opinion, this is the natural evolution of any technology trend. So, how do we move it forward, help DevOps continue to mature?

I would suggest that an earlier paper that I co-wrote with Tim Fessenden could provide a clue – building a center of excellence. I will spend some time in future posts working out the details of what that means, but I am convinced that a similar approach as we advocated is essential to implementing DevOps for the average enterprise IT shop (rather than extraordinary ones like Amazon or Etsy). The essential element of the argument is that most IT projects fail because they are incapable of proving their value to the business – usually because of a complete lack of established business metrics. Any why no business metrics – because nobody is held responsible for creating and reporting on business value. So, in short, here is one way to start fixing that:

1. Create a virtual “Center of Excellence” focusing on implementing DevOps at your company

2. Make sure to include business analysts on that team who will be responsible for business metrics

3. Agree on set of “success metrics” that will reported on something like a monthly basis to executives

4. Implement tools to gather and report on those metrics

5. Rinse and repeat for each new project


Over time, the hope is that you will have business expertise embedded in your operations team, and senior leadership will have a very clear idea of what value you are providing. Now, this does mean inviting the “suits” to the party, which might seem to be the end of the party. My suggestion is, ask them to lose the tie and jacket, make them pay for lunch, and you might find that you can work together pretty well after all.

Is the DevOps community setting the bar too low for automation tools?

I have been in the automation, particularly configuration and application automation, business for a while now. It is very good to see how the current trend of DevOps is pushing IT departments to really and truly embrace automation – and not just for the server. All that said, I am feeling a little like the mainframe guys do about cloud when I see blog posts like this. UrbanCode is now boasting of the fact that configuration-only deployments are the best thing since the first shell script emerged from the primordial ooze of UNIX. Really?! Configuration management systems should boast about being able to figure out on the fly what needs to deployed, and then deploying only that – not forcing their users to figure that out for them and calling that a feature.Have we really taken a step back in the configuration automation industry to a point where boasting about functions that should have been in your product years ago substitutes for substantive contributions? And if this is the new normal, is it working? This kind of relabeling and repacking of old ideas is not new to configuration automation software. Oh wait, now my scripts are “compiled scripts”. Or – My scripts are failing, so I moved from scripts to METAscripts written in a METAscripting language that only a METAcommunity knows :). And now I am rockin’ 25 service environments (who cares that the “last generation” tools can are known to manage 1,000+ service environments). Bottom line, can the current batch of self-styled DevOps automation tools really hang tight in the concrete jungle of enterprise IT operations?

Don’t get me wrong. It is this very willingness to thumb one’s nose at your predecessors, upon whose shoulders you currently stand, that is at the core of innovation. As Picasso said “Good Artists Copy, Great Artists Steal”. So, do we look to the purveyors of software for the solution to the problem?

The simple answer is, No. The reason why a lot of this happens, in my opinion, is because each new group in IT that finds itself tackling a new problem rarely looks backwards, or even in the next cubicle over, for solutions that have already worked. And with DevOps in particular, they are some questions that the Users (IT departments) should be asking the vendors, and themselves. Not every IT department out there will need or want the same solution, but they owe it to themselves to be thorough. So, what does an IT department do to make the right decision.

1) You have to weigh the short term needs of the immediate problem (say small-scale DevOps) against the longer term rollout (DevOps in full production). Many poor IT decisions are made on the basis of “cool feature”-itis, rather the mundane process of choosing what makes the best sense for the business.

2) Use business metrics. Every IT purchasing decision should be made on the basis of sound business metrics (We will save X% in costs, increase revenue by Y%, etc.). That means you need to invite those MBA graduates from the other office over to the team. I know – you don’t want them to bean count you into oblivion. Just realize that they speak the right language to get the project funded. And make them pay for lunch.

3) Hold the vendors (including us) accountable for the statements that we make. We should deliver references and case studies to back up our case. And, if those metrics stand up, you can use them for your business case.

Bottom, set the bar HIGHER DevOps community. You owe to yourselves, and your business, to expect more out of your vendors.

* Image from

Reposted from BMC Communities