DevOps needs a layered approach – Not only process or automation

With any new, emerging area the tendency is for advocates of each new approach to attempt to invalidate or minimize all earlier approaches. Sometimes this is appropriate, but rarely is progress so clear cut. In this vein, I would like to comment on Phil Cherry’s post on First off, I appreciate Phil’s addition to the discussion. I think his delineation between automation approaches is very interesting. However, the devil is in the details. Here are the highlights of my views on this:

Package-based automation

As a former BladeLogic guy, I would be remiss if I didn’t correct a few points in Phil’s analysis. Phil may be confusing OS Vendor packages (RPMs, MSIs, etc.) with configuration management packages. Systems like BladeLogic build packages based on some sort of configuration object system. In other words, the server is represented as a set of configuration objects (a registry key, setting in a config file, etc.) in a particular state. The packages are usually represented as desired states for configurations based on that same object model. There is no reason that those packages have to be applied “all in one go”, since packages can be chained and included in larger jobs with decision conditions. That said, I agree that this type of automation is per-server based, for the most part.

Application Understanding

I do agree that Phil’s definition of automation models don’t understand multi-server dependencies or really know what an “application” is. Phil does ignore in this context that there are other automation approaches that do bridge this multi-system approach by building on the automation platforms. In particular, the trends within virtualization and cloud have pushed vendors to create multi-server, application-focused automation platforms. You can find solutions with established vendors like BMC or VMWare, with open-source platforms like Puppet with OpenStack, as well as with startups like ElasticBox. Bottom line, it is vast oversimplification to limit an overview of DevOps-capable automation to automation tools with a server-heritage only. This area of automation is clearly evolving and growing, and deserves a more holistic overview.

How does process fit in?

As John Willis, and others, have said many times before, culture and process are just as much a part of a devops approach as basic automation. So, it appropriate for Phil to end with a process-based approach. Clearly rolling out an application requires an understanding of the end-to-end process, how steps are related, and how services are dependent. I do feel that Phil left out a few key points

Process Management and Deployment Automation are not the same

I feel like Phil blurs the line between managing the process of release, which is a people-process issue, versus managing the deployment of an application. The latter involves pulling together disparate automation with a cross-server/application-focused view. Process management, on the other hand, deals with the more holistic problem of driving the release of an application from development all the way to production. They are both needed, but they aren’t the same thing.

What about coordination

One of the biggest drivers of DevOps is getting Dev and Ops to coordinate and collaborate on application releases. This means driving Dev visibility forward into Ops, and Ops visibility back into Dev. It isn’t just about creating well-aligned deployment processes, but also managing the entire release process from the code repository to production. This means we need encapsulate pre-prod and prod processes, as well as non-system activities (like opening change tickets, etc.).

What about planning

Releasing and managing applications is just about the here and now. It is also about planning for the future. Any process-oriented approach has to allow not only for the coordination of deployment processes, but also needs to allow for the establishment of clear and flexible release processes visible to all stakeholders. In particular, a process management system should provide visibility to the decision makers, as well as the executors. Applications clearly affect non-technical activities like marketing and executive planning, so it is important that business leaders be able to understand where releases are out, and when important features are likely to arrive.

What we need is a layered approach

Bottom line, we need to solve all of the application release issues – process, deployment, and automation. In the spirit of DevOps, those “layers” can be solved somewhat independently with loose coupling between them. We drive the people-process coordinate, encapsulate all of the complexities necessary to deploy an application, and then drive the low-level automation necessary to actually implement the application. All of these work together to create a full approach to Application Release Automation. Any solution that ignores a layer risks solving one problem, and then creating a whole new set of them.

IT Automation Curator for DevOps – Part 2 – Collect and Catalog

This topic is far too interesting and deep to cover in just one blog post. So, I am going to split the discussion into a few sections. I’ll use my proposed “job description” for an IT Automation Curator as a starting point:

  • Collect existing automation, and then Catalog it where others can find it
  • Develop new automation based on requirements from IT
  • Train others on how to use the automated processes
  • Maintain the existing automation

This first step of collect and catalog is where I have seen many automation efforts stumble. The natural inclination of most techies (myself included) is to jump right into developing automation, no matter what is in place. As I learned the hard way, that is a bad idea. So, I will give a few reasons why this step is important:

Reason #1: If you don’t know about all the automation in place, you don’t really understand how your data center is operating

It’s great that you developed that new automated process that auto-magically deploys a set of configurations for you. Are you sure that other scripts or tools won’t change it or corrupt it? Most IT teams have scripts strewn all over the place – some well known , some the detritus of sysadmins past. They may have been scheduled centrally or on individual servers. This is very hard to get a grip on. There are a few tools out there, but it is hard to ensure that you have found all automation spread over all the systems. This is just another reason why you need to control access and even re-build some servers from scratch (hopefully in an automated way).

Most IT operations teams also have multiple automation tools in play. Each silo-ed team has their preferred tool, which they guard jealously. Overall, this is not a good approach. The more tools you have, the harder it is to standardize automation and create efficient end-to-end processes. At a minimum, all of these tools need to documented and managed centrally.

Reason #2: Don’t duplicate work and ignore experience

A lot of the automation in place may not be optimal, but it was most likely built to solve the same problems you will need to solve later. Tossing it out, or just ignoring it, is essentially disregarding the combined experience of the IT team. Even if you rebuild it in a better tool, and in a more efficient way – the lessons learned will be valuable.

There is also an important less here about prioritization. Just because you can make an automated process more elegant or more efficient, doesn’t mean you should. More often than not you will have no end of automation projects to look at. Why spend your time on what already works? What is important is to apply automation judiciously, where it provides the most value for the business.

Reason #3: More sharing will always lead to better results

Fostering a culture of sharing automation, essentially an open-source culture, will ensure that everyone has access to the best work on offer, that they don’t re-invent the wheel, and it will allow for continual improvement. That last point is crucial. The idea is not for the automation curator to control all the automation per se. They should be catalysts for making better automation, whether they do it or not. So, it is important to leave one’s ego at the door, and admit that your automation becomes better when you let others critique it and improve on it.

Bottom line, having a central place to share and continually improve automation is essential. This will most likely affect your choice of automation platforms as well. If you can’t share and improve, then you will be hobbling yourselves.

So, how do you do this in your own environment? Do you have ideas about the best way to go about it? Any success stories?

IT Automation Curator – Good for techies, good for business, good for DevOps

Recently my thoughts have been going back to a concept I like in the seminal IT operations book, The Visible Ops Handbook (By Gene Kim, Kevin Behr, and George Spafford). I have been doing a lot of thinking about how Lean, DevOps, Agile, etc. are changing IT culture, or at least pressing for change. Properly leveraged automation is a big part of that change process – which makes me think of the passage in Visible Ops where the authors discuss changing the behavior of senior IT staff:

“Their mastery of configurations continually increases while they integrate it into documented and repeatable processes. We jokingly refer to this phenomenon as ‘turning firefighters into curators’ […]”*

As a former IT techie myself, I get the need to challenge oneself in the often routine and monotonous world of IT. Personally, I think that is a lot of the grass-roots impetus behind the DevOps movement, and the adoption of open-source automation tools. Creating automation is a way of turning the mind-numblingly mundane into something exciting and intellectually challenging. So far so good. Boredom leads to sinking morale and productivity – poor morale is bad for business.

So, what’s not to like? In short, it goes back to focus and sustainability. No, I’m not talking green-energy windmills. How do you sustain and focus the efforts of these budding automation aficionados? Left to their own devices, they will likely create lots of useful, but narrowly directed scripts, packages, etc. All of these will be focused on the problems they face on a daily basis. For the problems outside of the automation guru’s gaze – those problems will most likely remain unsolved.

So, this is where the idea from Visible Ops comes to the rescue. The answer is that we pull these gurus out of their day-to-day grind in the IT trenches,and make them automation curators. Now, I know that many of you hear curator and think of a older man in a tweed jacket, peering over horn rimmed glasses, waxing rhapsodic about the various manufacturer stamps of 18th American chamber pots. So, as interesting as early american port-a-potties may be, let’s look at the definition of curator:

curator – one who has the care and superintendence of something (Marriam-Webster Dictionary)

Clearly tweed is not mentioned. In all seriousness, museum curators do much more than merely talk about old things. Considering the Smithsonian’s own description, curators:

  • Acquire new items for the collection
  • Research the collection
  • Display the collection
  • Maintain the collection

So, if we work off the Smithsonian’s “model”, I suggest that an IT Automation Curators would:

  • Collect existing automation, and then Catalog it where others can find it
  • Develop new automation based on requirements from IT
  • Train others on how to use the automated processes
  • Maintain the existing automation

This kind of role is exactly what I missed someone had offered me early in my career. I would have jumped at it. It would have been a great new challenge for me, I would have been creating value for the business, and IT would have been more efficient. And this isn’t really a new idea. Software developers have long needed to share code snippets and concepts with each other, and they defined the interfaces between code as well. The trick here is that Automation Curator needs to take an active role in both building the best automation and also in promoting the proper use of automation in IT.

One last comment. We might ask if this would be better classified as an Automation Librarian. I think it is good question. At the end of the day, I think having the existence of the position is more important than what you call it. However, in my mind the concept of curator leans more towards the acquisition, development, and training part. The words Library and Librarian in IT seem to lean more towards the maintenance and storage part of the equation (notwithstanding what traditional librarians actually do). Curator is also a cool word.

So, why aren’t more IT shops doing this? What do you think?

This is the first part of a multi-part series. Check out the other parts:

* Kim, Gene; George Spafford; Kevin Behr (2005-06-15). The Visible Ops Handbook: Implementing ITIL in 4 Practical and Auditable Steps (Kindle Locations 917-919). IT Process Institute, Inc.. Kindle Edition.