The Mythical Application Owner – And Why They Matter to DevOps

I have been on both the customer and the vendor side of Application Management over the last decade. So, I was surprised how much trouble my team and I had really defining the Application Owner as a sales target. With our collective backgrounds, I expected it to be a breeze. Of course the CxOs, VP Operations, and others are well know entities. So why is the application owner so difficult to find? One reason is because the definitions are all over the place.

For example, one definition on the IT Law Wiki says:

An application owner is the individual or group with the responsibility to ensure that the program or programs, which make up the application, accomplish the specified objective or set of user requirements established for that application, including appropriate security safeguards.

Snore… I could be in charge of Microsoft Office for my company and fit that technically focused, but inherently logical, definition. NIST has a similarly technically oriented definition. Now in all seriousness, we can see that part of the problem is that I really mean a business application owner, not a technical owner. In that vein, a definition much more to my liking can be found on this blog entry from Nick Spanos on his Lean IT blog. In the spirit of lean methodology, he brings in business/customer outcomes, business processes, etc.

So, who cares? I think anyone who cares about DevOps should care. As DevOps expands from a grassroots movement to a business-changing phenomenon, it needs to continue to develop thought patterns that appeal to those footing the bill. This is where so much good thinking in IT falls down – the inability to articulate the value to the business. That said, I don’t think it has to be complicated. I see two over-riding characteristics:

They live, eat, and breathe application revenue

For every revenue-generating application out there, there is someone being measured on that revenue. For a small company, that might be the CEO. For a Fortune 500 company, there could be dozens of applications, each with an owner. Regardless of industry, somebody goes to sleep at night worrying about whether is measuring up to expectations.

They care (or should care) about customer value

Customers pay the bills, so worrying about revenue means obsessing about customers. I think that one of the common themes for companies successfully implementing DevOps is the over-riding need to deliver customer value. This is coincidently very much in line with lean methodology – which is why lean and DevOps are so good together. An application that is always in danger of losing customers and competitiveness is fertile ground for DevOps. If your application isn’t in a competitive space, why would you even bother?!

Pretty much everything else is going to be different by industry. An application owner at SuperCoolStartup dot com will likely be very different than the app owner at BigRetail dot com or the MBA educated person running SomethingOrOtherFinancial dot com. What they should have in common in the commitment to increasing revenue by increasing customer value.

Now, one caveat. Not every application has revenue. I would argue that the lack of the profit imperative makes a wrenching change like DevOps much more difficult, but you can insert “mission” or “fund raising” for most of these points.

So, why does this matter? DevOps practitioners, and IT organizations in general, have an unprecedented opportunity to make IT matter to the business more than it ever has before. IT can become an enabler for revenue, rather than a pure cost center. However, to do that, they need to learn to express the value of DevOps in revenue and customer-focused terms, and begin to make the case directly to the application owner. Over time, I think the pendulum of power is swinging from the CIO and VP operations to the application owners. So, IT needs to make some new friends!


Kaizen and the Art of DevOps Automation Maintenance

And now we come to the most “boring” part. Right? Maintenance. The death of joy for the innovator. Or is it? I don’t think so. Continuous innovation is at the core of DevOps and Lean methodology. Maintenance is essential to keeping the spirit of DevOps strong, and automation that isn’t improving will grow stale and useless.

So, let’s review the IT Automation Curator’s job description on last time:

  • Collect existing automation, and then Catalog it where others can find it (See Part 2)
  • Develop new automation based on requirements from IT (See Part 3)
  • Train others on how to use the automated processes (See Part 4)
  • Maintain the existing automation

Going back to Lean methodology, we can look to the idea of Continual Improvement or Kaizen. There are 3 main areas from Masaaki Imai‘s 1986 book Kaizen: The Key to Japan’s Competitive Success.

  • Reflection of processes. (Feedback)
  • Identification, reduction, and elimination of suboptimal processes. (Efficiency)
  • Incremental, continual steps rather than giant leaps. (Evolution)

Use Metrics and Reporting for Feedback

How can you improve if you don’t know how far you’ve come and how well you are doing? That’s like going on a diet without ever weighing yourself or even looking in the mirror. Healthy weight loss involves such small changes every week that it would discouraged if you didn’t look at changes over the long term (I speak from experience). So, why do so few companies and automation vendors include metrics and reporting on automation efficiency?! The whole point of implementing automation is to reduce waste, reduce costs, and increase velocity. But you have no context for understand if you have succeeded if you don’t have metrics and reporting (I talked about this in a previous post).

Ruthlessly eliminate sub-optimal processes

The whole point of this process is get rid of wasted effort and time – muda. The hardpart here is that once you agree to continually improve your automated processes, you have to be ruthless in your evaluation of their efficiency. That means there are no sacred cows. Just because somebody smart and dedicated invested hours of their life into creating something, doesn’t mean it can’t be improved or scrapped entirely. The whole team has to be dedicated to, and incentivized towards, efficiency and continuous improvement. This point is important – these kind of improvements come from the grassroots – not the top. If the people in the trenches aren’t bought into continual improvement, it won’t work.

Baby Steps, not Leaps of Faith

I know the hardest part of this approach for me is the gradualism. I like to grandiloquently solve grandiose problems with lofty and visionary solutions. The problem is that most of those involve large amounts of kool-aid, and they are never finished. The truly mature IT organization has to keep their eye on the goals of the business, and relentlessly reduce muda – step by tortuous step. We can refer back to the weight loss analogy. You lose weight through all of the small victories – Do I really need that donut? One serving is good enough. But bringing us back to our first point – small victories only show up as victories when you can measure your long term progress. Otherwise it looks like tentative, timid, risk-adverse behavior.

And so, we reach the last chapter of my IT Automation Curator series. It has been a lot of fun writing it, and I hope that you enjoyed it as well. I am looking forward to continuing to explore how the proven methods of lean and agile can be applied to DevOps and Operations overall.

Developing Automation with Lean Methodology

So, now we are on what I consider the most fun part of the IT Automation Curator’s job description ( see the first blog post ) – developing new automation. Let’s first review the job description as I outlined it:

  • Collect existing automation, and then Catalog it where others can find it (See Part 2)
  • Develop new automation based on requirements from IT
  • Train others on how to use the automated processes
  • Maintain the existing automation

There is lots of great technical material out there about creating automation, and I won’t try to duplicate it here. And I also won’t push a particular toolset, though I have my opinions, of course. I am convinced that most automation project fail, not because of problems with the toolset, but rather due to problems with the approach. That doesn’t mean that one tool isn’t better than another, quite the contrary. The difference is just how successful can you be when you do it right.

To Automate or not to Automate – that is the question.

I think choosing what to automate is much more important to how you choose to achieve the automation. The best model we have for that, in my opinion, is Lean Methodology, and Lean Software Development, in particular. The Poppendiecks, creators of Lean Software Development, have their first principle as Optimize the Whole. The whole point of this step is to eliminate waste, known as muda in Lean methodology. Muda is “anything which does not provide customer value”. This is such a simple, yet revolutionary, concept. What if every sysadmin asked himself whether that script he was working on added customer value? Would they even know how to answer the question.

So, how do you answer that question? You look at the whole, end-to-end, process. This means no silos, no team-centric thinking, no “that isn’t my job”. What does it take to deliver a service to the end user/customer, and where are we wasting time and resources? In Lean Methodology you figure this out by drawing a value stream map. In the context of manufacturing, the source of lean methodology, it meant going from factory to factory, supplier to supplier, and putting together the complete picture of how a product is built and delivered.

So, what does that mean to our Automation Curator? Typically lean, or agile, software development stops at the delivering of the product to IT Operations. In the spirit of DevOps, that needs to stop. IT organizations need to start looking at the delivery of a service to a customer from requirements all the way through to production. That is the only context where it makes any sense to uncover muda. If you view IT Operations in isolation, you could very well create a highly efficient IT operation that doesn’t good services.

The IT Automation Curator has to focus on the automation projects that provide value to the customer – delivering new applications and features faster, restores services faster, provides better customer experience, etc.

What does success look like?

Once you have automated a process, how do you even know that you have achieved your goal of eliminating waste? That where measurement and reporting come in. If you have no way of measuring the effects of your hard work, how do you even know if you have been successful? You don’t. That means part of developing any automation has to be about measuring the impact on the customer. For example, you just automated the full-stack build of an application from scratch. How much more quickly can you deliver updated applications, or recover from problems? So, you have automated the application release process. Did that reduce downtime and error during new releases?

Once you have those reports – advertise them. Don’t be shy about showing how you are increasing value for the end customer. So many IT departments have been forced to show value by how much they can cut out of their budget. How much more would the business appreciate an IT department that demonstrably increases value for the customer?

I think this approach is so much more effective for determining what to automate and how. This singular customer and business focus is bound to make the IT Automation Curator a valuable member of the team. A lot more valuable than the “IT Hero” that corrects a preventable problem with heroic effort. I am also hoping to see this result-focused approach take over from the tools-focused approach so prevalent in DevOps today. If DevOps is to be relevant for more than just a few years, it must encourage behavior that adds value to the customer. Why? Because the customer is King!

Can you do DevOps without automation?

I was reading Tom Parish’s interview Lori MacVittie on the DevOps Leadership Series, and I came across her statement that DevOps and automation are not, in fact, tied at the hip. I think that asking if DevOps needs automation to be successful is a really good question. In fact, it may be one of the essential questions – which comes first? Process or Automation? Lori clearly lands on the process side, and I tend to agree.

I always make the analogy to a hammer and nails. After repairing my fence last year, I discovered what a wonderful thing a nail gun is compared to a hammer and nails. What would have taken me 20-30 minutes, took me 5 minutes – bam, bam, bam. But nail guns are also dangerous. You can’t shoot yourself in the foot with a hammer… I think automation is a lot like that. Just automating something with no firm process in mind is like handing a nail gun to someone who immediately attached their foot to the floor – in record time. And I am not innocent of this. At one point in my career I was fascinated with the power of JScript on Windows when enabled with Active Directory (yes – I am geeking out – bear with me). I could write one script that would execute across 10s of servers in a way that was more difficult in UNIX at the time. But that was the problem. I tested a script on a live environment, and almost took out a live site (we won’t discuss how close I came to doing just that…).

Now, the counter point is that well-tested automation following a good process can’t beaten. I remember back in my BladeLogic days when a sysadmin was intent on convincing me that she could make the same change to 20 UNIX servers, one after another, and be more accurate than an automated tool. Doubtful. But say she could. That only makes sense on the smallest of scales. But if she had automated that, she could have spent her on something actually useful. Imagine that. And isn’t that the point, to make ourselves more productive, so we can make the business (and ourselves) more successful?

That is why applying a methodology like KanBan to DevOps is so promising. Instead of blindingly applying automation with cool tools, you look at your processes holistically, and apply automation systematically where it provides the most value. Which is exactly what manufacturing has done over the last few years.

So, yes. You can do DevOps without automation. But that’s the wrong question. Can you do DevOps without well-understood process and culture change? No. And don’t apply automation unless you tackle that as well.