There is one last thing that ITPA is not. And that is glueware for large management suites.
Every large vendor of IT management solutions has a big problem that has been hounding them without resolution for many years. That is that they grew their portfolio of products through a combination of organic engineering and acquisition (mostly acquisition) and as a result ended up with a collection of products that are not well integrated. Whenever they make an acquisition they always intend to integrate it properly with the rest of their portfolio but it never happens, at least not in any meaningful way. And the reason for that is simple, the economics don't work. The number of integrations increases exponentially (technically factorially) with each new addition to the suite, but revenue associated with solving the problem does not. And so since these companies are businesses the projects do not proceed. However, after years of this both the customers and the analysts have become unruly and demanded true integration. Further, these companies have begun to view the problem more strategically, viewing true integration as a real means to drive adoption of entire suites rather than point products (often referred to as ERP for IT). But with so much catch up work to do what is one to do? Several years ago a number of these vendors embarked on SOA initiatives to solve the problem. None of them got there, largely because the self interest of individual product groups outweighed the architectural ideal but also because it was an imperfect solution. Then along came ITPA. It was an elegant solution to the problem. Build one integration for each of your tools, plug it into a workflow engine, and voila problem solved. But there was a fundamental problem with this approach and my boss, Todd DeLaughter sums it up most eloquently. He likes to say that suite integration is the vendors problem but it's not the customers problem. The customers problem is the need to automate their IT processes across their entire set of tools and unless they buy all of their tools from one vendor (and as much as large vendors would like this to be the case - ERP for IT - it is not nor will it be) they need a solution that operates across heterogeneous multi-vendor environments. They also need the solution to operate across more domains than any one vendor offers - for example across networks, servers, and storage. And so when large vendors build or buy ITPA technology and they use it to glue together their suite they think that they are doing something important, in fact their customers and the analysts have demanded it and they also think it will drive revenue - but in fact they are not solving the problem the market needs solved. When they run up against a customer who demands third party integration they will make a token gesture of support - typically a services delivered script with no more functionality than what will solve the initial use case - but that's a band-aid not a cure.
The situation only gets worse when these same vendors begin to bundle their ITPA technology with everything they sell. Again, they think they are doing good. After all they are adding automation to existing solutions, often at little incremental license cost to the user. But the largest costs in a data center are not license costs. Not even close. The majority of cost in the data center comes from managing complexity. And having little bits of automation peppered throughout each solution does not address the core complexity. The real solution that the market needs is a complete one that automates across the complexity, across the heterogeneous environments. But because the large vendors don't have this nor do they intend to deliver it they put bits of automation everywhere.
So what you can expect to get from the large vendors is a solution that integrates their own tools, delivered in pieces. What you need is a solution that automates across complex heterogeneous environments delivering true end to end automation and driving down complexity and therefore cost. You're only going to get that from an independent vendor who's been investing heavily in their technology for a decade. You will not get it from someone who's only been in the market a couple of years either because it's a hard problem that takes just a lot of time and effort to solve rather than a magic algorithm.
Sunday, July 19, 2009
ITPA Defined (Part Two)
BPM (Business Process Management) is another thing that is not ITPA.
Many of the BPM vendors, having long saturated their primary market, are looking for new markets in which to apply their technology and for a lot of reasons ITPA seems to look like the promised land. First of all it's a new market so it's nowhere near saturated (current estimates are that less than 10% of those who will buy ITPA in the next 5 years have already bought). Second while there are several early leaders there is not yet a $100M gorilla in this market
(not even HP or BMC who are rolling a copy of their ITPA products into everything they sell - but more on this later). Third predicted growth for this market is explosive (as far as corporate IT goes) with revenue growing from less than $100M to more than $700M in the next 5 years. Finally, the technology is nearly the same so it will take very little incremental time and money to enter the market. I was with them until that last point.
At first glance you might think that it is a small leap from BPM to ITPA.
Workflow engine - check.
Graphical designer - check.
Operations console - sort of. close enough. sure why not. check.
Integrations - supports SOA so integrates with everything. check.
Wrong. Way wrong.
I love the idea of SOA. I believe in the potential of SOA. I even think that many new applications built on SOA standards fulfill the promise of the architecture. But I am at the end of the day a pragmatist and I realize that when it comes to integrating with the highly complex and heterogeneous infrastructure found in a large datacenter it has a long way to go. In a typical datacenter we're dealing with a mix of operating systems, application stacks, management tools, etc... that defies description. This is the single largest reason why IT costs so much and why effective service delivery is so hard. It's also the primary raison d'etre for IT Process Automation tools. Mainframes are naturally very highly automated systems because they are by definition homogenous (one operating system, one console, one log, etc...). But distributed systems broke this paradigm. No single tool is capable of automating across every technology in your data center. And no suite of tools from a single vendor can do it either (they are either not broad enough or not deep enough or more likely both). Instead the only workable solution is to take the point tools that you are using to automate within each domain today and orchestrate them into end to end processes. This is the fundamental innovation in ITPA. And in order to get there you need more than a workflow engine. You need a complete set of integrations, both broad AND deep, into everything you use to manage your datacenter. And (finally getting back to my original point) not everything in your data center is SOA enabled. Not now, not in the next 5 years, which is eternity in IT. The simple truth is that the vast majority of the tools you have in your datacenter pre-date SOA and so they either lack web services interfaces entirely or they had them added on afterwards. Even those with web services interfaces will often find those interfaces less complete than the traditional APIs or lacking in some other way such as performance or reliability. How do I know this with such certainty? The team at Opalis has built over 40 integrations to data center tools and we know this stuff better than anybody else. I would estimate that less than 25% of our integrations use pure web services to do their job for mostly the reasons I listed above. So SOA is not the magic bullet for data center integrations in 2009 and probably won't be for some time. So all the BPM vendors have to do is build some integrations, right? Wrong. First of all building a comprehensive suite of meaningful integrations is one of the biggest barriers to entry in this market. I don't mean a bunch of scripts carried around by services teams and customized for every customer and left for them to maintain. I mean productized integrations that have advanced features like configuration discovery and integrated data handling. Second, in it's most usual form BPM/SOA implementations force a level of abstraction between the workflow and the integrations that fundamentally break the cardinal rule of ITPA - don't force the users to write code! Frankly I have yet to see a BPM-based ITPA contendor who has effectively solved the challenge of integration. I'm sure they will eventually but by then the customers will take integration for granted and the true ITPA innovators will have moved on to the next thing (at Opalis we're already several next things beyond this problem).
But let them try. Competition drives innovation. And innovation is good for everyone.
Many of the BPM vendors, having long saturated their primary market, are looking for new markets in which to apply their technology and for a lot of reasons ITPA seems to look like the promised land. First of all it's a new market so it's nowhere near saturated (current estimates are that less than 10% of those who will buy ITPA in the next 5 years have already bought). Second while there are several early leaders there is not yet a $100M gorilla in this market
(not even HP or BMC who are rolling a copy of their ITPA products into everything they sell - but more on this later). Third predicted growth for this market is explosive (as far as corporate IT goes) with revenue growing from less than $100M to more than $700M in the next 5 years. Finally, the technology is nearly the same so it will take very little incremental time and money to enter the market. I was with them until that last point.
At first glance you might think that it is a small leap from BPM to ITPA.
Workflow engine - check.
Graphical designer - check.
Operations console - sort of. close enough. sure why not. check.
Integrations - supports SOA so integrates with everything. check.
Wrong. Way wrong.
I love the idea of SOA. I believe in the potential of SOA. I even think that many new applications built on SOA standards fulfill the promise of the architecture. But I am at the end of the day a pragmatist and I realize that when it comes to integrating with the highly complex and heterogeneous infrastructure found in a large datacenter it has a long way to go. In a typical datacenter we're dealing with a mix of operating systems, application stacks, management tools, etc... that defies description. This is the single largest reason why IT costs so much and why effective service delivery is so hard. It's also the primary raison d'etre for IT Process Automation tools. Mainframes are naturally very highly automated systems because they are by definition homogenous (one operating system, one console, one log, etc...). But distributed systems broke this paradigm. No single tool is capable of automating across every technology in your data center. And no suite of tools from a single vendor can do it either (they are either not broad enough or not deep enough or more likely both). Instead the only workable solution is to take the point tools that you are using to automate within each domain today and orchestrate them into end to end processes. This is the fundamental innovation in ITPA. And in order to get there you need more than a workflow engine. You need a complete set of integrations, both broad AND deep, into everything you use to manage your datacenter. And (finally getting back to my original point) not everything in your data center is SOA enabled. Not now, not in the next 5 years, which is eternity in IT. The simple truth is that the vast majority of the tools you have in your datacenter pre-date SOA and so they either lack web services interfaces entirely or they had them added on afterwards. Even those with web services interfaces will often find those interfaces less complete than the traditional APIs or lacking in some other way such as performance or reliability. How do I know this with such certainty? The team at Opalis has built over 40 integrations to data center tools and we know this stuff better than anybody else. I would estimate that less than 25% of our integrations use pure web services to do their job for mostly the reasons I listed above. So SOA is not the magic bullet for data center integrations in 2009 and probably won't be for some time. So all the BPM vendors have to do is build some integrations, right? Wrong. First of all building a comprehensive suite of meaningful integrations is one of the biggest barriers to entry in this market. I don't mean a bunch of scripts carried around by services teams and customized for every customer and left for them to maintain. I mean productized integrations that have advanced features like configuration discovery and integrated data handling. Second, in it's most usual form BPM/SOA implementations force a level of abstraction between the workflow and the integrations that fundamentally break the cardinal rule of ITPA - don't force the users to write code! Frankly I have yet to see a BPM-based ITPA contendor who has effectively solved the challenge of integration. I'm sure they will eventually but by then the customers will take integration for granted and the true ITPA innovators will have moved on to the next thing (at Opalis we're already several next things beyond this problem).
But let them try. Competition drives innovation. And innovation is good for everyone.
ITPA Defined (Sort Of)
Like Data Center Automation, the term IT Process Automation (ITPA) is often misused. So much so that in order to define it I will begin by telling you what it isn't.
It is not job scheduling, or workload automation, or whatever new term is being used to describe the category of tools historically used to automate batch processing. Trust me on this one, I spent a decade in the job scheduling business building and selling some of the most innovative and successful products in the market. These tools are valuable and essential parts of any IT automation strategy but they do not fulfill the needs for which dedicated ITPA tools are intended. Specifically although their automation processes look like workflows they are not. Job schedulers do not execute workflows, they manage dependencies between jobs. This is a subtle but critical distinction. Because they do not contain full workflow implementations they cannot represent complex logic (the kind that define IT processes) without layering drastic amounts of scripts and code on top. They also do not have the appropriate sorts of integrations (or adapters, or in job scheduling parlance agents). True enterprise class job schedulers do a great job at covering many operating systems from Windows to the mainframe and they have very deep integration into commercial ERP applications like SAP and Oracle but they do not integrate broadly or deeply with the IT tools in your datacenter such as event monitors, service desks, virtualization, and provisioning and configuration tools. Without these integrations you cannot automate your IT processes without layering scripts and code on top (are you beginning to see a pattern?). Finally job schedulers typically excuse themselves entirely from the task of handling the data passing between jobs in a flow or in ITPA terminology activities in a workflow. They may offer simple symbolic variable substitution but for the most part they assume that the data between jobs will be passed either in a database or an external file system. Any ITPA solution that does not offer native data handling that goes beyond simple variables will necessitate an inordinate amount of (you guessed it) scripting and coding in order to make it work. There are many other differences between job schedulers and ITPA solutions (including many things that job schedulers do better than ITPA like complex calendar handling and forecasting) and there are in fact many similiarities (especially from an enterprise architecture perspective), but these three major differences - workflow, integrations, and data handling - define the major differences between the two. These are not simply features to be added to job schedulers to transform them into ITPA solutions thereby finding a new market for mature products, they are genetic differences that define separate species. So if someone tries to sell you a job scheduler (under whatever banner) to automate your IT processes simply ask him or her how it handles complex workflow, integration with standard IT tools, and integrated data handling, without additional scripts or code. If they have a satisfactory answer then they will have cracked a code I thought uncrackable. If not, then you'll have saved yourself a lot of wasted time, effort, and perhaps money.
It is not job scheduling, or workload automation, or whatever new term is being used to describe the category of tools historically used to automate batch processing. Trust me on this one, I spent a decade in the job scheduling business building and selling some of the most innovative and successful products in the market. These tools are valuable and essential parts of any IT automation strategy but they do not fulfill the needs for which dedicated ITPA tools are intended. Specifically although their automation processes look like workflows they are not. Job schedulers do not execute workflows, they manage dependencies between jobs. This is a subtle but critical distinction. Because they do not contain full workflow implementations they cannot represent complex logic (the kind that define IT processes) without layering drastic amounts of scripts and code on top. They also do not have the appropriate sorts of integrations (or adapters, or in job scheduling parlance agents). True enterprise class job schedulers do a great job at covering many operating systems from Windows to the mainframe and they have very deep integration into commercial ERP applications like SAP and Oracle but they do not integrate broadly or deeply with the IT tools in your datacenter such as event monitors, service desks, virtualization, and provisioning and configuration tools. Without these integrations you cannot automate your IT processes without layering scripts and code on top (are you beginning to see a pattern?). Finally job schedulers typically excuse themselves entirely from the task of handling the data passing between jobs in a flow or in ITPA terminology activities in a workflow. They may offer simple symbolic variable substitution but for the most part they assume that the data between jobs will be passed either in a database or an external file system. Any ITPA solution that does not offer native data handling that goes beyond simple variables will necessitate an inordinate amount of (you guessed it) scripting and coding in order to make it work. There are many other differences between job schedulers and ITPA solutions (including many things that job schedulers do better than ITPA like complex calendar handling and forecasting) and there are in fact many similiarities (especially from an enterprise architecture perspective), but these three major differences - workflow, integrations, and data handling - define the major differences between the two. These are not simply features to be added to job schedulers to transform them into ITPA solutions thereby finding a new market for mature products, they are genetic differences that define separate species. So if someone tries to sell you a job scheduler (under whatever banner) to automate your IT processes simply ask him or her how it handles complex workflow, integration with standard IT tools, and integrated data handling, without additional scripts or code. If they have a satisfactory answer then they will have cracked a code I thought uncrackable. If not, then you'll have saved yourself a lot of wasted time, effort, and perhaps money.
Wednesday, May 6, 2009
First an introduction
First of all let me introduce myself.
My name is Charles Crouchman and I'm the CTO for the leading independent provider of IT Process Automation software, Opalis. I've been designing and building software for a bit more than two decades and I've spent the last decade or so building commercial data center automation solutions that are now used in many of the largest and most advanced data centers in the world. And so not surprisingly my intent for this blog is to write about what I know best and that is (you guessed it!) data center automation.
Like many terms in the technology business data center automation is often misused and abused leading to confusion about what it really means. Most vendors will define data center automation as "Whatever it is I happen to be selling." Do a search on the term and you'll see what I mean. The definition that I like best is the one from Evelyn Hubbert from Forrester:
"Data center automation combines methods that enable hardware, software, and processes to work together, streamlining IT operations. It automates highly manual processes, which assists both the IT operations and IT service management teams in delivering services from design to operations and maintenance. Data center automation sits conceptually between the IT service operation and IT service management processes and reduces work for members of the IT service operation team, resulting in improved efficiency and reduced human error."
Evelyn Hubbert, Senior Analyst, Forrester Research
Data Center Automation Defined, February 26, 2008
But the best part of her definition is that she goes on to explicitly define the four product categories that are subcomponents of data center automation. These are:
My name is Charles Crouchman and I'm the CTO for the leading independent provider of IT Process Automation software, Opalis. I've been designing and building software for a bit more than two decades and I've spent the last decade or so building commercial data center automation solutions that are now used in many of the largest and most advanced data centers in the world. And so not surprisingly my intent for this blog is to write about what I know best and that is (you guessed it!) data center automation.
Like many terms in the technology business data center automation is often misused and abused leading to confusion about what it really means. Most vendors will define data center automation as "Whatever it is I happen to be selling." Do a search on the term and you'll see what I mean. The definition that I like best is the one from Evelyn Hubbert from Forrester:
"Data center automation combines methods that enable hardware, software, and processes to work together, streamlining IT operations. It automates highly manual processes, which assists both the IT operations and IT service management teams in delivering services from design to operations and maintenance. Data center automation sits conceptually between the IT service operation and IT service management processes and reduces work for members of the IT service operation team, resulting in improved efficiency and reduced human error."
Evelyn Hubbert, Senior Analyst, Forrester Research
Data Center Automation Defined, February 26, 2008
But the best part of her definition is that she goes on to explicitly define the four product categories that are subcomponents of data center automation. These are:
- Automatic discovery of IT infrastructure components.
- Change and configuration management.
- IT process automation.
- Audit and control.
I tend to agree with Evelyn that when using the term "Data Center Automation" most people are referring to one or several of these four areas.
Now because I happen to spend most of my days (and far too many of my nights) thinking about and delivering IT Process Automation solutions this is where I intend to spend most of my time in future blog posts. I hope you find them useful.
Subscribe to:
Posts (Atom)
Posts
