Is an ROI metric an essential tool for project management or merely smoke and mirrors?

InfoWorld | Feb 16, 2006

You’ve deployed the technology. Now it’s time to gauge the payoff. In many organizations, winning support for your initiatives from senior management means demonstrating the ROI of your IT expenditures. But is ROI a valid standard for proving the value of IT projects?

David S. Linthicum, CEO of Bridgewerx and a noted application integration and SOA expert, says yes. In a world where business and technology are tightly aligned, Linthicum feels it is only natural that shareholders and senior management should expect hard numbers to demonstrate that their money is being well spent. But Gene Rogers, chief technologist for Transformational Space Systems at Boeing, sees it differently. According to Rogers, it’s far too easy to read too much into numbers that are ultimately open to broad interpretation.

David S. Linthicum: While everyone is bound and determined to build IT architecture within their enterprise — perhaps even between enterprises — few have attempted to determine their return on investment to justify their approach and their technology choices. While it’s appropriate to approach new technology with fervor and excitement, we also need to figure out the business case for this movement.

These days, IT architects and businesspeople must work hand-in-hand to determine if changes proposed are the proper course for a business. For example, the application of an SOA has different degrees of ROI, depending on the problem domain. The cost of implementing an SOA should be directly related to the benefits to the business, in both hard and soft dollars. Indeed, you can determine ROI, and should do so.

It’s easy. The ROI of IT can be presented in a general formula (see “Formula for ROI,” page 38). The first steps are determining the variables and making sure they are acceptable and reasonable in the context of the business. Some of them, such as “cost of inflexibility” and the “lack of agility,” are judgment calls.

Gene Rogers: The problem with any metric is that you maximize what you ask for at the expense of everything else. Be very careful what you ask for; you just might get it. Generally, optimizing any single metric or small cluster of metrics results in a sub-optimal solution overall. Management by algorithm is the least effective way to run a company.

In David’s example, the core issues of profit and loss and value to customers, investors, or shareholders is ignored. Ambiguous terms such as “cost of inflexibility” and “benefit of agility” are exactly the kind of indeterminate numbers that can be spun up or down by a clever manager who is inclined to game the system. In the meantime, while our clever manager is maximizing your metric-of-the-month, the core business is likely to be falling apart.

When IT reduced the cost of manual processing, for example, what happened to the freed-up employees? Was the automated end product equally as effective as the earlier manual product? These are all questions that an experienced executive will want answered.

DSL: I’m not sure that “management by algorithm” is a good way to put it. We use math as a tool to determine productivity and results, and we need to ultimately determine the ROI from IT to justify its existence. The metrics are going to differ a bit from business to business because what’s important in each business is always something different. Moreover, you always have to consider factors such as the value IT delivers directly to the customer, competitive advantages, the dynamic nature of the marketplace, and other factors.

As I said, you can indeed model this. The efficiencies of a well-run IT shop go right to the bottom line; this is proven.

Any ROI calculation needs to consider “soft” issues such as the customer experience and effects on the marketplace. Thus, the value to the shareholder is also a result of having a good ROI from IT. Customer satisfaction goes up? That’s all a part of the metric.

To your point, IT ROI should consider how IT affects the entire organization. Indeed, in some cases it should include the removal of humans who are doing slow, costly, and error-prone manual processing. If we can automate that, all the better. But you never achieve the ROI if that has a negative effect on the core business in any way.

GR: Certainly, the efficiencies (and inefficiencies) of IT go directly to the bottom line. But, just as you say, ROI calculations need to consider the “soft” issues, and few people can agree on what the key metrics are or how to effectively measure them.

I’ve managed technology development and implementation for a couple of decades, and from year to year I’ve been asked to demonstrate my program’s contribution to ROI in terms of profit margins, price-to-earnings, return on net assets, profit before taxes, profit after taxes, reinvestment, product transitions, growth forecasts, or even inventory turnover. Boards of directors reflect and tier down the metrics that are being tracked by current or prospective investors and shareholders. Next year, they’re guaranteed to focus on a new and different set of hot-button metrics.

Technical managers tend to rise through the technical ranks, where they prospered due to their left-brain skills in deductive analysis and rational problem solving. These skills apply well in the deterministic world, where there is only a single correct answer, but they may not be of much benefit in the indeterminate world of management and executive decision-making.

Management is a right-brain skill, requiring inference, an ability to handle ambiguity, and a talent for making sound decisions on the basis of incomplete data. Technical types prefer to be graded on metrics rather than judgments when they enter management because metrics are unambiguous and present a clear challenge. However, taking the measure of an operation is usually too complicated to be reduced to a couple of numbers.

Modern managers need to be comfortable with satisfying metrics that are flowed down to them but recognize that metrics are merely decision aids, not automated decision-makers, and certainly not a replacement for the skills, experience, and judgment offered by a strong executive. Consequently, the information captured in metrics — particularly soft metrics — can inform the decision-making process, but responsibility and accountability for making the right judgment calls and for their consequences always rests with the executive, not with the metric.

DSL: If you’re being asked for all the figures you’ve cited to demonstrate your program’s contribution to ROI, then the people you’re working for don’t get ROI. It is, as I said, the ability for us to identify the benefit to the business, and we can do this in terms of hard savings and benefits — real money — as well as soft savings and benefits, like something we can track back to making the business better (for example, better customer services).

The mistake people make is over-analysis of ROI. You need only a few key measurements that reflect the business and that you can leverage consistently over time.

You must learn how to quantify the benefits of having any business unit around, including marketing, sales, operations, and so on. It’s just good business to attribute value to those components of a business; IT is no different.

Indeed, in my new role as CEO, I’m finding that investors desire proof of value, and they won’t just take my word for it. This is not a trend, but these demands on owners, investors, and leaders are here to stay. You might as well accept it and learn how to determine the ROI for everything that requires resources.

To your point, in some cases it’s easier than others to determine the value of IT — or any other component to your business, for that matter. However, clearly, you can do it; it’s been proven time and time again. The question is not if you can do it, but should you do it. I’m pretty sure those who invest in companies, many of whom are providing IT with 25 percent of the company’s resources, will want to know how their money is being spent. I’m no different.

GR: As for whether we “get” ROI, I think a century of growth and profitability is a track record that few critics could match.

Even a board of directors has an ultimate boss — the investor and shareholder. As long as the investment community continues to migrate their attention from one trendy ROI metric to another, the corporate world will follow suit. The CTO or IT exec is generally given a set of performance targets that mandate which metrics will be valued this year. Rarely does this set remain constant from year to year.

So, as an IT executive, you have to manage the ROI hot-button metric as simply one of the many deliverable end products by which IT adds value to the company. However, don’t confuse satisfying the board and the investors with measuring the effectiveness of your IT organization.

In the most simplistic business scenario, IT costs and benefits may be clearly calculable, but nearly all real-world situations beyond the level of a lemonade stand are hazy because of the soft — difficult to quantify — cost/benefit variables that are unique to that business.

Every IT project has unique goals and targets that require unique KPIs (key performance indicators). Project-specific, time-phased IT KPI metrics flowed down from individual project goals and targets can effectively measure the success of a single project. The aggregation of these disparate KPIs reflects the success of the entire IT organization, but it is a complex data set, and as a decision aid it is subject to broad interpretation.

Metrics are a good thing as long as they are   interpreted as indicators, not precise measures, of performance. Allowing those metrics to drive executive decisions amounts to management by algorithm. Fuzzy numbers massaged by fuzzy logic lead to consistently bad decisions.