Consumer Reports is a U.S. magazine that is widely respected for the advice it provides to people who are facing difficult purchasing decisions for expensive items like autos, cameras, computers, televisions and washers. Their typical approach is to define the best product evaluation criteria to use (as well as their respective weights), and then use them to rate different options. Technically, this is known as multi-attribute utility analysis and in general it is a solid approach to making complex decisions. However, there are situations where it is not appropriate, at least in its traditional form. In these cases, the use of multi-attribute utility analysis runs the risk of producing answers that are at best misleading and sometimes flat-out wrong. Unfortunately, in its February, 2005 issue on choosing mutual funds, Consumer Reports appears to have made just this mistake.

The article starts out on a promising tack: It happens all the time. While reading the newspaper or watching a finance program on TV, you learn about a mutual fund whose returns are zooming upward. You check it out and learn that it outpaced the Standard and Poor’s 500 Index not only in the last year, but also for the last three years and five years. Congratulating yourself on finding a solid investment, you write out a check. Almost immediately, however, the fund plunges, and it stays in a performance trough for what seems like years. Why is it, you ask yourself, that every mutual fund you buy drops like a rock in a well the moment you invest? Don’t think for a moment that your experience is unique. Millions of [investors] wind up buying high and selling low. There’s an explanation: A mutual fund, like most other investments, may hit highs once in a while but perform listlessly most of the time. And the highs may be high enough to make the fund’s long-term returns look respectable…

The article goes on to quote an analyst from Morningstar, who notes that most funds that end up leading or lagging the pack for short periods are very concentrated in certain areas. Consumer Reports notes that as soon as these funds cool off, their performance chills. That’s why funds at the top of the annual performance charts seldom repeat from one year to next.

So far, so good. The article even goes on to note the potential advantages of investing in index funds: There are ways to avoid this investment roller coaster. One is to invest in index funds or exchange traded funds (ETFs). Both mirror the performance of a broad market index…You earn exactly what the market earns, minus expenses. Unfortunately, in the next sentence, Consumer Reports begins to go off the tracks: [But there is a] downside [to index funds]: You can never do better than the index, and when it drops, you’re stuck with losses.

Consumer Reports then proposes an alternative: Choose funds that outperform the market — not just once or twice a decade, but repeatedly. With such funds, you would not have to try to time market swings, a pursuit that baffles even professionals. The article then describes the approach Consumer Reports used to identify the actively managed mutual funds it recommends to investors: Starting with the 1,327 actively managed U.S. equity mutual funds that had at least a ten-year performance history, it applied the following screening criteria and weights: (1) The standard deviation of the fund’s monthly returns over five and ten years (40% weighting); (2) The fund’s excess return versus either the Standard and Poor’s 500 Index (for large cap, mid-cap, asset-allocation, and specialty funds) or the Russell 2000 Index (for small cap funds), weighted 40%; (3) The fund’s worst one and two year results, weighted 15%, and (4) The length of time the current manager has been managing the fund, weighted 5%. From the funds that passed this screen, Consumer Reports eliminated those with annual expenses of more than 1.6%, and those that received a failing fiduciary grade from Morningstar.

What does this mean in practical terms? Consumer Reports appears to making the assumption that people making investments are pursuing multiple objectives, including (a) beating the annual performance of a benchmark index; (b) minimizing the variability of annual returns; and (c) avoiding significant losses. Furthermore, it would appear that Consumer Reports assumes that most investors believe that low expenses, long manager tenure, and passing fiduciary grades are associated with the achievement of some or all of these objectives.

After applying these screening criteria, Consumer Reports was left with 70 funds (or 5.28% of the 1,327 it started with). Of these 70 funds, 18 were large cap (1 growth, 5 blend, and 12 value), 19 were mid-cap (7 blend, 12 value), 15 were small cap (6 blend and 9 value), 11 were asset allocation funds (that switch between bonds and equities), 4 were real estate investment trusts (REITS), and three were industry sector funds (one each in health care, defense, and natural resources).

Saying that we disagree with Consumer Reports ‘ analytical methodology and conclusions is, to put it mildly, a vast understatement.

Let us start with the nature of the investor objectives assumed by Consumer Reports. As we have repeatedly written, we do not believe that beating the annual performance of a benchmark index should be the primary goal of prudent investors. Rather, these investors would be much better off if they focused on achieving a long-term (i.e. a compound annual, or, to put it differently, a geometric average) rate of return that is sufficient to achieve their goals over their target time horizon. This is equivalent to saying that a pension fund should focus on adequately funding its long-term liabilities (at the lowest possible risk) rather than beating a market index.

We have noted many times that the reference point you use to measure your investment performance is critical. Daniel Kahneman won the 2002 Nobel Prize in Economics for his writing on this subject. To oversimplify, a key insight is that for most people, losing hurts twice as much as winning feels good. If you use the annual performance of an index as your benchmark, you will regularly be tempted to bail out of investments that have had a bad year, and move them into investments that have just had a good one. More often than not, this leads to the dreaded buy high, sell low outcome.

The alternative is to focus not on the performance of individual asset classes versus a benchmark (or benchmarks), but rather on the overall performance of your portfolio compared to your long-term goals. This forces you evaluate your performance over longer periods of time than one year, and substantially reduces the temptation to act precipitously (and often harmfully) on the basis of relatively little information.

However, if we take Consumer Reports ‘ annually beat the index goal as a given, we have another problem with their choice of minimizing the total standard deviation of a fund’s returns as an objective. In point of fact, the returns on the actively managed funds cited by Consumer Reports are a function of two factors: the style tilt they take away from the broad equity market index (e.g. toward small caps. and/or value), and manager skill. I can obtain the benefits of the style tilt quite cheaply by investing in a fund that tracks an appropriate index (e.g. a small cap value index fund). Therefore, the real issue is how much additional return the actively managed fund provides relative to the additional risks it takes to obtain it. What counts is not the overall volatility (standard deviation) of the actively managed fund, but rather the volatility of its returns compared to those on the index fund. In technical terms, returns relative to the index fund are called alpha, and the additional risk relative to the index fund is called tracking error (which is the standard deviation of the alphas). Consumer Reports made a major error in using the alphas of its actively managed funds and their total volatility. The correct approach would have been to relate their alphas to their tracking errors. This is called the Information Ratio, and it is quite well known. We don’t understand why Consumer Reports did not use it.

We also question the other two screening criteria used by Consumer Reports. What does the fact that a manager has been with a mutual fund for a long time really tell you? That he or she has yet to get a high enough offer from a hedge fund? Or that he or she doesn’t want to go to a hedge fund because — why? Fear that he or she is losing his or her edge? Or something else? Who really knows? On the other hand, we agree with Consumer Reports ‘ implicit assumption that investors’ risk preferences encompass more than volatility (standard deviation). Kahneman’s work showed that they also prefer to avoid large drawdowns, or negative returns (technically, this is a function of skewness and kurtosis). However, we disagree with Consumer Reports apparent view that this preference should be taken into account at the fund level. Rather, it should apply to portfolio returns, which include the beneficial impact of diversification across asset classes.

Another major problem we have with the Consumer Reports article is its unstated assumption that successful active management is possible for great numbers of people (if only they had the sense to use the Consumer Reports funds screening process). We have written at length about the many challenges that make consistent active management success such a rarity (see, for example, in the Members’ Section of our site The Case for Active Management ). Let me briefly summarize them. First, successful active management ultimately rests on successful forecasting. In turn, this must rest on some combination of superior information and/or a superior model for making sense of it. Active managers’ ability to consistently obtain superior information has been dramatically reduced due to both technological changes (e.g. the internet) and regulatory changes (e.g. Securities and Exchange Commission Regulation FD in the United States). As for superior models, most are eventually made obsolete by some combination of copying by competitors and/or changes in the underlying economy that invalidate their key assumptions.

Second, even those managers in possession of superior information or a superior model often find themselves unable to fully act on it. For example, most mutual fund managers are prevented from taking short positions in a security, or from adding leverage to maximize the benefits from their insights. Other active managers face constraints on the maximum amount of active risk they can take in pursuit of better than index fund returns. Finally, as people pour more money into high performing mutual funds, their managers typically find it exponentially more difficult to find ever larger, but still highly profitable investment opportunities.

Of course, many of these limitations do not apply to hedge funds (though the challenge of managing a larger fund certainly does). However, even if you have sufficient wealth to qualify as a hedge fund investor, you are still left with the challenge of identifying truly skilled active managers. Unfortunately, research has shown that impressive track records aren’t any help. Many of them are not statistically different from what would expect from luck alone, and even those that are have been shown to be poor predictors of future performance (like we said, advantages based on superior information and superior models have increasingly short half-lives). The inescapable reality is that it is much easier to identify Warren Buffet than it is his successor.

But suppose you think you have. You are still left with the fourth challenge: negotiating a compensation arrangement with this superstar active manager that will leave you feeling confident that you will be better off than you would have been had you invested in an index fund. Given that many hedge fund managers charge fees equal to 2% of their assets under management, plus 20% of the profits they generate, this is no easy task (though it certainly explains why so many mutual fund managers have left to run hedge funds!).

In short, one reaction to the Consumer Reports article is so what? They have identified a small number of funds that, by their standards, have outperformed over the past ten years. Our response is that plenty of research shows that this provides no statistically significant insight about whether they will outperform over the next ten years.

But let’s go a step further and ask whether Consumer Reports ‘ 70 star mutual funds have actually outperformed. The first thing we insist on is a fair, apples to apples comparison. Unfortunately, Consumer Reports mostly doesn’t provide one. For example, included in the 70 funds are 11 asset allocation funds that switch between stocks and bonds. That they have outperformed the S&P 500 Index tells us very little. It may well have been the case that all of them would have been outperformed by a mix of stock and bond index funds. The only fair way to make this comparison would have been to (1) identify each of these fund’s basic stock/bond asset allocation (also known as its policy mix), (2) construct an index based on this mix, and then (3) see if its active management (i.e. departures from its policy mix) caused the fund to outperform the index fund benchmark. Unfortunately, Consumer Reports did not do this analysis, so we will eliminate the asset allocation funds from out analysis.

We will also eliminate the REIT funds, as they are, in our view, a separate asset class from the domestic equity funds. We will also eliminate the three narrowly focused sector funds, as their performance was not compared to an equally narrow index.

This leaves us with 52 general equity funds. As we already noted, Consumer Reports measured these funds’ excess returns (i.e. their alpha) using either the Standard and Poors 500 or the Russell 2000 Index. We believe this is an incorrect methodology. Specifically, it conflates true active management skill with the benefits of style tilts that could have been taken, at much lower cost, via index funds. For example, the proper benchmark for an actively managed, large cap value fund is not the S&P 500 (which includes a mix of growth and value companies); rather, it is a fund that tracks a large cap value index. In our analysis, we used the following funds as benchmarks:

Equity Style Tilt