Timothy Lee’s case for index-fund investing is obviously one that I’m familiar with, but I’ve never heard an adequate answer to my fundamental question: what’s the difference between a manager of a fund and a manager of an index?

Indices, after all, don’t create themselves. Employees in the Standard and Poor’s division of McGraw-Hill are responsible for choosing which stocks to include in each of their numerous indices — and which to exclude. They have algorithms for making these selections which take into account a variety of factors (since the index is supposed to be both representative of the market and readily tradable), and no doubt there is some degree of human discretion involved as well. At a 20,000-foot level, the index manager is picking stocks: which ones to buy, and which to sell. Which is exactly what the manager of an equity mutual fund is doing. So what’s the difference?

A huge difference is that the equity mutual fund manager has a financial incentive tied — in some measure — to performance. The people who select stocks for the S&P SmallCap Value Index are perceived to be performing a technical operation — selecting stocks that fit certain criteria such that the resulting index replicates the performance of a particular segment of the market. They are not trying to make money. And they are not compensated as if they are. By contrast, the manager of a small-cap value fund is supposed to be making money for her clients. If she does very well, she’ll probably get a bigger bonus; moreover, she’ll probably attract more capital, which will increase her compensation further; and, finally, her career prospects will be improved.

We’d expect that an incentive structure like this would drive improved performance. But, as Lee notes, the opposite is true: the managers who are compensated on performance do worse, for their clients, than the managers who are not. This should be a surprise.

Moreover, it’s worth thinking about the free-riding problem with index funds. Consider what would happen if all funds invested in the stock market were indexed. What, in that case, would drive prices properly to reflect the prospects of the various companies? What moves markets to equilibrium is the action of market-makers seeking to find the price at which buyers and sellers balance out. If all buyers and sellers were index funds, all stocks would move up and down together, because the only activity would be generated by net inflows or outflows to or from index funds, plus the sudden jumps associated with adding or dropping a name from the index. In fact, if you take the example seriously, investing in index funds would be impossible if nobody invested except through such funds, because there would be no buyers to sell to, and no owners to buy from, except other index funds. Who will always be going in the same direction. Index funds, then, are only able to operate because of the existence of liquid markets, which are themselves dependent on large numbers of buyers and sellers operating from premises other than those of the indexers.

If, on average, mutual funds underperform the market and, on average, index funds match market performance (that’s their goal, after all), then there is some other set of market participants who are, on average, outperforming the market. Presumably, these are professionals with some advantage relative to mutual fund managers — superior market information, superior execution capabilities, etc. The focus of equity market reforms since the 1980s has been to police these advantages — deregulating commissions, allowing for the development of crossing networks (used by the biggest mutual funds as well as other large investors) in order to avoid paying a full spread to market-makers and specialists, etc. But these reforms do not appear to have improved the performance of active mutual funds; rather, the advantages accruing to professional traders (such as high-frequency trading shops) only appear to have grown, with the cost being born by other market participants.

The relative outperformance of index funds is frequently told as a triumphant story about market efficiency. If the market is efficient, then on average nobody can beat it, so why bother trying. But this is not the real story. If it were, then on average professionally-managed funds would deliver average performance. They would beat the index half the time and would be beaten by it half the time. Indeed, given that there are still individuals who play the market, who you would expect to be at a consistent information disadvantage relative to professional managers, you’d expect that, in an efficient market, managed funds would generally beat the market, indices would match it, and individual investors would underperform. Choosing an actively managed fund would mean taking more risk — some funds will outperform while others will underperform relative to a given index benchmark — in exchange for a higher expected return. The fact that, on average and over long periods of time, managed funds underperform net of expenses suggests that there’s something wrong with the incentive structure. And yet, at first glance the incentive structure looks proper.

For an individual investor, opting for an index fund may be an adequate solution. But for a policymaker, it’s totally inadequate, because indices can only exist in an ecosystem dominated by non-indexed stock investors. If that ecosystem is dominated by investment propositions that amount to scams, that’s a serious problem.