By Jay D. Franklin and Mark Hebner

Tuesday, November 19, 2013

This is an article about statistical significance. If those two words sound like finger nails on a chalkboard to you, you may want to skip this article or find a friend who can assist you in understanding it.

In IFA’s opinion, this type of analysis is very important for investors because it is a common academic method to quantify the existence of manager stock picking skill. Stated another way, it addresses the question of whether a manager’s past performance was the result of skill or luck. This is very important because investors should expect skill to persist in the future and luck to fade away. Other than academic papers and information from IFA or Dimensional Fund Advisors, we have yet to see any financial services company, financial journalist, or fund rating service such as Morningstar or Lipper, discuss the standard deviation of the alpha, the sample size needed to distinguish luck from skill or the t-Statistic of the Alpha. In statistical analysis, a number without a t-statistic of 2 or more is not a reliable number. In our review of active management investment returns, a t-statistic of 2, which indicates a 95% confidence level, is rare. Please review the formula and input form at this link to better understand this type of analysis. This article titled What’s the Significance?. is for the advanced students of financial analysis. Finally, this article titled The Paradox of Skill provides further background information on the question of stock picking skill.

The Mutual Fund Alpha Charts shown below compare 37 mutual funds from American Funds, all A Shares, to their Morningstar analyst-designated benchmark. Please note that the Morningstar Benchmark may differ from the benchmark that the managers of the mutual fund chose. Morningstar attempts to select benchmarks that, in their opinion, more closely approximates the asset allocation of the fund’s holdings. In some cases (such as the Growth Fund of America) the mutual fund data goes back further than the benchmark data, so the alphas for the years prior to the existence of the benchmark are not included in the Alpha Charts. Also note that funds with less than three full calendar years of data are excluded from this article. Average Alpha is defined as the average of the annual return differences between the fund return and the benchmark return. Alpha is commonly thought of as the value added by active managers.

For the funds with positive Average Alpha (shown in the bottom left corner of each Alpha Chart), the Years Needed (shown in the bottom right corner of each Alpha Chart) tells us the number of years required of similar average alphas and similar volatilties of alphas (standard deviations) to conclude the manager(s) had skill. When the number of years of fund data, shown in the gold bar subtitle, exceeds the number of Years Needed, we can be about 95% confident the alpha is due to skill rather than luck.

Of these 37 American funds, 35% (or 13 funds) had positive Average Alpha over the period, and only 5.4% (or 2 funds) had a consistently high enough alpha, and a low enough standard deviation of alpha, and a large enough sample size of data that we could be about 95% confident the alpha was due to skill rather than luck. Which still leaves room for a 5% error in certainty, or a 1 in 20 chance of concluding a manager had skill, when in fact it was just luck.

For 11 of the 13 funds with positive alpha, the number of years needed to confirm skill exceeded the number of years of historical returns shown in the subtitle, sometimes by a very wide margin. While having two funds (ANWPX and CWGIX. both funds have Front Loads of 5.75%) with alpha apparently attributable to skill may be considered praiseworthy, it is diluted by the fact that out of 37 different funds, we would expect to see at least two that have statistically significant positive alpha, with about a 95% level of certainty.

This is similar to the findings of Barras, Scalliet, and Wermers in False Discoveries in Mutual Fund Performance: Measuring Luck in Estimating Alphas. When a mutual fund manager has a statistically significant different performance than the fund’s benchmark (i.e. positive or negative alpha with a t-stat of 2 or more), there are two possible explanations: skill or luck. The authors define a false discovery as a mutual fund that exhibits significant alpha by luck alone. Using a sample of 2,076 actively managed US equity funds between 1975 and 2006, the authors found that total observed alpha is consistent with the following breakdown of the population: 75.4% of the funds have a true alpha of zero after costs and 24.0% have a true alpha that is negative, which leaves only 0.6% with a true positive alpha, a number that the authors consider to be statistically indistinguishable from zero. IFA’s Step 3: Stock Pickers admonishes investors not to pick stocks for themselves or expect an active manager of a mutual fund to add value by picking stocks on their behalf. The chart below summarizes this paper.

Putting aside the question of luck vs. skill, when an actively managed fund exhibits positive alpha, a current or prospective investor should be concerned with the source of that alphawhether it came from superior security selection, market-timing, or style-picking. For one of the two funds with a t-statistic greater than two, the chart below (generated with Morningstar data) suggests that both market-timing and style drift occurred over the life of the fund. At times, the fund appears to have been up to 20% cash or short-term bonds, and the tilt towards international equities has steadily increased over time to current allocation of just under 50% of the portfolio. Furthermore, there has been a substantial amount of movement within U.S. equities. An investor who is concerned with maintaining the integrity of her asset allocation should think twice about investing in a fund that has a history of taking large cash positions as well as drifting in the allocation among different classes of equities.

Below are the Alpha Charts for the American Funds (A shares) compared to their Morningstar analyst-designated benchmark. The mutual fund trading symbol is shown just above each chart.