An increasing number of the systems we have been talking about are using ETFs to easily gain exposure to many different types of assets and indexes. ETFs seem to make trading multiple markets much simpler, but can we really trust that they are effectively tracking the underlying assets that they are supposed to be representing?

Nathan Faber from NewFound Financial Innovators took a look at this idea on their blog, Flirting With Models. In his article. he set out to find a quantitative method to test the accuracy of different ETFs.

Tracking the accuracy of an ETF could be a good way to gain a better understanding of its volatility.

Faber begins by pointing out that this tracking concept has been around for years in the mutual fund world. He discusses how two different measures can be used to track this accuracy. The first is tracking error:

The tracking error, ψ, is computed as the annualized standard deviation of the daily return difference of the ETF and its benchmark:

ψ = stdev(r_ETF-r_index)

The other measure is tracking difference:

Morningstar also uses a metric called tracking difference, which is the difference between the annual returns of the ETF and the benchmark.

This metric captures the sign and magnitude of the deviation of the ETF from the index over a longer period of time, while the tracking error represents the day-to-day volatility of the absolute return difference.

Faber points out that there are three relevant prices when it comes to testing the accuracy of an ETF. We have to consider the price of the underlying index, the ETFs NAV, and the actual price of the ETF. He further explains the relationship between and ETFs NAV and its price:

The price of an ETF can vary from its NAV, sometimes significantly.  However, for large ETFs that trade liquid instruments, the price should remain close to the NAV to wipe out arbitrage opportunities.

The article goes on to test some ETFs against their underlying indexes. Not surprisingly, the ETF accuracy of more common ETFs like SPY and IVV does a very impressive job of tracking their underlying index. It is nearly impossible to tell them apart on a long term chart.

However, when Faber tested CWI and ACWX, the results showed that they did not track their indexes as closely. Faber points out that this is not an indicator of future performance:

Tracking error alone cannot tell us how an investment will perform relative to a benchmark.  It can only tell us that the investment will be more volatile.

It is also possible to have low tracking error and not match the index at all (consider an extreme case of a fund that consistently beats the benchmark).

He suggests that the goal here is simply to gain a better understanding of the financial instruments that we might be trading:

By peering behind the curtain, we can become more knowledgeable about how our investment vehicles are managed and gain more confidence that they will function well in both tactical and strategic investment strategies.

It is one thing to suffer a loss due to a given macroeconomic environment – the nature of investing makes this unavoidable – but underperformance due to ambiguous ETF/mutual fund management can limit the reliability of a strategy to perform as intended, during good times and bad.