Originally published by Managed Account Research, Inc. on July 4, 2007

CTA Confidential SM

An ongoing series of qualitative investigations

into managed futures trading programs

SPECIAL CASE STUDY

Managed Futures:

Pitfalls in Performance Evaluation

PAST PERFORMANCE IS NOT NECESSARILY INDICATIVE OF FUTURE RESULTS. INVESTING IN FUTURES AND OPTIONS INVOLVES RISK AND MAY NOT BE SUITABLE FOR ALL INVESTORS. THE HIGH DEGREE OF LEVERAGE THAT IS OFTEN OBTAINABLE IN COMMODITY TRADING CAN WORK AGAINST YOU AS WELL AS FOR YOU. THE USE OF LEVERAGE CAN LEAD TO LARGE LOSSES AS WELL AS GAINS. THEREFORE, INVESTORS SHOULD CAREFULLY CONSIDER THESE RISKS AND DETERMINE WHETHER THEY ARE SUITABLE FOR INVESTING IN LIGHT OF THEIR FINANCIAL CONDITION AND INVESTMENT OBJECTIVES.

In the first article of a three-part series about managed futures we alluded to concerns about the intricacies of adept CTA performance evaluation. This article further explores that topic.

While the futures industry’s regulatory rules provide clear guidance as to managed account composite performance reporting, there are pitfalls in making investment decisions based on a track record’s outward appearance without considering the potential for internal distortions.

For those unfamiliar with managed futures, it is a niche sector of alternative investments and refers to professionally managed assets in the commodity and financial futures markets. Management is facilitated by Commodity Trading Advisors (CTAs) who are registered with and regulated by the Commodity Futures Trading Commission (CFTC) and the National Futures Association (NFA).

The institutionalization of alternative investments can be traced back to the development of CTA performance tracking databases first established around 1979. This data became the basis for an academic body of research on managed futures beginning with the seminal study by Harvard Business School professor, Dr. John E. Lintner. [1]

In the old days CFTC regulations required that CTAs produce and present in their disclosure document an accounting called the 13 column performance table. This table is useful for CTA analysis in that it shows on a composite basis beginning net asset value, additions, withdrawals, net performance, ending equity and period rate-of-return, as well as the components of net performance: gross realized profits/losses, changes in unrealized profits/losses, commission expense including accrued commissions, interest income, management fees and incentive fees.

The 13 column performance table later became distilled into the 7 column performance table, and now the rules surrounding performance reporting provide for the presentation of capsule performance table and capsule performance information. These condensed data tables neatly encapsulates the CTA’s percentage rate-of-return during the most recent five calendar years and the current year to date, but they also exclude other useful data embedded in the older layouts.

Plaudits go to CTAs who provide the 13 column performance table freely upon request. While CTAs are no longer required to produce this table by regulatory statute, as we shall see investors would be wise to first obtain and analyze this table prior to investing in a CTA’s program.

When comparing rates-of-return a common mistake for investors new to managed futures is to assume that for any equivalent period of time, a 20% annual return for one CTA is the same as a 20% return for another CTA. This is not necessarily the case.

As a hypothetical example, we’ll compare one CTA, a floor trader who is actively engaged in scalping bond arbitrage strategies, against another CTA who trades a long-term diversified trend following system. The trading velocity, enumerated as round turns per million dollars per year or RT/MM/YY [2] for the floor trader averages 12,000, whereas the trend follower rarely trades and averages 600 RT/MM/YY with clients charged $15 round-turn commission.

Supposing that the floor trader’s commission cost is currently $5 round-turn but will increase to $15 when the strategy is transitioned upstairs, the $10 increase in cost would negatively impact his performance by 12% for the year, reducing the 20% return originally presented to just 8%. Under this scenario, all factors being equal including commission costs, the trend following program has superior past performance then the bond arbitrage program if traded off-the-floor. [3]

This ‘apples versus oranges’ conundrum is often ignored, glossed over or not even recognized by many managed futures investors in their search for superior performance statistics.

Other factors that can materially influence returns, besides trading velocity and commission costs, include interest income, as well as management fees and incentive fees (which in actuality may on average be different than the stated rate in a CTA’s disclosure document). That is why we think the 13 column performance table is so usefulthe data provided within can be incredibly informative as to the influence these factors have upon performance.

Optimally, investors should obtain the 13 column performance table, separate out gross realized profits/losses and changes in unrealized profits/losses, and from that gross return create a pro forma performance table where the assumptions on commission expenses, interest income, management and incentives are standardized across all CTAs being compared.

But the above discussion is just a first level consideration of factors that impact performance evaluation. Investors should also be careful to recognize the faulty logic of comparing dissimilar trading strategies, and make sure they compare upside and downside performance under parallel time periods and analogous market conditions which underpins like trading approaches.

Another area where potential distortions may arise is rate-of-return calculations. The CFTC appreciates that Past performance information presented to clients and prospective clients is a primary marketing tool for CTA programs. Accordingly, the Commission states that its aim is that information provided to clients be accurate, complete, and understandable. [4]

That said regulatory rules pertaining to performance computation and reporting methodologies have evolved based on CFTC Advisories dating back to 1981. [5] A long-standing subject of contention between regulators and industry participants is notional funding and the use of nominal account size as the denominator in calculating rate-of-returns. [6]

In July 2003 the CFTC adopted regulations establishing a core principle for CTAs with regards to performance disclosures concerning partially-funded accounts. The Commission’s own words provide the best insight into both sides’ perspective of this protracted industry debate.

The key component of the Commission’s rule proposal is the use of nominal account size, rather than actual funds, as the basis for CTAs’ computation of rates of return. The consumer organization and members of the public noted that investors look at actual funds when making investment decisions, and expressed concern that performance based on nominal account size would reduce the appearance of volatility of the CTA’s trading program. Commentators who supported use of nominal account size noted that it is the amount both the CTA and the client consider to be the account size. They also pointed out that use of actual funds can result in widely divergent return figures for similarly traded accounts; exaggerates positive and negative rates of return; and measures the cash management strategies of clients rather than the performance of the CTA.

From this the CFTC concluded the following:

The Commission has not found persuasive the comments opposing use of nominal account size rather than actual funds as the basis for computing CTA rates-of-return.

As it stands now, rate-of-return is defined as net performance for a period divided by the beginning net asset value for that period, which may include notional and/or committed funds if properly documented). Using this method, however, can still sometimes result in distortions in computed rate-of-returns under certain circumstances.

For example, distortions can result when additions and/or withdrawals are large and are made early in the reporting period. In those instances using the method in the paragraph above would result in rate-of-returns that are inaccurate. Because this is well-recognized, regulations provide for the use of the time-weighting and only accounts traded computation methods. Nevertheless, even these permitted adjustments also have their drawbacks.

With respect to the time-weighting method, the beginning net asset value is adjusted upward by time-weighted additions and downward by time-weighted withdrawals. While using this method will certainly smooth out distortions generally, it can still obfuscate accurate reporting. This can happen when the introduction of a large new account, which is not immediately aligned to the CTA’s full set of positions, reduces the overall return (negatively or positively) in comparison to the returns of older accounts with legacy positions.

In some cases, and it happens from time-to-time, the CTA will report a lower negative return in relation to volatile market conditions then the CTA would have otherwise reported had a large addition not been made. Likewise it should be recognize that the opposite impact on reporting can occur too. This is another situation where the 13 column performance table would be of use.

These time-weighting issues can be eliminated with the use of the only accounts traded or OAT method, but this methodology for return computations also has inherent problems. The OAT method calculates the monthly rate-of-return in the conventional manner except that accounts that traded for only part of the month or accounts that witnessed material additions/withdrawals during the reporting period are excluded from the calculations.

By excluding these accounts, it is assumed that the calculated figure will reflect the rate-of-return that would have been realized by an investor with an account that was active at the start of the month and held until the end of the month. In theory this approach removes the influence of intra-month additions/withdrawals yielding an undistorted actual return figure. However.

Imagine a long term trend following system that takes several months for new accounts to fully align to legacy accounts. If this CTA began trading numerous new accounts, it is theoretically possible that the reported composite performance in the subsequent reporting period(s) would be weighted towards the return of these new accounts. In order words, while the older accounts may be more representative of the fully implemented trading program, the newer accounts’ dissimilar and arguably unrepresentative returns could skew the composite return presentation. [7]

In another situation the CTA may have only two accounts which both receive material additions/ withdrawals during the same reporting period. In these types of situations, there is no regulatory guidance on how to report performance assuming that all accounts in the composite are suppose to be excluded under the OAT method. In such cases, CTAs usually revert to time-weighting.

As one can derive from this examination so far, reliance on CTA return presentations without proper due diligence can be hazardous to making an informed investment decision. We are not aware of any studies, but one could hypothesize that these distortions in performance reporting may have influenced the results and conclusions of certain commonly cited academic studies.

Additionally, the reliance on month-end numbers conceals extremely valuable daily return and volatility data, which unfortunately is an industry-wide logistics hurdle. Another issue which the industry needs to address is the standardization of performance analytic calculations. [8]

This is all important but perhaps a more interesting analysis for investors to consider involves quality-of-return concepts often referred to as risk-adjusted returns. Margin-to-equity [9] is the simplest but also coarsest way to gauge returns relative risk. In futures trading margin is a good faith deposit which by design requires minimal actual funds to control the nominal contracts size. This leverage is largely responsible for the potential of outsize returns in managed futures.

If we were to compare two CTAs, one with an annualized return of 40% whereas the second returned 10%, most investors new to futures would gravitate to the first program. However, if the average margin-to-equity for the first CTA is 80% versus 20% for the second, and all other factors are assumed equal, risk adjusted performance on that basis is actually the same.

This is why sophisticated investors may recognize value-added in an ostensibly underperforming CTA and use notional funding to leverage that CTA to balance out his portfolio. [10 ]

Margin-to-equity is not, however, the best way to standardize risk-return. Value-at-risk, modified Sharpe ratio, rank correlation analysis and other measures provide in combination the best means in which to quantitatively analyze the risk-adjusted returns of managed futures programs.

In our opinion the use of quantitative analysis is too often over-emphasized for reasons provided so far in this article. But if quantitative analysis can only be partially relied on, what other analytical approach should investors used to evaluate CTAs?

The answer, in our opinion, relies on emphasizing a qualitative approach.

In a media driven world of sound-bites and fifteen minutes, there is unfortunately a predisposition for investors to glance at past performance and the executive summary for the easy money.

Unfortunately not all CTA performance numbers are directly comparable. Fact is, in managed futures there is no such thing as outsize profitability without leverage and additional exposure to risk. For that reason astute investors should look beyond the quantitative into the qualitative.

Due to constraints on editorial length, this article cannot thoroughly exam in detail all of the relevant analytical issues involving CTA performance reporting and analysis. For a comprehensive discussion and analysis of quantitative performance considerations when investing in managed futures, contact Managed Account Research at (800)308-1495 or via email at info@ma-research.com .

[1] Lintner, John E. The Potential Role of Managed CommodityFinancial Futures Accounts (and/or Funds) in Portfolios of Stocks and Bonds. Presented at the Annual Conference of the Financial Analysts Federation. May 1983.

[2] A round turn is a completed futures transaction involving both a purchase and a liquidating sale, or a sale followed by a covering purchase. ‘Round turns per million per year’ is a standardized measure of how many times a year a CTA would trade for a one million dollar size account. This number is usually extrapolated and not accurate.

[3] Example from Managed Trading, Myths & Truths by Jack D. Schwager, publisher John Wiley & Sons, Inc. ISBN -0471-02057-5. (Author’s personal note of appreciation to Jack for having the courage to first write about what some of us had recognized. )

[4] Federal Register/Vol. 63, No. 117/Thursday, June 18, 1998, Commodity Futures Trading Commission, 17 CFR Chapter I, Concept Release: Performance Data and Disclosure for Commodity Trading Advisors and Commodity Pools

[5] See 46 FR 26005, 26009 (May 8, 1981). Pursuant to the original Part 4 disclosure rules adopted in 1979, CTAs were permitted, but not required, to disclose their past performance in accordance with the format specified.

[6] The issue of notional funding at one point resulted in CFTC Interpretative Letter 93-13 that required the use of the fully-funded subset methodology for calculating rates-of-return. In the opinion of this article’s author, this particular regulatory-required methodology resulted in grotesque distortions in performance reporting during the years of its application and usage. Fortunately, required use of this methodology has since been repealed.

[7] CFTC Interpretative Letter No. 93-13 provides for certain materiality tests to determine if accounts are traded in materially the same way. In cases where the NFA may determine this is not the case, the author is aware of a case were the CTA was required to separate the performance of each client account into separate performance tables.

[8] Performance analytics commonly used in the managed futures industry include: Sharpe ratio, standard deviation, RT/MM/YY, margin-to-equity, etc. A survey of popular CTA tracking databases reveals that each provider’s reporting is not comparable from one database to another due to different assumptions underlying each provider’s calculations.

[9] The margin-to-equity ratio calculation divides the initial or maintenance margin requirement and divides it into the net asset value of the account. This ratio, typically presented as a static number, is in fact a volatile ratio which in most trading materially changes from day-to-day. In practice, CTAs provide the margin-to-equity ratio for guidance purposes to show generally what the average margin requirement is for their trading program. Investors should be aware that reported margin-to-equity ratios are usually rough estimates and in practice not updated often.

[10] Experienced managed futures investors often utilize what is known as a cross-margin account structure, where the cash account is capitalized and collateralizes trading accounts which are traded on a nominal or notional basis.