How the Crandall and Case-Shiller series are constructed

While the Crandall data (real median new-home sale prices) are pretty transparent, as are their limitations, the Case-Shiller data are more complex. For 1953-1975, Case and Shiller use the Home Purchase Component of the U.S. CPI, collected by the Bureau of Labor Statistics and covering prices for homes where the age and square footage was held constant. This component was eventually discontinued for the CPI due to several well-known deficiencies acknowledged by Shiller. 7 Given those deficiencies and the anecdotal evidence of the time, 8 one can make a pretty good case that the short, sharp housing bear market of 1968-1970 really happened, followed by a quick recovery. There is also little question that the bear markets of 1979-1982 and 1988-1992 really happened, although the latter was concentrated in specific regions of the country.

From 1975 to 1987, Case and Shiller use the House Price Index from the Federal Housing Finance Agency (FHFA), formerly the Office of Federal Housing Enterprise Oversight (OFHEO). This is also a repeat sales index, which controls for quality changes. Starting in 1987, Case and Shiller constructed their own data, and their method from 1987 to the present was used in the Case-Shiller series going forward.

We care about both history and the future, for slightly different reasons. Recall our original reason for asking the question, Which index is better? We want to know whether housing is a conventional volatile capital asset. To answer this, we need to know history we need to know whether housing experienced one bear market or four in the last half-century. It is clear that three of the four shown by Crandall did happen, and the evidence for the first one is mixed. Real estate is indeed a capital asset that fluctuates with changes in expected future cash flows and discount rates; how could it be otherwise? There are no magical assets, no assets that present reward but little or no risk.

But the most recent crisis has many features distinguishing it from all others (except the Great Depression), and we need to know how bad the crisis was, because it could happen again and its important to know how bad things can get. Therefore, we are curious whether the 2006-2012 crash involved a 23% or a 43% housing price decline.

This question isnt decided by the relative accuracy of the Crandall and Case-Shiller series pre-1987, because Case and Shillers data sources before that date are no longer used. We simply want to know which one is the more accurate measure of price return today: The median new-home sales price used in Crandall or the repeat-sale regression constructed by Case and Shiller.

And the answer to that question is almost certainly Case and Shiller. During a housing depression, there arent many new homes, and those that are sold may have special characteristics they may be cheap or expensive, or located in a part of the country that is little affected by the depression. Existing homes are a much deeper market, and when prices gap down for a seller, the market price is also likely to have gapped down for houses that are not for sale, or that are hard to sell and that thus do not record repeat-sale prices.

The countervailing bias in Case-Shiller is that their index is transaction-weighted and boomtowns such as New York, San Francisco, Chicago and Washington have an outsized influence on the index. Housing prices in these locations almost certainly rose more and then fell more than the national average. Therefore the 43% decline may be somewhat overstated. But it is probably closer to reality to a typical homeowners experience than 23% is.

The 2006-2012 downturn was qualitatively worse than the others

Thus, this most recent housing downturn was quantitatively different from the earlier ones. Notably, it was also qualitatively different:

Pre-World War II events aside, the nationwide character of the 2006-2012 crisis is unprecedented. 9 While there were regional differences, big price gains accrued along the whole West Coast, in Arizona, Nevada, Florida and practically the entire Northeast. These areas taken together represent a majority of the U.S. population. Declines in smaller regions, such as those occurring in previous housing bear markets, are much less serious.

Nominal versus real effects: Because of low inflation and low down payments in 2006-2012, many more homeowners were underwater than in previous downturns. The behavioral effects of being underwater are huge, especially for unseasoned homebuyers who were recently renters. They walk away from their mortgages in much greater numbers than long-term property owners who can afford to keep paying the mortgage while they wait for the recovery. Moreover, when inflation is low, the financial burden of a fixed monthly payment doesnt go down in real terms very quickly, compounding the problem.

Ownership base: As suggested above, extending the ownership base causes the pain of any given downturn to become dramatically amplified. Not only were new owners in more vulnerable groups than before, many seasoned owners borrowed additional funds in the run-up and came to have financial characteristics resembling the newest owners. Figure 3 shows how the base of owners had grown leading up to the crisis. While those last few percentage points in the ownership base may not seem all that significant, remember that most of the people who wanted and could afford houses already had them, so those added last to the ownership base were marginal indeed.

Figure 3

Homeownership rate in the United States, 1965-2014

Source: U.S. Census Bureau

With all this information in hand, lets compare the 2006-2012 bear market to the one in 1979-1982, which was similar if one uses the Crandall data. (We do not know enough about 1968-1970.) The big difference is that in 1979-1982, inflation was high (good for borrowers), nominal prices were not falling (just real prices), ownership rates peaked at just 66% (vs. 69% in 2007), and homeowner leverage was much lower. Therefore, even if we take the Crandall 23% decline in 2006-2012 at face value, instead of the gargantuan 43% of Case-Shiller, the recent crisis should have been much worse than the one in 1979, simply owing to its idiosyncrasies.

In other words, even if we agree that the recent crisis was worse than the others, it doesnt immediately follow that Case-Shiller is the better index for representing reality simply because it recorded a larger drop. Even if the real percentage price drop was the same as in 1979-1982, we should have anticipated greater pain!

1. To date the recent crash we use the peak and trough index dates from Case and Shiller, December 2005 and March 2012, even though the steepest part of the decline was much shorter (end of 2007 through the first quarter of 2009).
2. Shiller, Robert J. 2005, Irrational Exuberance, second edition, Princeton University Press, Princeton, NJ, chapter 2.
3. Wallison, Peter J. 2009. The True Origins of This Financial Crisis, American Spectator (February). Wallison writes, By 1997, Fannie was offering a 97 percent loan-to-value mortgage. By 2001, it was offering mortgages with no down payment at all. By 2007, Fannie and Freddie were required to show that 55 percent of their mortgage purchases were [low and middle income] loans and, within that goal, 38 percent of all purchases were to come from underserved areasand 25 percent were to be loans to low-income and very-low-income borrowers.
4. Shiller [2005], chapter 2, footnote 3. The Case-Shiller data extend back to 1890. For 1890-1934 they use data from Grebler, Leo, David M. Blank, and Louis Winnick, 1956, Capital Formation in Residential Real Estate: Trends and Prospects , Princeton University Press, Princeton, NJ . The data for 1934-1953 are a simple average of five cities median home prices advertised in newspapers, collected by students of Shiller. This last period is the only part of the historical series for which median prices, which dont correct for quality, are used by Case and Shiller.
5. Corroborative evidence includes a sharp downturn in new construction and new housing sales, cited by almost all sources. See, for example, Gjerstad, Steven D. and Vernon L. Smith, Rethinking Housing Bubbles. figure 5.7 on p. 135. (Smith was a 2002 Nobel economics laureate.) Non-corroborative evidence includes the chart Long decline in home prices relative to income reversed in the 1970, on p. 9 of Sayre, William R. Cyclical Downturn in Housing, Economic Perspectives, Federal Reserve Bank of Chicago, May-June 1980; the 1969-1970 decline is (visually) 7% at most, and then only relative to disposable income, which was rising but Sayre may have been looking at the Home Purchase Component of the CPI.
6. There was almost certainly a nationwide decline in housing prices during the Great Depression, and possibly in earlier depressions and recessions.