patrick.net

So the Case-Shiller index measures exactly what?, and this measurement is applicable to my real life how? In as few words as possible. . . .

Case-Shiller tries to show how your investment will appreciate over time if you buy a house. It measures how an individual purchase in the past would have appreciated over time (if it is typical of the region being measured, national or one of the cities). Obviously, a landlord would have to take into account rent and upkeep. The chart is showing how the appreciation part has changed over the years.

That 1100 ft2 is not from the census. Also, it is an estimate of houses build in the 1940s and 1950s, so call it representative of new houses in 1950 (smack in the middle of the 40s and 50s). Yet, you insist that it is exactly what the average house was in 1940, and that 900 SF is too small.

If you want to be so nit-picky and insistent, then you ought to be more precise.

robertoaribas says

1. coastal california prices have of course outpaced inflation for a generation. that is sort of a natural fact, as these cities increased in density in far more drastic manners than probably anywhere else in the US save New York.

See, but here's the thing: It would seem to me the average property from 1940 is in an area that has increased in density, for the simple reason that density as a whole has increased. People haven't just spread out - though clearly there is a lot of that, they've gotten more dense within the original habitation limits from 70 years ago, and that's where all the homes from 70 years ago are (by virtue of being in the original habitation limits before sprawl). Sure coastal CA and NY are extreme examples, but density overall is up in the areas where 1940's homes were.

gameisrigged says

I have acknowledged that it is POSSIBLE that different areas appreciate at different rates, but that is a far cry from you showing any quantifiable, non-anecdotal evidence.

Wow. You are one tough customer if you won't even concede that. I give up.

10. What housing sales are excluded from the index calculation?
The indices measure changes in housing market prices given a constant level of
quality. The repeat sales method limits the effect that changes in the types and
sizes of houses sold or changes in the physical characteristics of houses have on
the indices. Filters are used to maintain the integrity of the index as an indicator
of market trend. These filters identify and exclude: non-arms length transactions
(e.g., property transfers between family members); transactions immediately
preceding or subsequent to substantial physical changes to a property;

http://www2.standardandpoors.com/spf/pdf/index/SP_Case_Shiller_Home_Price_Indices_FAQ.pdf

The "transactions immediately preceeding" bit has always bugged me. Why should it be excluded just because a developer bought the property? Not to mention, if a developer buys a property to develop it, it's likely because the land value has gone way up, and the existing structures are no longed a best economic use.

MarkInSF says

gameisrigged says

I have acknowledged that it is POSSIBLE that different areas appreciate at different rates, but that is a far cry from you showing any quantifiable, non-anecdotal evidence.

Wow. You are one tough customer if you won’t even concede that. I give up.

Huh? I said different areas can appreciate at different rates. I conceded it, then you said I won't concede it and you give up?

Whatever, dude...

Shiller went out on a limb and correctly predicted the crash at a time when every other mainstream media source was saying "real estate always goes up", "it's a new economy", "it's different this time", "not here", and even after the impending crash was obvious, still saying "soft landing". If one is claiming the chart that did this is nonsense, I think it's contingent on THAT person to prove it.

gameisrigged says

Shiller went out on a limb and correctly predicted the crash at a time when every other mainstream media source was saying “real estate always goes up”, “it’s a new economy”, “it’s different this time”, “not here”, and even after the impending crash was obvious, still saying “soft landing”. If one is claiming the chart that did this is nonsense, I think it’s contingent on THAT person to prove it.

That he called the housing bubble is irrelevant to the question. Shiller is a very smart guy, but I just don't think the data he collected, with the methodology he used of specifically excluding transactions where redevelopment was done on the land, tells the story that he says it does.

By the way, Shiller was hardly alone. None other than The Economist magazine was running articles on the housing bubble starting in 2003, and even ran a few cover stories, the one in 2005 calling it the "biggest bubble in history."

Obviously it was hyperbole for me to say "every other mainstream media source". Of course Shiller was not literally the only person who predicted the crash. I wouldn't say "hardly alone" though. There were a number of bloggers who called it, but I still say he was, among the mainstream media, alone for the most part, with the notable exception you have mentioned.

For 1890 to 1934, Mr. Shiller used data from a trio of economists led by Leo Grebler. Because he found no index for 1934 through 1953, Mr. Shiller wrote, "I had my research assistants fill that gap by tabulating prices in for-sale-by-owner ads in old newspapers," covering just five cities. For 1953 through 1975, Mr. Shiller used an index compiled by the U.S. Bureau of Labor Statistics, or BLS. An index from the regulator of Fannie Mae and Freddie Mac covers the period through 1986, after which Mr. Shiller uses the S&P/Case-Shiller index he created with another economist, Karl Case.

http://online.wsj.com/article/SB124051414611649135.html

OK, so the methodology for 1934-1953 is laughable.

From 1953-1986, the chart is not using any methodology Shiller came up with at all. I though the were trying to use sales pairs, just with less reliable data, but they didn't even do that. It's just using government statistics. I'm curious what exactly those indexes referred to are, but I don't think it's too wild a statement to say that that they were NOT designed to track the value of a home as an investment over time, like the Case Shiller methodology is.

I've already stated my beef with C/S methology, that's was used since 1987, and how it would underweight the appreciation of raw land value component of a homes value, but on short time scales like a decade or two, I doubt it makes much difference, unless you're looking at a boom town like Bend Oregon.

Wow, MarkInSF, I had given up on this thread. Thanks for posting the pre-1987 methodology; I actually learned something. And no, I won't throw out my dogeared inflation adjusted Shiller long chart that I carry around in my wallet -- yet. In the mean time, I've got a house along a canal in Amsterdam that I'd like to sell one of you... oh, my head.

gameisrigged says

. I also don’t know what “excluding transactions where redevelopment was done on the land” means...In the meantime, until you explain that, I would say that if the methodology is to use matched sales pairs,

Simply put, C/S does not include a sales pair if the the end sale is bought by a developer (maybe splitting the property into 2 or ten properties, or into some other more economic use like a condo). When is that most likely to happen? It's when the value of the land has gone way up - even beyond inflation, to the point it makes economic sense sense to redevelop it.

If you have a town that used to be 10 acre family farms, and over time they were all redeveloped and sold to families in 1/2 acre plots, what do you think happens to the value of the land to the one holdout who keeps his 10 acre farm while the town grows up around him over 50 years? It will go up a lot faster than inflation, that's for sure. But when that last holdout finally sells his home to a developer, for a price the has dramatically outpaced inflation, Case & Shiller choses not to include this sale in their data.

That's an extreme example, but the same things happens on a smaller scale all the time. Not enough to make a huge difference in the short fun, since most transactions are not to developers, but over long periods a significant portion of properties have been redeveloped.

Essentially, it excludes sales pairs exactly when their appreciation is most likely to have been dramatic.

Mark,

If a 10 acre lot is sold to a developer, then the developer divides it into 20 1/2 acre lots as in your example, the first sale from the original owner of the 10 acre lot to the developer counts. That is representative of appreciation of the original house.

The next sale from the developer to the new owner of a 1/2 acre lot is excluded. The assumption is that the developer built a house on the lot, so that price is not due to appreciation.

I'm sure that there are some issues where positive appreciation is excluded. There are probably also cases where some improvement sneaks in. Hard to say what the net effect is.

YesYNot says

If a 10 acre lot is sold to a developer, then the developer divides it into 20 1/2 acre lots as in your example, the first sale from the original owner of the 10 acre lot to the developer counts. That is representative of appreciation of the original house.

Hmmm. That's not entirely clear to me.

The pairing process is also designed to exclude sales of properties that may have been subject to substantial physical changes immediately preceding or following the transaction....Transactions where the seller may be a real estate developer (based on the seller’s name) are also excluded, since it is likely that this is the sale of a newly constructed home built on a previously vacant or occupied lot or a rebuilt existing home.

Buying for subdividing sounds a lot like a substantial physical change immediately following the transaction, but it is only specific about developers with regard to them being the seller.

YesYNot says

The next sale from the developer to the new owner of a 1/2 acre lot is excluded. The assumption is that the developer built a house on the lot, so that price is not due to appreciation.

Hmmm... If you're right, I think I see where the of lot of the value of land appreciation went, that seems to be missing from the Shiller chart over time: into the developers pocket.

MarkInSF says

By the way, Shiller was hardly alone

Actually he was very much alone. His book Irrational Exuberance 2nd edition came out in 2005. However he did make many appearence on CNBC as well as write a couple articles around 2002 and 2004. Around that time, he was the only one talking about "bubbles" and much laughed at.

http://www.businessweek.com/bwdaily/dnflash/aug2002/nf2002082_7132.htm

Home Is Not Where the Returns Are AUGUST 2, 2002

"The fear doesn't seem irrational, especially with signs that the economy is rapidly losing momentum. Home prices are unhinged from the economic fundamentals in some parts of the country, such as the San Francisco Bay Area. "The clearest case of a speculative bubble in the U.S. recently, I believe, is the Bay Area -- San Francisco, Oakland, San Jose, Silicon Valley -- where prices were rising at a tremendous clip," says Robert Shiller, economist at Yale University."

MarkInSF, YesYNot - you seem to disagree on whether these sales pairs are excluded or not, so I don't know who to believe here.

Mark, the post before that is intriguing, and I will definitely take a look at that article.

YesYNot says

The CPI data used for 1953 to 75 is based on homes with constant age & size, because they were trying to measure increases in cost of living for the same quality items.

That's not the same as a "same home" sale. BLS is trying to track inflation. If I'm trying to judge the price of a 1200sft, 30 year old home over time, the average 30 year old 1200sft home is going to move AWAY from the center of an metropolitan area.

Why? Simple. All the old homes are at the center of a metropolitan area, since they were buit first, and newer homes were built on the outskirts.

Now, I think it's pretty well known that housing that is more central is more expensive than outlying housing. While BLS controls for size and age of the housing, it does not at all control for the desirability of the location.

To take the 30 year old home example again: In 1950 BLS would be looking at the price of a 1200sft home built in 1920 in San Francisco. Today it would be comparing it to the price of a 1200sft home built in Pleasanton in 1980.

That looks a lot like apples and oranges to me.

game, I think that both the Shiller and Lawler plots are reasonable. Shiller himself isn't sitting around calling for a reversion right back to 100 on his graph. A little while back, he was saying that another dip was possible, but that prices could continue up as people get excited again.
I've been guessing that prices still have some falling to do based on the Shiller plot, but am unsure. That is why I spent so much time on this thread.

thomas.wong1986 says

MarkInSF says

By the way, Shiller was hardly alone

Actually he was very much alone. His book Irrational Exuberance 2nd edition came out in 2005. However he did make many appearence on CNBC as well as write a couple articles around 2002 and 2004. Around that time, he was the only one talking about “bubbles” and much laughed at.
http://www.businessweek.com/bwdaily/dnflash/aug2002/nf2002082_7132.htm
Home Is Not Where the Returns Are AUGUST 2, 2002
“The fear doesn’t seem irrational, especially with signs that the economy is rapidly losing momentum. Home prices are unhinged from the economic fundamentals in some parts of the country, such as the San Francisco Bay Area. “The clearest case of a speculative bubble in the U.S. recently, I believe, is the Bay Area — San Francisco, Oakland, San Jose, Silicon Valley — where prices were rising at a tremendous clip,” says Robert Shiller, economist at Yale University.”

If you are trying to support the notion that Shiller is great at predictions, I wouldn't link an article that shows he was early by 5 years in calling the peak, not to mention the fact that for some parts of the BA, he's still just flat out wrong and prices are well above their 2002 prices.

MarkInSF says

For 1890 to 1934, Mr. Shiller used data from a trio of economists led by Leo Grebler. Because he found no index for 1934 through 1953, Mr. Shiller wrote, “I had my research assistants fill that gap by tabulating prices in for-sale-by-owner ads in old newspapers,” covering just five cities. For 1953 through 1975, Mr. Shiller used an index compiled by the U.S. Bureau of Labor Statistics, or BLS. An index from the regulator of Fannie Mae and Freddie Mac covers the period through 1986, after which Mr. Shiller uses the S&P/Case-Shiller index he created with another economist, Karl Case.

http://online.wsj.com/article/SB124051414611649135.html

This is good to know. After all that heated exchange, it's nice to see some factual information on Shiller's data sources.

roberto,
I guess the fact that the article said that prices were at a peak twice and used Shiller's comments to support this lead me to believe that someone saying the BA is in a speculative bubble meant that an impending burst of the bubble was near. So the fact that Shiller said the BA is in a speculative bubble in 2002 and now we're in 2011 with prices higher than 2002, what does this tell us about his predictive skills? Are you saying he doesn't predict, and anything he creates should not be used for predictive purposes?

YesYNot says

Shillers long term chart is for the US, and shows a $30,000 house in 1940 (adjusted for inflation) going for $50,000 in 2000. The median sales price over that time went from $30,000 to $120,000

Whoa, threre. Let's stick with the section of the graph that's based on BLS data I think I've already shown 1940 to 1953 data is highly suspect. 1953 to 75 was the BLS data.

The Shiller/BLS chart shows housing prices going slightly DOWN in inflation adjusted terms from 1953-1975. If I linearly interpolate the decade census data, it shows the median home price going up by 42% in those 22 years.

Does improvements in the median home account all that 42% rise? Possible... but that's would be awful lot of old homes torn down, and/or modern homes built in 22 years. The housing stock likely grew by about 34%, given the population did too. I'm not even sure that if the newly added housing stock was double the real construction costs of the median 1953 home it could budge the median by 42%.

Another data check: http://www.census.gov/const/pricerega.pdf shows the inflation adjusted median home price going up 24% from 1963-1975. BLS/Shiller show adjusted homes prices almost exactly flat over the same time period. Does improvements in size and plumbing or what have you account for a 24% difference in the median home price in just 12 years, given that the housing stock could not have changed that significantly? I find that much harder to believe.

Shiller asserting the the BLS data tracks the expected appreciation of a home standing in 1953, until 1975, seems highly suspect to me.

"Lawrence Yun of the National Associations of Realtors has ratcheted up his criticism of Shiller. Yun calls Shiller's index a "total distortion of market conditions based upon a small selection of falling local metro coverage."

And the minions of the NAR are still spreading their gospel.

Yun suggests that Shiller profits from creating fear. He creates fear so that people will want to hedge themselves against price declines by buying insurance against a free fall in prices. Where is such insurance available? Answer: on the S & P Case Shiller index traded on the Chicago Mercantile Exchange. According to Yun, Shiller has a "side business in Chicago. His index is used at the Chicago Mercantile Exchange for hedging housing futures values. The more hedging of bets that occur, the more profits go into Dr. Shiller's bank account. And more hedging of the bets will take place if people believe there will be a crash in housing values. So naturally he has a financial incentive to ‘scare' the market."

At the end of the day, we have a lot of Realtors, slick talking salesmen, on this web site... and they are very afraid the gravy train will stop.

YesYNot says

. I estimated that the price increase in median sales price due to size increases was from 50,000 to 105,000. That gets it up to 105,000 or within 15%.

Actually, saying the increase in median price from 1940-2000 is fully accounted for by changes in size and other improvements is a different assertion than saying a home that was standing in 1940, that is still as is today has tracked inflation.

It is entirely possible that appreciation is very slow initially after new construction, not even keeping up with inflation, but accelerates as the house gets older, and it's land value become a greater component of it's value due to crowding.

MarkInSF says

It is entirely possible that appreciation is very slow initially after new construction, not even keeping up with inflation, but accelerates as the house gets older, and it’s land value become a greater component of it’s value due to crowding.

Until prices drop as was the case in 89-92 and more recently. Anyway, land around here isnt that valuable when you have alternatives. Expensive yes. Nowdays industries have parked their operations far away from the Bay Area. So to speak of valuable land ignores that industries that fual paychecks are moving elsewhere. And that eventually will correct the imbalances we see and have seen in the past.

MarkInSF says

Actually, saying the increase in median price from 1940-2000 is fully accounted for by changes in size and other improvements is a different assertion than saying a home that was standing in 1940, that is still as is today has tracked inflation.

It pretty much is the same thing.
The WSJ article you cited says

Mr. Shiller's chart shows that home prices from 1940 through 2000 rose at an annual real, or inflation-adjusted, rate of 0.7%. Data from the Census Bureau, however, puts the real rate at 2.3% for that period. Part of the difference may be due to improvements in the quality of homes, Mr. Lawler says, but he doubts that accounts for the whole gap.

Mr. Shiller responds in an email that much of the difference "is indeed caused by quality change." He cites Census data showing that nearly a third of American homes lacked running water in 1940.

Both Lawler and Shiller agree that is what it means.

MarkInSF says

It is entirely possible that appreciation is very slow initially after new construction, not even keeping up with inflation, but accelerates as the house gets older, and it’s land value become a greater component of it’s value due to crowding.

I agree. A new house structure should depreciate at first, and that depreciation could offset or outpace appreciation of land. The index does not apply perfectly for any particular home be it new, old, in a tony neighborhood or in Detroit, so expecting that kind of accuracy is asking too much. The current methodology only reflects the average growth in a particular market. They define the average as the aggregate price. So each percent appreciation is weighted by the house price.

MarkInSF says

Another data check: http://www.census.gov/const/pricerega.pdf shows the inflation adjusted median home price going up 24% from 1963-1975. BLS/Shiller sho

Those look like new construction prices. The median price of all sales in 2000 was $120,000. That link shows $169,000, which is the median price for new construction. As you said, most new construction is in the region where a city is expanding and land values are increasing the most.

ch_tah says

If you are trying to support the notion that Shiller is great at predictions, I wouldn’t link an article that shows he was early by 5 years in calling the peak, not to mention the fact that for some parts of the BA, he’s still just flat out wrong and prices are well above their 2002 prices.

I think the bubble would have popped MUCH earlier had the government not heavily intervened in the market. By all rights it shouldn't have taken 5 years. He simply called it as being a bubble, and he was right.