patrick.net

I saw this. I'm not really a big fan of Consumer Reports' reliability rankings. The types of people who rely heavily on Consumer Reports are a self-selecting group and sometimes want someone else to tell them to buy a Toyota. It also makes me wonder if sample sizes are really sufficient to be accurate for CR on luxury cars. JD Power's rankings show differently -- both in initial quality survey (IQS) and vehicle dependability survey (VDS).

This year's IQS:
http://www.jdpower.com/news/pressrelease.aspx?ID=2011089

This year's VDS:
http://www.jdpower.com/news/pressrelease.aspx?ID=2011029

The divergences are just bizarre. How are Buick, Cadillac, and Porsche so low for CR and yet above average for JD Power (with individual models tops in their categories). How the hell is Chrysler rated so frickin' high by CR, when it fails the VDS so miserably (where Chrysler, Dodge, and Jeep are 3 of the bottom 5)? Scion is not even close to above average on JD Power -- it is far below on both IQS and VDS (although the #1 compact utility vehicle is the xB), so how the hell is it #1 on CR with the xB as its worst model? Wtf?

Two weird results:
The QX56 is Infiniti's best model under CR. The Escalade is Cadillac's worst model under CR. Yet on JD Power, the Escalade #1 in reliability for large SUVs in the IQS (the QX56 is not even in the Top 3).

Porsche has one of the top factories for in the world -- JD Power rates individual factories. That factory makes 911s and Boxsters. Are the Cayennes really dragging Porsche down so much to make it near last in CR's rankings?

Where CR and JD Power agree:
The Fusion does well.
The Miata does well
The Lincoln MKZ does well.

The C-Class does well.

luxury cars typically are the first to employ new technology and often times that new technology is not as reliable as the old technology (but as least you have it). A trade off. Over time the engineers figure out how to make the new tech more reliable (and over time it becomes the old tech) and the next model year will have new tech with new bugs to be worked out. Also luxury held to higher standards of precision (like calibrating a fine piece of equipment)

Oh, I worked as an engineer for consumer reports and my first car out of college was a tested car from consumer reports (as an employee I got to buy the cars they sell after they test them)

New Mercedes are not well built, that much I can tell you.

NYblogger says

uxury cars typically are the first to employ new technology and often times that new technology is not as reliable as the old technology (but as least you have it).

corntrollio says

I saw this. I'm not really a big fan of Consumer Reports' reliability rankings.

I can agree with that. I've been amused by the occasional large differences reported between cars which are actually twins.

But I'm not sure JD Power is any better. I'll give them credit for confirming ownership but they are so damn sure of their results, in a self-serving way, that it sets off my BS alarm.

This is from the JD Power FAQ:

"Results are based on measures of customer satisfaction and quality by consumers that are proprietary to J.D. Power and Associates. Awards and performance ranking are based on numerical scores, and not necessarily on statistical significance. Our experience indicates that even when small differences occur in the scores of those being measured, these small perceived differences help drive competition in the marketplace, thereby improving product and service quality and driving increases in overall customer satisfaction."

So might JD Power be cooking their results to avoid the embarrassments which CR publishes right out in the open? I suppose the answer can be found in the raw data, but that raw data is available only to the paying customers such as manufacturers and advertisers, not us.

Well, whatever. JD Power comes up with results, results which are so free of statistical noise that even small perceived differences are important. How do we know these small differences are important? JD Power says so, that's why!

Maybe J.D. Power can be ummmm..., how should I say this,...."lobbied?"

Another tidbit from the JD Power FAQ:

Why doesn’t JDPower.com list actual survey (index) scores for specific models?

"J.D. Power and Associates provides manufacturers and suppliers with diagnostic information to help them improve the quality of the products and services they provide. In most cases, this type of detailed research information is not presented in a “consumer friendly” format. While tables of numbers and volumes of data may be helpful to engineers or manufacturing experts, consumers may desire an easily understandable format such as the Power Circle Ratings on this site. The goal of JDPower.com is to help consumers make more informed decisions through a consumer-friendly rating system that we believe is more helpful and less confusing than actual index scores."

marcus says

Maybe J.D. Power can be ummmm..., how should I say this,...."lobbied?"

I don't think JD Power could be influenced, unless that influence resulted in further consumer satisfaction for us confused simpletons in the consuming public.

Although Toyota and Honda ranks high overall reliability, popular models like camry, corolla, accord, civic are not really highly reliable car any more. They rank about industry average now.

eoulim says

Although Toyota and Honda ranks high overall reliability, popular models like camry, corolla, accord, civic are not really highly reliable car any more. They rank about industry average now.

I maintain a 2000 Toyota Camry, that has ~ 104,000 miles on it. I can confirm that it need everything that a car with 100,000 miles usually needs. A timing belt, new struts and tires (damaged by bad struts). That and the paint is bad in places, the door handle broke, and the fuel filler door needs that spring that keeps on breaking on Toyotas. It has a lot of other small problems too.

The only particularly good thing about this car is that it has a non-interference engine, so when the timing belt breaks, it won't ruin the engine like a lot of other Japanese cars will. This same person owns a Subaru that they won't drive until I can change the timing belt on it (engine will self destruct on that one if the timing belt breaks).

Cook County resident says

Well, whatever. JD Power comes up with results, results which are so free of statistical noise that even small perceived differences are important. How do we know these small differences are important? JD Power says so, that's why!

Cook County resident says

I can agree with that. I've been amused by the occasional large differences reported between cars which are actually twins.

But I'm not sure JD Power is any better. I'll give them credit for confirming ownership but they are so damn sure of their results, in a self-serving way, that it sets off my BS alarm.

Yes, it's funny when Consumer Reports cites huge differences between twin cars that are often made in the same factories to the same tolerances. However, sometimes the Lincoln, Cadillac, and Buick versions are better, which is probably accurate. Not sure about Chrysler vs. its twins though.

With respect to statistics, it's not clear that Consumer Reports is adequately showing any statistical differences either. Look at how big the ranges are for certain cars. Maybe it's just that very few CR readers buy Mazdas. The biggest flaw for CR is that it's only CR subscribers who are self-selected and have certain biases.

There are certainly valid criticisms of JD Power:
1) is the IQS valid since it's only 90 days? Maybe, maybe not, but they also have the VDS. CR provides no granularity.
2) was the VDS better when it was 5 years instead of 3? Probably. CR is weird because it looks at the last 10 years and doesn't necessarily make valid comparisons when cars get refreshed or an entirely new version comes out.
3) does the IQS have some flaws because it lumps together production issues with design issues? Sure. It's interesting to see the difference, by the way. CR has the same flaw, of course.
4) The biggest thing to note, however, is that the difference between the best car and the worst car on JD Power is not all that much, which is probably also true of CR. We are really talking about small differences here in both cases because the overall percentages are so low.

Another thought on CR -- is a comparison to average really the right answer? What about a comparison to median?