JD Power Survey Data Shows a more Positive Outlook for Automotive Quality...At least in the Short Term

While we are slightly past the initial news cycle, the latest JD Power Initial Quality Study presents a surprisingly positive outlook on short-term vehicle quality. The real test for consumers will be whether these early indicators translate into long-term mechanical reliability. The total number of reported problems with new vehicles improves to 175 Problems per 100 vehicles (PP100) from 192 PP100 a year ago. Despite the overall improvements, software quality remains a distinct vulnerability. JD Power saw a rise to with a total of 44.4 PP100 in the mass market segment and 38.3 PP100 in the premium segment. Infotainment problems per hundred vehicles increased across the board for major automakers. This continued slide in software performance is a troubling indicator, particularly as automotive executives push aggressively toward fully software defined vehicle architectures. This article will explore the nuances of the recent metrics, examine the disconnect between corporate optimism and the ownership experiences shared on social media, and analyze what these changing quality trends mean for the future of the automotive aftermarket.

Keeping Perspective on JD Power Scores

JD Power surveys have their value, but there are nuances to their approach that can significantly change the story about their prescriptions related to quality in the automotive industry. JD Power sends out a 200 question survey to drivers that attempts to ascertain their perception of quality across the major areas of the vehicle. When I was working at GM the most important scores to my team was the infotainment domain. We'd read all of the customer verbatims they had and would categorize them manually. Eventually, we used machine learning to do that work, but for many years we toiled over the abusive comments to find nuggets of feedback that could be used to fix issues.

Once the results are tallied, a normalized score is created per number of vehicles for a given population level. Those population groups could be by mass market, premium, Chevrolet, Buick, or basically any combination of variables. Using a PPH calculation allows JD Power to compare survey results across different categories with varying levels of production.

While there are certainly some useful insights, consumers should be leery of making too many judgements about quality from JD Power.

Sample Sizes are Small Relative to Production Numbers

There simply is not always a lot of data to make statistically valid conclusions about quality from the survey responses. It also is not transparent, except to OEMs, which vehicles and categories had small sampling. We would often have to supplement JD Power results with internal data to get a clearer picture of how much statistical power we actually had when studying results. Many times, it was not sufficient when there 100,000 vehicles produced. There might be 78,000 drivers surveyed but that means very little when it is spread across over a dozen OEMs and 50 different models.

Selection Bias

Some of the most angry and incensed drivers are the ones to respond to a 200 question survey. And, their experience elsewhere in the car can taint their perception of other aspects that might be working perfectly. OEMs have to be extremely careful making decisions off data collected in this manner because it is impossible to have a sense what percentage of customers actually feel a certain way about a category. Selection bias is also why the narrative on quality is so different on social media. Most people do not respond to surveys or post about their negative experiences on Instagram. However, because the scale of social media presences are so much larger than JD Power, some might notice that the optimism shared in the surveys does not comport with the 'automotive journalists'

Certain Ratings are Non-Specific or Vague

Quite a few questions are not answered with specific, actionable feedback from customers. Questions are often cascading through a very dense tree of questions that are arduous to answer and do not have text boxes to answer with specific feedback. Fatigue plays a significant factor which needs to be considered when assessing the validity of results. Some surveys clearly showed patterns of answering the same way for the remainder of the survey. All of this had to be cleaned by the OEM in post processing. Customers would often write 'fuck you' or 'this infotainment pisses me off'. Some customers might just be having a bad day and it gets reflected in the responses.

JD Power Data is a Nightmare for Temporal Analysis

Vehicles tend to be built in buckets--or batches. And usually in a ramp up schedule. When JD Power samples drivers, there is no way to know which software or hardware versions that customer is evaluating. Nor do we know how large the population of each is in those surveyed. You can certainly guess; however, during launches some vehicles are getting new software every day. Manufacturing will update some cars and others get updated at the dealer. Customers will sometimes be asked to come back for an update or wait for an OTA as well. OTA taints the strategy as well because the software shipped with the vehicle can change, which might then also change the customers feedback (for better and sometimes worse).

A further complication for temporal analysis is that by the time OEMs recieve the data, fixes have been put in place for the following year! And, the change management cycles usually line up poorly. Usually many insights that drive engineering changes are not seen or evaluated for 1-3 years. Changes from year to year are liekly just statistical noise.

What these results mean for the Aftermarket

While there are flaws with the JD Power approach, there still might be some interesting insights for those in the aftermarket.

Some Optimism is Warranted for Future Used Vehicles

Because the Initial Quality Study is geared towards the initial builds, any positive feedback means there are ideally less issues making it from engineering to the field. A good thing for the used market. Robustness could still be bad longer term, but we won't know for awhile. But, my prognostication is that if the OEMs launch well, the likelihood of an incredibly impactful spill later is mitigated.

The Silicon-Valle-Fication is Not Working

American OEMs have made massive bets that Silicon Valley holds the keys to software defined success. This early data does not indicate the plan is working. Infotainment quality is getting worse after years of improving steadily. Infotainment performance is a decent proxy for how these teams are going to architect software defined vehicle platforms. We even talk about the alarm bells in our other article: Ford's ITM Recall is a Troubling Sign for the Future of Software Defined Vehicles and the JD Power results bolster the hypothesis. Hiring because of proximity to California is not an engineering strategy.

The Car Conservatory expects that software integration failures will continue to plague the aftermarket. Buyers beware in 2028 or 2029 when the used vehicles start hitting the market that have newer software defined vehicle platforms. You will need a trusted partner in maintaining your vehicle that is capable of recognizing hardware versus software failures. Our shop has the experienced team to guide the right diagnostic strategies to save you time, money, and headaches.