Morningstar Gets Most Of It Wrong - An Illustration Of The Hazards Of Writing Your Own Survey

Monday, September 19, 2011 16:47
edit
Morningstar Gets Most Of It Wrong - An Illustration Of The Hazards Of Writing Your Own Survey

 

Last week, Morningstar Advisor ran an article reviewing a survey they recently did with 980 investors and discussing some of the conclusions. The article provides a great illustration of the mistakes you can make surveying clients, with examples of the two most common errors advisors make when writing their own.

This Website Is For Financial Professionals Only


 
The article reviews respondents’ answers to the question of what is most important to them about a financial plan. The sample size was relevant, the selection of survey participants seems reasonable, and the numbers are clear. These facts, in conjunction with the fact that it was Morningstar sponsoring the survey, make the first obvious mistake surprising. The author disregards the data and presents the conclusions he wants to be true. One example is the first comment about the relative ranking of Quality of the Systems/Reporting: “Although this is not a highly ranked item, it is important nonetheless.” Let me reword that: The clients did not think this was so important, but they should have. Even more dramatic is the discussion of the response to Stability and Quality of the Firm: “That could be interpreted to mean that clients value advice they get regardless of the stability and quality of the firm. However, I believe that clients want to do business with stable, high-quality firms…” My own inclination is to believe that clients place some importance on the stability and quality of the firm that provides them advice; but it is amateurish to argue against your own data when presenting survey results.
 
Most advisors I see are not so shortsighted as to argue against their clients’ responses. The second mistake of this survey is the one I see most commonly – poorly worded questions. The survey purports to find out what clients value most about a financial plan. However, there are essentially no questions about a plan. All the questions relate to investment management or the firm that provides it. Many advisors, when drafting their own surveys, word their questions in a way that reflects what they believe the answers should be. They inadvertently reflect their assumptions and biases in the questions themselves, skewing the outcomes and providing misleading conclusions. The creator of the survey clearly has an investment management orientation. If he does any actual financial planning for clients, he does not take it very seriously. Consequently, the results of the survey are of no real value to a practitioner who is interested in what clients consider important about financial planning. Beyond the choices that the clients can make to answer this question, there is the issue of the nature of the question: Should this be a multiple choice question without an “other” category, or should there be an additional short answer question to pick up responses not anticipated by the author? It is the unexpected answers that actually provide the most value.
 
Obtaining client feedback is the foundation of improving your service, increasing client loyalty, and generating referrals. Surveying is a science and there is a lot of value that comes from retaining a professional’s guidance. Firms like Advisor Impact can deliver this expertise at a very reasonable cost. If you are sincerely interested in finding out what's important to your clients and using it to improve the experience they have of your firm, an investment in an experts help will pay itself back many times over.

 

Comments (4)

...
agluck
Mr. Wershing, criticism of Morningstar's survey seems reasonable. Not too many venues provide advisors a critical view of a powerful vendor's work product. So thanks.

But there are two sides of every story.

If Morningstar wants to respond, it would be appreciated. Happy to post a guest entry to refute what Steve Wershing says.

The idea is to help readers find truth.


agluck , September 20, 2011
...
stephenw585
I'd love to hear their side of it!
stephenw585 , September 21, 2011
...
steveambuul
I'd love to hear their side of it also!
steveambuul , October 04, 2011
...
agluck
Morningstar's Michael Pompian responded in an email:

The purpose of the survey-based articles I write on MorningstarAdvisor.com website are to (1) present data and (2) provide my interpretation of the data based on my experience. My opinions are not to be taken as "absolutes" nor are they meant to represent any particular side of the business. Obviously there are numerous approaches to providing advice and anyone can draw their own conclusions about the data; I am certain that there will be differing interpretations of the data than mine. I appreciate the responses and am glad the articles provoke thought and discussion.
agluck , October 06, 2011

Write comment

You must be logged in to post a comment. Please register if you do not have an account yet.

busy