When it comes to evaluation asking the right question is crucial, but complex. Some of the most commonly asked questions are also the most unreliable. This means that projects gather misleading data resulting in a poor understanding of the project, the audiences and their impact.
As an evaluation consultant I'm brought in to help people ask the right questions to get the most reliable data for planning, reporting and REF. I also just find evaluation FASCINATING. So when Instagram "specially selected" me to answer a few questions how could I resist! For a company worth $100 billion they clearly don't value audience insight. Here are some of the most problematic questions.
"If you could choose one part of instagram to improve, which would it be?"
What do they want to know: Looking to highlight least popular features and know how to make them better.
Question issues:
a) The question is not optional. You must either answer or discard the entire survey.
b) Respondents may not be well enough informed about the features and functionality to know which need improvement.
c) Without additional information it is impossible to extrapolate the actual issues and improve them.
d) What if you feel two or more areas need improved?
Question result:
Forcing people to choose will result in many users (myself included) choosing one at random, or the one with the least positive associations, eg "Ads". These random contributions will dilute any meaningful data gathered. Even if 85% of people chose "stories" it would rely entirely on the qualitative question below giving any meaningful justification for the choice.
What should they have done:
a) Where possible all question should have a skip and a don't know option, this allows people the ability to move on without being forced to give a random or least-wrong answer. b) Split the question into several sub questions. Which features do people like, what do they like about them and how does this compare to less popular features? It's the developers job to work out how to improve features, it is at the user's discretion to say how they feel about the them.
"How important is instagram for staying connected to...."
What do they want to know:
Who people are using Instagram to connect with.
Question Issues:
a) It has "and" in it.
b) It has "and" in it.
c) IT HAS "AND" IN IT
Any question with the word "and" is two questions.
d) Missing "don't know" and "skip"
Question result:
This will give fairly meaningless data. I use instagram to stay connected to friends, this is important to me. I am not connected to any family members on instagram and this is not important to me. What should I have answered?
In theory the only person capable of answering this would have the same answer to both "Friends" and "Family" but in practice there is no way to interpret the meaning of the respondent.
What should they have done:
a) Where possible all question should have a skip and a don't know option.
b) Separated out any question using "and" into two questions.
"How useful is Instagram overall?"
What do they want to know:
Best guess, they are fishing for praise.
Question issues:
a) It is too broad to give meaningful data.
b) Missing "don't know" and "skip"
c) Bias: four options indicate Instagram is useful, one option indicates that it is not.
Question result:
Forcing people to answer will mean the inclusion of meaningless data and dilute any meaningful response. Also asking such a broad question is likely to produce a result which means something different to each respondent, rendering the collective response meaningless. Finally introducing bias with four out of five options indicating the that Instagram is useful the results are likely to be unrealistically positive.
What they should have done:
Asked about satisfaction related to certain features and functions. Included a Don't Know and Skip option.
"Instagram is developing breakthrough technology"
What do they want to know:
They want check if their brand is associated with excellence in development.
Question issues:
a) It is unlikely that the average user of Instagram knows what technologies the company currently have in development and how they relate to the current tech landscape.
b) Missing "don't know" and "skip"
Question result: I don't believe that the people being asked will be well enough informed on what Instagram are currently developing to give a reliable answer. Most people will be forced to guess (as they can't say that they don't know) meaning the answers are likely to be broadly neutral.
What they should have done:
Firstly they shouldn't ask a question likely to be outside of the average user's knowledge and expertise. If they wanted to know about brand recognition/association they could have asked for words associated with their brand. They should also have included don't know and skip options.
If you enjoyed this little dissection of the instgram survey you might also enjoy this twitter thread on the Evaluation forms from the NCCPE's Engage Conference.
If you have any questions, comments or want my opinion on a survey that you are about to run then just get in touch.