When it comes to evaluation asking the right question is crucial, but complex. Some of the most commonly asked questions are also the most unreliable. This means that projects gather misleading data resulting in a poor understanding of the project, the audiences and their impact.
As an evaluation consultant I'm brought in to help people ask the right questions to get the most reliable data for planning, reporting and REF. I also just find evaluation FASCINATING. So when Instagram "specially selected" me to answer a few questions how could I resist! For a company worth $100 billion they clearly don't value audience insight. Here are some of the most problematic questions.
"If you could choose one part of instagram to improve, which would it be?"
What do they want to know: Looking to highlight least popular features and know how to make them better.
a) The question is not optional. You must either answer or discard the entire survey.
b) Respondents may not be well enough informed about the features and functionality to know which need improvement.
c) Without additional information it is impossible to extrapolate the actual issues and improve them.
d) What if you feel two or more areas need improved?
Forcing people to choose will result in many users (myself included) choosing one at random, or the one with the least positive associations, eg "Ads". These random contributions will dilute any meaningful data gathered. Even if 85% of people chose "stories" it would rely entirely on the qualitative question below giving any meaningful justification for the choice.
What should they have done:
a) Where possible all question should have a skip and a don't know option, this allows people the ability to move on without being forced to give a random or least-wrong answer. b) Split the question into several sub questions. Which features do people like, what do they like about them and how does this compare to less popular features? It's the developers job to work out how to improve features, it is at the user's discretion to say how they feel about the them.
"How important is instagram for staying connected to...."
What do they want to know:
Who people are using Instagram to connect with.
a) It has "and" in it.
b) It has "and" in it.
c) IT HAS "AND" IN IT