Jimmy Kimmel, Starbucks, and Sleeper Questions
Late night television shows, news specials, and talk shows all include segments from time to time that are labeled as “social experiments.” But do their surveys, interviews, or investigation tactics have any connections to actual research methods sociologists would use when studying human participants? In this post, Mediha Din discusses the complexity involved in designing survey questions.
Jimmy Kimmel sets up an experiment. He wants to know if the new $7 cup of coffee offered at Starbucks lives up to the hefty price tag. His crew heads to Hollywood Blvd. in Los Angeles to ask coffee drinkers for their opinions. The participants sit down in front of two Starbucks cups, labeled A and B. They are asked to take a sip from each, and see if they can tell the difference between the regular coffee, and the premium $7 Costa Rica Finca Palmilera. The participants taste the beverages and quickly begin giving their feedback on which is the more expensive brew. They describe their choice as “richer” “bolder” “smoother” even “beany.” His experiment has one very interesting catch though-the cup of coffee in both cup A and cup B is exactly the same!
The video of this segment from Jimmy Kimmel Live isn’t just a good laugh, it’s also a perfect example of a type of question used in social research known as a sleeper question.
A sleeper question is designed to ensure that a respondent is accurately reporting their knowledge. The question can be about non-existent people, events, or places to determine honesty. How is this done? Here is an example:
Take a look at the images of Civil Rights leaders below and rate your knowledge for each as a, b, or c.
a. I recognize this civil rights leader and know his or her name
b. I recognize the leader but can’t remember his or her name
c. I don’t recognize this leader
3 
5 
6 Image 5 is an example of a sleeper question. All of the other images are of actual civil rights leaders (Martin Luther King Jr, Mahatma Ghandi, Dolores Huerta, Malcolm X, and Cesar Chavez). Image 5 is a random image from Wikipedia Commons of an African-American businessman named Chuck Smith. If you created a survey question like the one above, you would look at how many respondents claimed to recognize image 5, and you may decide that all of their responses may be invalid or dishonest. If respondents give false information about the sleeper question, it can be assumed that they may have been dishonest about other questions in the survey.
This type of sleeper question was actually used in a study to determine which civil rights leaders U.S. adults recognize. The name of a fictitious person was added to the survey, and 15% of respondents said they “recognized” the leader.
If Jimmy Kimmel actually wanted to survey participants regarding coffee flavor, he may find that only the responses from the last participant (the guy who admitted he couldn’t taste a difference between cup A and cup B) are valid.
So why did the others claim to notice a difference? The participants aren’t pretending to taste a difference between the cups of coffee because they are generally dishonest and misleading people. It is actually quite common for respondents to give inaccurate information on surveys, for a variety of reasons. They might be embarrassed, misunderstand a question, be in denial about the truth (especially if the questions are about addictions), not remember the correct information, want to give the researcher the answer they feel he/she is looking for, or simply respond by selecting the answer they wish was true.
This happened to me just today when I was a purchasing a new pair of shoes from a running specialty store. The clerk asked me a series of questions that she was inputting into the computer system to help me select the right running shoe for my needs. She asked how many miles I run a week. As I was thinking, I glanced at the computer screen and saw the choices were: a. 1-10 miles b. 11-20, or c. 21 or more miles. I usually run 9 miles a week at the most, but having run cross-country all through high school, and considering myself ‘a runner’, I somehow didn’t feel good about selecting what I considered to be the “beginner” response. I thought to myself “Well, I know I CAN run more than 9 miles if I want to, it’s just that I’m also spending time cycling and swimming.” So I told her to select 11 or above. Why? I knew she wasn’t judging me, I knew no one else would even see my responses, and I knew the questions were only meant to help me select a sneaker I would be happy with; but somehow I just didn’t want to tell her the truth.
Along with exaggerating exercise habits or how many servings of green vegetables one eats, it is also very common for people to downplay “bad” habits. When asking respondents how much time they spend each week watching reality television shows, how often they smoke, or if they eat fried food-a researcher must be aware that respondents may not always be completely honest, even if the survey is anonymous.
For this reason, sociologists often use sleeper questions in surveys, or other methods of data collection (such as participant observation) to increase the validity of their data.
If I ask my students questions about how many hours they study at the library each week, I may get one piece of the picture from their responses. However, if I also observe students at the library, I may find out a lot more. I might see that some students don’t go to the library nearly as often as they report. I may also see that some spend 2 hours in the library, but only study 1 hour, while the other hour is spent texting, emailing, on Facebook, or talking with their friends.
Sociologists also work to design research questions that are not loaded, double-barreled, threatening, sensitive, unclear, include built-in assumptions, double-negatives, or technical jargon, but that’s for another day.
Dig Deeper:
- Watch a television interview (Oprah’s Next Chapter- Interview with Lance Armstrong for example). Write down three of the questions asked and analyze the strength of the questions. Would a sociologist ask the questions in the same way, why or why not?
- Find an actual survey about any topic and design a sleeper question for the survey that would help a researcher ‘weed out’ dishonest respondents.
- Watch this video about designing survey questions and create a survey for friends or family on a topic of your choice. You can use a free service like Google Docs Forms using SurveyMonkey.
- Think back on how I answered the clerk’s question about how many miles I run a week. Do you think that when people take surveys they answer the question that is asked or that they interpret what they think the question means and answer that question? How might this affect how we design surveys?