It’s graduation season! That means caps, gowns, and awards ceremonies that sometimes include celebrity commencement speeches. Oprah Winfrey spoke at Harvard University this year, and was also awarded an honorary degree by the prestigious institution. One of the things that Oprah is respected for is speaking with passion. This skill is not only important for public speeches, but also for her current television show, Oprah’s Next Chapter. The show includes interviews with celebrities in their homes, reminiscent of the interviews she was known for on the Oprah Winfrey Show. Oprah’s interview skills often make for great television, but would a sociologist use the same methods for interviews in social research? In this post, Mediha Din describes research methods.
In our previous post, Jimmy Kimmel, Starbucks, and Sleeper Questions we explored how researchers may encounter problems with reliability when conducting an interview or surveying participants.
To increase reliability, sociologists work to avoid common errors in the design of research questions. The most common mistakes are using questions that are double-barreled, loaded, threatening, unclear, or include built-in assumptions.
Double-barreled questions make the mistake of asking two or more questions in one. For example, if you ask your friend “Is Professor Din’s class easy and interesting?” the response “yes” is unclear. The class might be easy, but not interesting, or the other way around. As sociologists, we work to ensure that double-barreled questions are broken into two separate questions to avoid an unclear response: “Is the class easy?” and “Is the class interesting?”
“You read the whole chapter, didn’t you?” Leading or loaded questions should also be avoided. These are questions that subtly push a respondent to give a certain response. Think of movies with courtroom scenes of an attorney jumping up while their witness is being questioned by the prosecution to say “Objection your honor! Leading the witness!” Leading/loaded questions can be worded to get either a positive or negative response. Compare the next two examples:
“Should the president of the college fix the dangerous and potholed streets around campus?”
“Should the president of the college spend even more money on repairing streets when the students are in need of those funds for classes?”
In the first question, by adding qualifying terms like “dangerous” you lead a respondent to say yes. Contrastingly, by saying “even more money” you lead the respondent to believe that perhaps money is being wasted, and lead them to say no.
A researcher also has to be aware of questions that may be worded in a way that is threatening: “Have you ever compromised your academic integrity and violated the college code of conduct by cheating on exam?” sounds very different than a simple “Have you ever cheated on an exam?” It is preferable to keep questions straightforward, especially if the topic may be sensitive.
Built-in assumptions should also be avoided in questions. When Oprah recently interviewed singer Shakira, who is a coach on the hit T.V. show The Voice, and also a new mother, Oprah asked “How has motherhood changed you?” For an average conversation or television interview, this question is fine. For a sociologist however, we would not want to make the assumption that Shakira feels motherhood has changed her. We would first want to ask “Do you feel motherhood has changed you?” Then we can ask “How has it changed you?”
Here is another example:
Mark one response in regards to the following statement:
The textbook we use for our class is well-written.
Strongly Agree, Agree, Disagree, Strongly Disagree
Here the assumption is that everyone responding has read the text book, which we all know is honestly not likely. The option “No Opinion” or “Not Applicable” should be included in this type of question. Contingency questions can also be used to avoid assumptions. Contingency questions are two or more part questions:
1. Do you study in the campus library? ____(If yes, answer question 1a, if no skip to question 2)
1a. How often do you study in the library? _____________
2. Do you study alone?_________________
Using the contingency question avoids the built-in assumption that would occur if the question asked was “How many days a week do you study in the campus library?”
Unclear questions also include assumptions. They assume that the respondent understands how you define a term. If I ask, “Do you study with friends often?” my idea of “often” may be very different than yours, so it is important that I clarify any ambiguous terms.
Technical jargon should also be avoided in research questions because the respondent may not be as familiar with an industry as the researcher.
For example, when asking students to evaluate teachers, the question “Is the pedagogy of this instructor effective?” may not seem problematic to educators because we use the term “pedagogy” (the art, method, or science of teaching) often, but students might not be as familiar with the term.
The next time you watch an interview on television, keep your eye out for questions asked that are double-barreled, loaded, threatening, and include assumptions. Sociologists work hard to design questions that avoid these mistakes.
How a researcher designs every part of their study has a significant impact on the validity of his/her results. Designing questions for surveys or interviews carefully, while keeping in mind the potential pitfalls, is extremely important.
- Watch a television interview (You can find clips here of Oprah’s Next Chapter– interviews with Shakira, Usher, Adam Levine, Tyler Perry, Jason Collins, Jane Fonda, and many others.) Write down three of the questions asked and analyze the strength of the questions. Would a sociologist ask the questions in the same way, why or why not?
- Read the Basics of Survey and Question Design . Find an actual survey about any topic and discuss if it has been created following these guidelines.
- Watch this video about designing survey questions and create a survey for friends or family on a topic of your choice. You can use a free service like Google Docs Forms or SurveyMonkey.
- Read the sample survey questions attached below. Look for questions that are weak and re-word any that are loaded, double-barreled, threatening, sensitive, or otherwise problematic.