Can the richness of your life be boiled down into statistics? In this post, Nathan Palmer explores the challenges of using surveys and quantitative methods to understand the human experience.
“Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.”
In this quote that is often attributed to him, Albert Einstein sounds like a qualitative researcher (a term I’ll explain in a second). The truth is, Einstein was a theoretical physicist and it looks like that quotation actually came from the sociologist William Bruce Cameron. Regardless of who gave us this turn of phrase, the reason it is so often quoted is that it hits at one of the fundamental questions that social scientists often disagree on; can the human experience be measured in numbers?
A Tale of Two Methodologies: Qualitative and Quantitative
All social science research can be broken up into two camps based on the type of scientific method they use to analyze the social world. Quantitative research studies use statistics to measure the human experience. Most often quantitative research collects their data through surveys (e.g. the Census) or they use data collected by institutions (e.g. police arrest records). Qualitative methods most commonly use in-depth interviews and prolonged observations to better understand the motivations and ways of thinking that govern human behavior. Think of it like this: quantitative methods primarily focus on the what, when, and where questions of human behavior, while qualitative methods focus on the how and the why.
To demonstrate the “everything that counts cannot be counted” dilemma, let’s try our hand at using quantitative methods to measure something.
Are You Happy in Your Love Life? Prove It.
Let’s say we are social scientists looking to research how happy people are with their romantic partners (i.e. their spouses, partners, boyfriends, girlfriends, etc.). How could we measure or observe this?
“On a scale of 1 to 10 with 10 being the most happy, how happy would you say you are in your relationship?”
This is a bad survey question. We can’t just come right out and ask people questions like this, because people will might be dishonest. For many people “being successful” means finding a loving partner. So if we just asked, “are you happy in your romantic relationship?” many people might say yes to protect their self-image. When people answer questions falsely on a survey to protect themselves from social shame, social scientists call this the social desirability bias. Instead of coming through the front door with questions like the one above, we should try a side-door approach to minimize the effects of social desirability.
“Over the last seven days, how many days did you and your romantic partner have an argument?”
If a respondent answers seven, does that really mean they have low romantic happiness? Or if a respondent answers zero, does that mean they have high romantic happiness? Can you think of romantic couples who argue all the time, but seem to be very happy despite all the fighting? Some of the older couples in my family are always chirping at one another, but they fight as a way of communicating with one another. They fight because they care. On the flip side, most of us have known a romantic couple that never fight because they have checked out of the relationship. They don’t fight because they don’t care enough about the other person to get angry.
In the real world, sociologists would probably not use just one question to assess romantic happiness. But the point here is that regardless of how many questions you use the vastness of the human experience resists being quantified.
We can see the limitations of quantitative research in the expression “when the only tool you have is a hammer, everything starts to look like a nail.” Instead of measuring the human experience by collecting the datapoints that best reflect reality, quantitative methods often collects the datapoints that are most easily quantified. In our example, the number of arguments is easily quantified, but not necessarily the best representation of romantic happiness.
Qualitative researchers have long argued that to truly understand the human experience, researchers must use more in depth methods. And while it’s true that there has been a long cold war raging between quantitative and qualitative researchers with each side thinking they have the better approach to exploring the social world “accurately”, we should remember that these two approaches are often complementary. Qualitative measures often explore the areas of life that aren’t easily quantified or easily measured and then quantitative researchers use the findings of these qualitative explorations to better quantify their measurements of the humane experience. Mixed Method studies (i.e. studies that use both qualitative and quantitative) are on the rise as scientists embrace their complementary powers. In the end, both approaches help us better understand the world we live in.
- Create a list of 5 sociological topics that would be hard to quantify into survey questions. Explain why each would be challenging.
- Quantitative survey questions are often “closed-ended” (i.e. you are forced to pick one answer from a list of provided response options). How could forcing a survey respondent to pick from a list of choices be seen as limiting their ability to share their experience with the researchers?
- Write a closed-ended quantitative survey question that could measure college student’s procrastination.
- Let’s say you wanted to quantitatively research how often college students illegally downloaded music, movies, and media files. How might the social desirability bias impact how your respondents would answer your questions? How could you write a survey question to minimize the effects of social desirability?