Oct 28, 2022 03:55 AM
https://www.sciencenews.org/article/soci...measurment
EXCERPTS: I have encountered this type of fuzziness around definitions of all sorts of terms and concepts in the years I’ve covered the social sciences. Sometimes researchers simply assume that their definition of a key concept is the definition. Or they nod briefly at other definitions, and then go forth with whichever one they choose, without much explanation why. Other times, researchers in one subfield choose one definition, and researchers in another subfield choose a different one — each without ever knowing of the other’s existence. It’s enough to drive any reporter to tear their hair out.
“If you look … you will find this morass of definitions and measurements” in the social sciences, says quantitative psychologist Jessica Flake of McGill University in Montreal. My experience was a common one, she assured me.
Definitional morasses exist in other scientific fields too. Biologists frequently disagree about how best to define the word “species”. Virologists squabble over what counts as “alive” when it comes to viruses. And not all astronomers are happy with the decision to define the word “planet” in a way that left Pluto out in the cold as a mere dwarf planet.
But the social sciences have some special challenges, Flake says. The field is a youngster compared with a discipline like astronomy, so has had less time to sort out its definitions. And social science concepts are often inherently subjective. Describing abstract ideas like motivation or feelings can be squishier than describing, say, a meteorite.
[...] Defining one’s key terms and then selecting the right tool is somewhat straightforward when relying on large, external datasets. ... But often social scientists, particularly psychologists, develop their own scales and surveys to quantify subjective concepts, such as self-esteem, mood or well-being. Definitions of those terms — and the instruments used to study them — can take on a life of their own, Flake says.
She and her team recently showed how this process plays out in the May-June American Psychologist. They combed through the 100 original studies and 100 replications included in a massive reproducibility project in psychology. The researchers zoomed in on 97 multi-item scales — measuring concepts such as gratitude, motivation and self-esteem — used in the original studies, and found that 54 of those scales had no citations to show where the scales originated. That suggests that the original authors defined their idea, and the tool used to measure that idea, on the fly, Flake says. Research teams then attempted to replicate 29 of those studies without digging into the scales’ sources, calling into question the meaning of their results.
For Flake, the way to achieve conceptual clarity is straightforward, if unlikely. Researchers must hit the brakes on generating new ideas, or replicating old ideas, and instead interrogate the morass of old ones... (MORE - missing details)
EXCERPTS: I have encountered this type of fuzziness around definitions of all sorts of terms and concepts in the years I’ve covered the social sciences. Sometimes researchers simply assume that their definition of a key concept is the definition. Or they nod briefly at other definitions, and then go forth with whichever one they choose, without much explanation why. Other times, researchers in one subfield choose one definition, and researchers in another subfield choose a different one — each without ever knowing of the other’s existence. It’s enough to drive any reporter to tear their hair out.
“If you look … you will find this morass of definitions and measurements” in the social sciences, says quantitative psychologist Jessica Flake of McGill University in Montreal. My experience was a common one, she assured me.
Definitional morasses exist in other scientific fields too. Biologists frequently disagree about how best to define the word “species”. Virologists squabble over what counts as “alive” when it comes to viruses. And not all astronomers are happy with the decision to define the word “planet” in a way that left Pluto out in the cold as a mere dwarf planet.
But the social sciences have some special challenges, Flake says. The field is a youngster compared with a discipline like astronomy, so has had less time to sort out its definitions. And social science concepts are often inherently subjective. Describing abstract ideas like motivation or feelings can be squishier than describing, say, a meteorite.
[...] Defining one’s key terms and then selecting the right tool is somewhat straightforward when relying on large, external datasets. ... But often social scientists, particularly psychologists, develop their own scales and surveys to quantify subjective concepts, such as self-esteem, mood or well-being. Definitions of those terms — and the instruments used to study them — can take on a life of their own, Flake says.
She and her team recently showed how this process plays out in the May-June American Psychologist. They combed through the 100 original studies and 100 replications included in a massive reproducibility project in psychology. The researchers zoomed in on 97 multi-item scales — measuring concepts such as gratitude, motivation and self-esteem — used in the original studies, and found that 54 of those scales had no citations to show where the scales originated. That suggests that the original authors defined their idea, and the tool used to measure that idea, on the fly, Flake says. Research teams then attempted to replicate 29 of those studies without digging into the scales’ sources, calling into question the meaning of their results.
For Flake, the way to achieve conceptual clarity is straightforward, if unlikely. Researchers must hit the brakes on generating new ideas, or replicating old ideas, and instead interrogate the morass of old ones... (MORE - missing details)