Our aversion to A/B testing on humans is dangerous


EXCERPT: . . . Facebook’s emotional contagion study is just one of several examples of businesses trying to better understand their products and services via randomized experiments, or “A/B tests”—that is, experiments in which subjects are randomly assigned to receive one of two different treatments and then compared on some outcome measure. Like the emotional contagion study, many of these experiments caused public outcry for reasons that may seem intuitive and obvious, such as the lack of informed consent, or the idea that it’s wrong to knowingly treat people unequally. But a recent series of studies from Michelle Meyer and her colleagues casts doubt on these explanations. Their research suggests that people have an irrational aversion to A/B tests, which could limit the extent to which important institutions like hospitals, legislatures, and corporations base their decisions on objective evidence.

[...] Meyer and her team found that in nearly every situation they tested, the decision to conduct an A/B test was deemed least appropriate by a considerable margin. In other words, people preferred those in power to universally implement untested policies at their discretion instead of testing them first. This so-called “A/B effect” held up even in participants with a STEM education or with high levels of scientific literacy.

[...] One possible cause of the A/B effect, according to the researchers, is the “proxy illusion of knowledge,” or the belief that other people know more than they actually do. Participants may have found it unsettling to see people in power admit a need to conduct tests—admit, in other words, that they don’t know enough. [...] It may also be that A/B tests are off-putting because of the cultural baggage associated with science and experimentation. “From the Nazis to Tuskegee to human vivisection to Frankenstein—most of which, by the way, were not actual scientific experiments,” said Meyer, “there’s a colloquial use of the word ‘experiment’ and also ‘random’ that has very negative connotations.”

Whatever the cause, there appears to be a persistent distaste for A/B tests across a wide swath of the population. Yet abjuring randomized experiments, Meyer said, can put more power in the hands of a few. “I want to live in a world where practices and policies and treatments are as evidence-based as possible, and not based on the intuitions of people who happen to become CEO of a company or happen to become head of a hospital,” Meyer said. “There are many attributes that lead people into those positions of power, but magically knowing in advance what does and doesn’t work is not one of them.” (MORE - details)

Possibly Related Threads…
Thread Author Replies Views Last Post
  Illness that affects one in six + Failure of intelligence testing with chimpanzees C C 1 201 Oct 1, 2017 11:28 PM
Last Post: Syne

Users browsing this thread: 1 Guest(s)