Sunday
May272018
RS 209 - Christopher Chabris on "Collective intelligence & the ethics of A/B tests"
Release date: May 27th, 2018
Christopher Chabris
This episode features cognitive psychologist Christopher Chabris discussing his research on "collective intelligence" -- why do some teams perform better than others at a wide variety of tasks? Julia discusses potential objections to the findings and how gender-related publication bias should affect our interpretation of them. In the second half of the episode, Julia and Chris discuss why people get so upset at companies like Facebook and OKCupid for doing experiments on their users, and whether that's fair.
Chris' Website
Edited by Brent Silk
Music by Miracles of Modern Science
Full Transcripts
10 Comments
Reader Comments (10)
Definitely socially aggressive people need to learn how to cooperate with others. A good leader will learn how to shut up and listen to others, and thereby maximize the benefit from the knowledge and skill of other people.
Julia makes such a good point about how a study that found groups benefit from more women would go to publication, but that a study that found women decrease the performance of groups would not go to publication. I would not want to be the fellow who publishes a study that shows women perform poorly in group performance or leadership roles, no matter how good the data !!
People probably dislike A/B Tests due to their inherent sense of Fairness. They misperceive A/B Tests as an unfair favoritism towards one group, and an unfair prejudice towards another. If two new promising therapies or processes develop, we need to compare the two with RCTs. Facebook should run A/B Tests. OKCupid almost necessarily has to run covert A/B Tests since their service involves wholly psychological results. Running a covert medical test involving treatment of a physical ailment without disclosing the possibility of a placebo would definitely create an ethical issue.
However, as Chabris mentions, if we have a new therapy or process A, or perhaps two new therapies A and B, and A and B clearly have superiority over the old standard therapy L, instead of requiring RCTs that include L versus A or B, we should only have RCTs with A versus B, and then compare those results to the Legacy Results from earlier RCTs, case reports, and other evidence of the efficacy of L. The FDA, however, usually strictly requires RCTs that include L, and frequently requires more than a decade of research before permitting widespread use of A or B. This is how the FDA kills a lot of people.
Implicit Bias Tests really don't work. Furthermore, all types of bias training usually fail. Really, Starbucks took the actions it took to save face and preserve its reputation. This has much more to due to marketing and brand protection, essential to customer retention and profits, than it has anything to do with a real concern for eliminating bias.
A General Disclosure of cover A/B experiments in every service agreement might make people feel better about the whole process. People would have to actually read the service agreement, or have someone point out this provision to them.
Gladwell's failure to consider an empirical test of his politically correct intervention in incarceration policy, see link above, is an excellent example of the failure of intellectuals to accept responsibility for the 'ideas' that they promulgate, and the failure to empirical test same, that Thomas Sowell and Nassim Taleb critique.
I find this piece of the Gladwell interview[1] telling:
<Begin quote ...>
Q. Your critics argue that your ideas are simplistic or overreliant on anecdotal evidence. Is that something you’ve taken to heart and tried to improve on?
A. Of course I rely on anecdotal evidence and simplify things; that’s what it means to be a popular writer.
<... end quote>
... my jaundiced ear hears, instead, 'you don't expect me to think-out complicated stuff and write it down for my audience of goodthinking humanities majors do you?
[1] https://www.nytimes.com/2018/05/29/magazine/malcolm-gladwell-likes-things-better-in-canada.html