Search Episodes
Listen, Share, & Support
Listen to the latest episode
Subscribe via iTunes
Subscribe via RSS
Become a fan
Follow on Twitter

Support Us:

Please consider making a donation to help make this podcast possible. Any contribution, great or small, helps tremendously!

Subscribe to E-Mail Updates

Related Readings
  • Answers for Aristotle: How Science and Philosophy Can Lead Us to A More Meaningful Life
    Answers for Aristotle: How Science and Philosophy Can Lead Us to A More Meaningful Life
    by Massimo Pigliucci
  • Nonsense on Stilts: How to Tell Science from Bunk
    Nonsense on Stilts: How to Tell Science from Bunk
    by Massimo Pigliucci
  • Denying Evolution: Creationism, Scientism, and the Nature of Science
    Denying Evolution: Creationism, Scientism, and the Nature of Science
    by Massimo Pigliucci

RS 192 - Jesse Singal on “The problems with implicit bias tests”

Release date: September 3rd, 2017

Jesse Singal

You may have heard of the Implicit Associations Test (IAT) -- one of the most famous instruments from social psychology, it's frequently cited as evidence that most people harbor implicit racism or sexism, even if they aren't aware of it. This episode features science journalist Jesse Singal, who argues that the IAT has been massively overhyped, and that in fact there's little evidence that it's measuring real-life bias. Jesse and Julia discuss how to interpret the IAT, why it became so popular, and why it's still likely that implicit bias is real, even if the IAT isn't capturing it.

Jesse's Pick: "Galileo's Middle Finger" by Alice Dreger

Jesse's Article: "Psychology’s Favorite Tool for Measuring Racism Isn’t Up to the Job"

"My IRB Nightmare" by Scott Alexander

Edited by Brent Silk

Music by Miracles of Modern Science

 Full Transcripts 

Reader Comments (24)

In 2013, I repeated their test ranking four races, twice in a row.
First time, Whites were presented first, and in the end they were near the top, and Hispanics near the bottom.
Second time, Hispanics were presented first, and in the end they were near the top, and Whites were near the bottom.

Their FAQ says the influence of order is small, but in my case it was big. They talk about reversing the order, but with four races there are many more permutations.
"Question 1. Could the result be a function of the order in which I did the two parts? I had to group one category together with pleasant words first. I then found it difficult when I later had to group the other category with pleasant words.
Answer: The order in which tests are administered does make a difference to the overall result in some tests. However, the difference is small and recent changes to the test have sharply reduced the influence of order. Because of this order effect, the orders used for IATs presented on this website are assigned at random. For any data we present, we are careful to be sure that half the test-takers got the A then B order and the other half got the B then A order. With the revised task design, the order has only a minimal influence on task performance. If you want to check whether the order made a difference for you, you can take the test again and complete it if you get assigned to the reverse order. If you do take the test twice in different orders and get different outcomes, the best estimate of your result is intermediate between the two. For more information about the order effect, see this paper (Nosek, Greenwald, & Banaji, in press)."

They say right there: "If you want to check whether the order made a difference for you, you can take the test again and complete it if you get assigned to the reverse order."
So that's what I did, but how many people will do that? It's like getting out a ruler to measure whether a "footlong sub" is really 12 inches long.
September 3, 2017 | Unregistered CommenterMax
Two thoughts:

1. If women have negative immediate reaction to images of other women, the first thing I would look at is sexual competition. Women compete on looks a lot more than men, and maybe an instinctual "is she prettier than me?" reflex kicks in. Men compete too, of course, but in different ways, and not nearly as much on looks.

2. I heard one interesting theory on the white/black resume test. All distinctly "black" names are also "poor" names. So does "black poor" Jamal lose to "white middle class" Jake because of the race or the class? You'd find out by throwing "white poor" Billy-Bob into the mix. I assume that's been done, but I haven't heard of it.
September 3, 2017 | Unregistered CommenterLars
Companies like Google like the idea of implicit bias because it explains why women, blacks, and Hispanics are underrepresented there, without admitting that the company engages in illegal discrimination that would invite a lawsuit, and without pointing to biological differences which is enough to get their employees fired.
So they'll teach you that people have an implicit bias in favor of tall white handsome men, which is why so many CEOs are tall, but it's sexist to suggest that a woman got preferential treatment because she's attractive. As Elissa Shevinsky wrote, "Title VII of the Civil Rights Act prohibits employment discrimination based on race, color, or gender. And speech questioning the technical qualifications of people based on race or gender arguably falls under this category." Unless the people are tall white guys, then it's just implicit bias training.
September 4, 2017 | Unregistered CommenterMax
Roland Fryer and Freakonomics guy Steve Levitt wrote "The Causes and Consequences of Distinctively Black Names."

"The last two decades, however, have led to a 'ghettoization' of distinctively Black names, namely, a distinctively Black name is now a much stronger predictor of socio-economic status. Among the theories we consider, models in which the rise of the Black Power movement triggers important changes in Black identity appear to be most consistent with our data. In contrast with prior audit studies of Black names on resumes, we find little evidence that names have a causal impact on adult life outcomes. More generally, this paper takes first steps towards an attempt to understand what role Black culture might play in explaining continued poverty and racial isolation. With respect to this particular aspect of distinctive Black culture, we conclude that carrying a black name is primarily a CONSEQUENCE rather than a cause of poverty and segregation."

Roland Fryer also wrote "An Empirical Analysis of Racial Differences in Police Use of Force," which concluded that although blacks and Hispanics are more likely to experience police use of force than whites, they were not more likely to be shot by police than whites.
September 4, 2017 | Unregistered CommenterMax
Criticizing the IAT must be a microaggression, almost as bad as not dismissing IQ tests, which used to be championed as a tool to identify talented students from underprivileged backgrounds.

Who came up with microaggressions, safe spaces, and trigger warnings anyway? Sounds like psychobabble.
September 4, 2017 | Unregistered CommenterMax
Jessie Virtue Singal? Thoughtful and literate? I think not. He repeats what others tell him. He is a grocery clerk who does not understand what he writes, furiously virtue singaling that he is a good person.
September 5, 2017 | Unregistered CommenterAsh
Has anyone tried modifying the Race IAT to use white and black squares or cats instead of faces? Because our culture does associate the color black with evil, black magic, the dark side, etc., in contrast with white knights, unicorns, and angels in white robes. For that matter, the color red may also be associated with anger and hate and hell, but it's not due to racism.

There are video games where the left and right keys can suddenly swap, so pressing the left arrow makes you go right. I bet people who are good at such games will show less bias on the IAT, which likewise forces you to mentally swap left and right in the second half.
September 5, 2017 | Unregistered CommenterMax
OK, I took the Race IAT again a couple of times. Yesterday it said I had a slight preference for whites, like 17% of web respondents. Now it says I have a moderate preference for blacks, like 5% of web respondents, even though it started out with white/pleasant associations in the first half, where the test ITSELF trains you to associate white faces with pleasant words.

What a joke. Worse than the polygraph lie detector, which was also used to screen for racism by a police department. Their police chief boasted, "I think the polygraph will definitely keep these people from applying."
September 5, 2017 | Unregistered CommenterMax
That the IAT would find liberal women have more "bias against women" than conservative men seems to me a perfectly intuitive outcome!

Some of the liberal women I know essentially make worrying about bias against women a central organizing theme of their life. If there are exist ANY negative stereotypes about women, these activists spend a LOT of time thinking about and reading about and arguing about and listening to discussions about or premised on these stereotypes. This makes those stereotypes SALIENT.

If you are the sort of person who has gotten a degree in gender studies and has read a dozen books on feminism and has attended panel discussions with names like "women in the workplace", it is quite easy for you to come up with examples of negative stereotypes against women; those associations have been heavily reinforced by stuff you think about every day!

By contrast, the average ultraconservative man probably thinks men are different than women in various way and might reflexively accept a few of those stereotypical beliefs, but he doesn't spend nearly as much time THINKING ABOUT those beliefs so they are LESS salient.

Additionally, our hypothetical liberal activist is (painfully) aware of ALL the negative beliefs anyone might have on this subject - and of the very worst versions of those beliefs. Whereas our hypothetical conservative - having done less research - is only aware of the negative belief forms to which he himself subscribes. He can model his own particular middling views, she can model all the worst views anyone ever had.

Given that context, it seems plausible the IAT could actually be measuring "how badly sexist are the views that most quickly come to your mind and afflict your reasoning process" and produce the result that liberal women's thought processes tend to be "more sexist" than conservative men. Right?
September 6, 2017 | Unregistered CommenterGlen Raphael
Elevatorgate showed that liberal feminists don't hesitate to profile men as potential rapists, and are afraid of men in an elevator. Offended men quickly made an analogy to profiling blacks as potential robbers, but it's true that men are much more likely than women to sexually assault a woman in an elevator, and blacks are more likely than whites or Asians to rob or murder someone, especially another black person. But it's still pretty unlikely that a random man is a rapist or that a random black person is a robber, though there are factors that can raise the probability. A drunk woman is more vulnerable and a drunk man is probably a bigger threat, and a young black man with face tattoos wearing a hoodie in warm weather in a high-crime neighborhood is probably a bigger threat than a random black person. It's common sense.
September 6, 2017 | Unregistered CommenterMax
Your article reflects the issue people are concerned about. The article provides timely information that reflects multi-dimensional views from multiple perspectives. I look forward to reading quality articles that contain timely information from you.
September 8, 2017 | Unregistered Commenterwings io
I do believe in implicit bias on many fronts, its not just limited to race, gender, ethnicity or sexual orientation but also based on characteristics such as beauty, success, preferring hiring graduates from ivy league colleges etc. I struggle to understand how people can make decisions small or big without having any implicit bias/preference that is based on reason in one's brain. A reason that is completely derived from environment. May be a stretch to associate the bias to evolutionary psychology but isn't implicit bias natural and necessary for human brains to function effectively?

I have missed the data/evidence presented from a scientific social research, could you please link to any kind of reliable evidence showing the weakness of the design of the most IAT tests as stated in the podcast? Has anyone come up with a better way to measure implicit bias yet?
September 11, 2017 | Unregistered CommenterSunee
Regarding women and stereotypes — what was measured here? Negative stereotypes against women or also positive stereotypes? Positive discrimination is a thing. If it's negative, I agree with another commentator here — intrasexual competition springs to mind. And BTW, what are the effect sizes?

Regarding cannot game but can train — sure, compare it with weight. If you measure it correctly (scales) you cannot game it, but you can "train" your weight (diet). These two aren't mutually exclusive. You cannot use retest reliability if the thing that you measure changes (e.g., a state, or again, take weight).
September 11, 2017 | Unregistered CommenterDaniel
When you take the IAT, they warn you that results may vary depending on various factors like how tired you are. Aspects of the test are also deliberately varied, like half the test-takers get a test that compares blacks and whites, and half get a test that compares European Americans and African Americans. Half the test-takers answer survey questions before the test, and half do it afterwards.
Well if such small changes are enough to completely flip your test result, then I doubt that it measures deep-seated racism or bias.
September 11, 2017 | Unregistered CommenterMax
Nice episode. The first one I've listened, btw.

I knew you from youtube, but I had no clue that you also have a podcast. I came here from a feedback that a listener gave to an episode of another great podcast (You are not so smart 110, episode "Sleep Deprivation Bias",

Just two quick feedbacks for the RS 192:
1. I thought about the "The Dragon in My Garage", when you guys were kind of defending the premise that the method tests.
2. Maybe a biased academy, specially on humanities, helps to perpetuate wrong finds or to give disproportional weight to a factor. Check
September 13, 2017 | Unregistered CommenterAlexandre Gandra
"Only commonly given black names from lower social status origins are a strong signal of a person’s race. We are sending signals of both social class and race when we use names like Lakisha and Jamal."
September 15, 2017 | Unregistered CommenterMax

Do people named "Billy-Bob" really put "Billy-Bob" on their resumes, or "William Robert"?
September 20, 2017 | Unregistered CommenterJ. Goard
This is the problem that I am looking to learn. May more people share more. I will follow up on this topic
September 22, 2017 | Unregistered Commenterwings io
Lars and Daniel: Regarding the idea that intrasexual competition explains women's anti-women answers on the IAT: this presumes that the woman doing the test sees other pictures of women as external agents with which she is in competition. Perhaps, if every woman pictured in the IAT was attraction, this could be the case. This would be quite a methodological flaw imo, because if you're trying to draw impressions about women as a whole, displaying only pictures of attractive women will produce some pretty skewed results.

I think another possible explanation is that, rather than the participant evaluating these women as OTHERS, she is instead responding as a MEMBER OF THE SAME DEMOGRAPHIC. That is, she is responding to implicit biases about the group of "women" as a whole.

Anecdotally, I am highly conscious of negative stereotypes about women, more so than I would expect men to be, because they directly affect me. If a word that equates "woman" with "homemaker" comes up, I'm perhaps more likely to make this connection, because this stereotype affects me (negatively, no less) and is thus more salient in my mind.

Related to this thought is the idea of stereotype threat. Research has indicated that when primed with stereotypes about one's own demographic, people internalize the stereotype so that it, in a way, "comes true." For example, when women are primed with the idea, "Women are bad at math," they perform worse on math tests.
November 1, 2017 | Unregistered CommenterLaura
I found a lot of interesting information here. A really good post, very thankful and hopeful that you will write many more posts like this one.
November 1, 2017 | Unregistered Commenterbloxorz
click and download your office setup from ,if you face any error while installing office setup call toll free - +1-855-550-9333
November 16, 2017 | Unregistered Commenteralex khushi
This from the Harvard IAT website at the end of the test (26 NOV 2017):

"Why Should I Care About My IAT Score? Implicit preferences can predict behavior. Implicit preferences are related to discrimination in hiring and promotion, medical treatment, and decisions related to criminal justice."

I just took the test today, 26 NOV 2017, so apparently the Harvard IAT team still claims their test works. Furthermore, for these types of issues, Employment and Guilt, DOES implicit bias really matter as much, since after all, we would expect a non prejudicial HR Manager or Juror to actually analyze someone's resume or case before rendering a judgment. So what Julia proposed, using the resume method or case analysis method would show racial prejudice in a way that actually matters towards society.

Additionally, we already know that people favor good looking tall people, so don't we really need a better system to judge potential employees and guilt anyway? Should we judge resumes without the name/picture?

Finally, when I took the test, I found the first part really really easy. However, once the test reversed the I and the E, I made lots of mistakes and had lots of delays due to the tremendous confusion. The Harvard IAT dismisses this effect as minimal, but is it really?
November 26, 2017 | Unregistered CommenterJameson
These are great experiences. You made me write a lot better thanks
February 26, 2019 | Unregistered Commentercandy crush soda
Nice blog, Get you website designing responsive and creative and also digital marketing services at ogen info system Delhi, India.

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
All HTML will be escaped. Hyperlinks will be created for URLs automatically.