Careful with drawing too many conclusions from that book. Your take away is pretty ironic, considering that was Kahneman's problem: "I placed too much faith in underpowered studies" [1]
There's 1 specific chapter that he has publicly acknowledged relied on weak data. You're significantly overstating your point. Here's the actual blog post, with Kahneman responding in the comments (publicly confirmed it's him) if you page back : https://replicationindex.com/2017/02/02/reconstruction-of-a-...
Yes, I've read this book and liked it, although it has a big mistake: Many cited studies can't be replicated. Kahneman even apologized because of that [1].
It's a bummer, but I still believe in many of the listed cognitive biases.
[1] below goes into detail on one of the topics, [2] is a less technical / more journalistic take.
Essentially Kahneman ended up being super confident ("disbelief is not an option" he said) about the findings he cited, some of which have been shown to suffer from lack of rigor.
If you're wondering "well, just some of them right?" I will ask you to ponder for a minute over the fact that this is not supposed to be some impulse aisle magazine article but a book applied epistemology ("behavioural economics" is to me just what gave this and related books some sales wheels).
Without a copy of the book, I don't remember which parts were based on Kahneman’s own work, and I don't see that we can/should just skip the other chapters.
They said [0]: "Table 1 shows the number of results that were available and the R-Index for chapters that mentioned empirical results."
Chapters where estimated R-index < 50: Ch 3,4,6,7,11,14,16
Chapters where estimated R-index > 50: Ch 5,8,9,12,17,24
Chapters that don't cite empirical results (by Kahneman, or who?): 1,2,10,13,15,18-23, all of 25-38
As to the chapters that had empirical results, and had an estimated R-index > 50: scores of 55, 57, 60, 62 are really scraping by; saying that means they "probably replicate" is setting the bar really low, even quoting Tversky and Kahneman (1971) back at themselves.
And take the rest with a bit of salt given that this was Kahneman's comments on the research in that chapter: "Disbelief is not an option. The results are not made up nor are they statistical flukes. You have no choice but to accept that the major conclusions of these studies are true."
"b. Parts of Kahneman's book were undermined by psychology's "replication crisis," which affected some of its findings, but not the idea of system 1 and 2."
Would have been a better footnote if additional references were provided for the latest in that area of discourse.
Although the book still has tremendous value (FWIW, I've read it too), I hope more people also read the above blog (& the comment on it by Kahneman himself), to keep a balance of perspectives and the current "replication crisis" in psychology studies.
Kahneman writes:
[quote]
What the blog gets absolutely right is that I placed too much faith in underpowered studies. As pointed out in the blog, and earlier by Andrew Gelman, there is a special irony in my mistake because the first paper that Amos Tversky and I published was about the belief in the “law of small numbers,” which allows researchers to trust the results of underpowered studies with unreasonably small samples.
[/quote]
> "Table 1 shows the number of results that were available and the R-Index for chapters that mentioned empirical results. The chapters vary dramatically in terms of the number of studies that are presented (Table 1). The number of results ranges from 2 for chapters 14 and 16 to 55 for Chapter 5. For small sets of studies, the R-Index may not be very reliable, but it is all we have unless we do a careful analysis of each effect and replication studies.
Chapter 4 is the priming chapter that we carefully analyzed (Schimmack, Heene, & Kesavan, 2017). Table 1 shows that Chapter 4 is the worst chapter with an R-Index of 19. An R-Index below 50 implies that there is a less than 50% chance that a result will replicate. Tversky and Kahneman (1971) themselves warned against studies that provide so little evidence for a hypothesis. A 50% probability of answering multiple choice questions correctly is also used to fail students. So, we decided to give chapters with an R-Index below 50 a failing grade. Other chapters with failing grades are Chapter 3, 6, 7, 11, 14, 16. Chapter 24 has the highest highest score (80, which is an A- in the Canadian grading scheme), but there are only 8 results.
Regarding Kahneman, keep in mind that in the following years there has been a lot of doubt - including by Kahneman himself - about the scientific validity of the evidence.
Yes, that one. You exaggerate when you say that “most” of the book’s findings were undermined, but a good chunk of them were, including most of the principal findings of one of the chapters. And note that Kahneman had the integrity and objectivity to accept that he had been taken in by some non-replicable results, and has been totally clear, honest, and straightforward about what results that he described in his book are reliable and which should probably be discounted. Maybe he was too credulous when he wrote the book, but no one realized at the time how much of social psychology would collapse. He has provided a model of how an academic should react when faced with uncomfortable evidence.
> The papers that do replicate are tightly clustered a small number of researchers (Kahneman & Tversky)
This doesn't surprise me at all. Kahneman wrote a book, "Thinking, Fast and Slow", in which he spends a long time talking about how most psychologists and social scientists do not understand the basics of statistics and therefore end up with flawed conclusions.
It's been years since I read it (and I didn't actually finish, but that's another story), so I might not remember things very well.
I think some of the examples of bias or other cognitive misgivings that he gives throughout the book could be insightful by themselves. It might be easy to express a general idea in a couple of pages, but concrete examples of the phenomena help bridge it to real life. It could also be interesting as a quick peek at how some of the studies were designed and conducted.
The problem for me is that, since at least some (perhaps many?) of the results Kahneman presents haven't actually been replicable, and I'm not an expert in the field, I have trouble figuring out which of them to believe and which, despite his convincing tone, might not be true after all.
The blog post of the title doesn't actually seem to even mention the problems with replication at all even though they're moderately well known, which kind of makes me want to question whether the "must-read" list has been really considered that critically at all.
Regarding fast and slow:
"It is likely that Kahneman’s book, or at least some of his chapters, would be very different from the actual book, if it had been written just a few years later. However, in 2011 most psychologists believed that most published results in their journals can be trusted."
My point is not that this work is flawed, or that there should not be an article reporting on this research or the topic of interviewing. Rather, I think it'd be better for a third party to write about the topic in a more objective manner, rather than a professor promoting his own research (and thus with skewed incentives).
In particular, I was disappointed to find a (short) paragraph in the article that I find bogus. That does not mean the article shouldn't have been posted in the first place, but just that this paragraph should have been edited or removed.
I think there is something wrong when I feel like I have to look up the actual research paper and check whether the claims made in an article are supported by data and methodology. I should not be a skeptic when reading New York times articles.
To be fair, it is posted in the opinion section, but should we really just take this article as an opinion? That doesn't feel right to me either.
Then onto your last question and Daniel Kahneman, we can talk about that for a long time, but let me keep it short. The best place I know (though technical) is the blog by Andrew Gelman ([1][2] turned up in a 5 second Google, but there is way more on his blog), and Daniel Kahneman himself has "admitted" flaws in his studies [3][4].
I started that book long before the controversy around the reproducibility of some of the sections was widely discussed in public.
I got to the section on priming and immediately thought "what ridiculous bullshit", looked up the original paper and saw that my suspicions we absolutely justified. The statistical analysis of the results is garbage and the experiment design doesn't pass some basic sanity checks.
If this were some sort of purely pop-psych book I would have just brushed it off. Journalists often have to write to a deadline and don't have to time to dive into the details of everything, especially if it makes a good story otherwise.
However I was completely put off by the fact that this was written by someone who made understanding how we reason his life's work. In the book he even says something along the lines of "this sounds unbelievable but you have to accept it!"
I felt somewhat validated that years later this became a more widely accepted criticism but the popularity of both that book and the unquestioned esteem that Kahneman commands really repulsed me. The biggest value of that book is as an object lesson on why we have a reproducibility crisis in the first place. If even prize winning economists/psychologists won't ask the most basic questions about what they're being told there's no hope for the field.
https://replicationindex.com/2017/02/02/reconstruction-of-a-...
reply