Carcinisation is a personal blog on psychology, survey methods, cultural evolution, and the placebo effect, with a skeptical view of common research claims.
Commenters link its posts to argue that the Big Five replicates but predicts little, and to discuss why surveys are unreliable instruments.
The Big Five is a much less impressive accomplishment than you’d think for how much people talk about it.
https://carcinisation.com/2020/07/04/the-ongoing-accomplishment-of-the-big-five/
> The interesting thing about the Five Factor Model is what it gets away with, in terms of being considered a theory, even though it is not causal, and makes no predictions. What counts as a “replication” of the Five Factor Model, as in Soto (2019), is the following: a correlation is found between one or more factors of the Five Factor Model and some other construct, and that correlation is found again in another sample, regardless of the size of the correlation. In almost all cases, and in 100% of Soto (2019)’s measures, the construct compared to a Big Five factor is derived from an online survey instrument.
> What counts as a “consequential life outcome” is also fascinating. In most cases, the life outcome constructs are vague abstractions measured with survey instruments, much like the Big Five themselves. For instance, the life outcome “Inspiration” is measured with the Inspiration Scale, which asks the subject in four ways how often and how deeply inspired they are. Amazingly, this scale correlates a little bit with Extraversion and with Open-mindedness. Do these personality traits “predict” the life outcome of inspiration? Is “Inspiration” as instrumentalized here meaningfully different from the Big Five constructs, such that this correlation is meaningful?
> With answers as "very often" means very different things to different people.
Agree, I think it's certainly a problem and I too wish it was more defined, but alas. Related, Literal Banana on surveys: https://carcinisation.com/2020/12/11/survey-chicken/
> In addition these general questions that a lot of people can relate to will cause a lot of people to get unneeded screening
Yes, a lot of people have ADHD. I link to the study that shows high specificity in the general population. Empirically, the tool works. You gotta start getting concerned about your symptoms from somewhere.
Well, it’s based on a survey. (And surveys are cursed instruments [1].)
> Generally, owners' equivalent rent is obtained through surveys asking homeowners the following question: "If someone were to rent your home today, how much do you think it would rent for monthly, unfurnished and without utilities?" [2]
I would guess that some owners’ ideas of what their house would rent for might lag the market?
[1] https://carcinisation.com/2020/12/11/survey-chicken/
[2] https://www.investopedia.com/terms/o/owners-equivalent-rent.asp
I think you're examples are great, and you're right, but I also think you're ignoring the specific context of the actual quote. This is a psychologist talking about people in our culture dealing with social issues. So yeah, I think the statement taken as such is strictly false as you noted, but that the meaning is highly indexical[0] and probably has merit.
[0]: https://carcinisation.com/2020/01/27/ignorance-a-skilled-practice/
It's funny I seem to have expressed my thoughts on this very recently [1]. Also a FAQ that you might be interested in [2].
[1] https://news.ycombinator.com/item?id=14205938
[2] https://carcinisation.com/2014/07/13/an-anti-deathist-f-a-q/
I like the scenarios described in https://carcinisation.com/2015/07/23/defensive-epistemology/.
One of them is using the efficient market hypothesis (true enough for this application) to avoid being taken in by a real estate broker.