OtherGround Forums The Replication Crisis in Psychology

3 days ago
1/1/01
Posts: 10795

It's interesting times we're living in, particularly when we're discovering that so many of the famous experiments and ideas that have been considered "stable" in the realms of psychology are in fact either misleading or entirely wrong.

- The whole Myers-Briggs personality test is bullshit:

https://www.psychologytoday.com/us/blog/give-and-take/201309/goodbye-mbti-the-fad-won-t-die?page=1

- The Stanford Prison Experiment has been debunked:

https://psycnet.apa.org/record/2019-45337-001

https://www.livescience.com/62832-stanford-prison-experiment-flawed.html

- The Milgrim experiment used to demonstrate how easily people follow orders was at the very least heavily flawed:

https://digest.bps.org.uk/2017/12/12/interviews-with-milgram-participants-provide-little-support-for-the-contemporary-theory-of-engaged-followership/

- That famous rat experiment about addiction was heavily flawed and entirely wrong about the causes of addiction:

https://theoutline.com/post/2205/this-38-year-old-study-is-still-spreading-bad-ideas-about-addiction?zd=1&zi=7shmhkm5

-------------

And now we have a broader replication crisis that pokes holes in everything from the role of "power posing" to numerous other "right sounding" tidbits we've picked up from psychology over the last few decades:

https://en.wikipedia.org/wiki/Replication_crisis

---------------

While it can be disturbing to see that much of what we thought we knew was wrong (or at least not the whole story), it's good that these problems are actually coming out so we can correct them.

3 days ago
1/9/02
Posts: 49652

Ironically, you will get the most resistance to this change from the non-conservatives

 

the left has built their entire value system on a alot of this stuff coming our of the university system since the 60's

3 days ago
1/1/01
Posts: 10796
gregbrady - 

Ironically, you will get the most resistance to this change from the non-conservatives

 

the left has built their entire value system on a alot of this stuff coming our of the university system since the 60's


Primarily yes. I think though it might depend on the specific theory in mind.

Having said that - one of the worst things in the world of psychology, sociology, anthropology - is the dearth of conservatives taking up those fields.

Personally - I fall on the left, but I'm very much aware that conservative or "right wing" voices are also important. Particularly in fields that require checks and balances. One of the most effective ways at correcting bad ideas coming from a left-wing source would be vigorous opposition from conservative thinkers.

One of the things that bothers me the most about some of the people I share *some* political ideas with, is their prarticular form of anti-science behavior, Not only will they resist these discoveries as you mention, they'll also dismiss other discoveries purely for being inconvenient. Their general opposition to the idea that biology has *anything* to do with the brain for example. This should be almost trivial and uncontroversial - but they think that if they admit the brain evolved that it would justify awful deeds; not realizing that any such implication is squashed by recognizing the naturalistic fallacy.

Edited: 3 days ago
1/1/01
Posts: 18622

The replication crisis goes beyond pyschology.  Biology and it's flavours (microbiology etc) are also hit by this but to a lesser extent.  Scientific method needs to be respected always, otherwise don't call yourself a science.  I first heard about this when a paper was published (maybe even a phd thesis,  I forget) where the author(s) tried to repro 100 famous recent psych experiments.  Less than 50 were reproducable.

3 days ago
6/29/09
Posts: 11088
It isnt really surprising to me. Psychology and psychiatry are pseudo-science. Soft science. If youve ever had the chance to examine treatment plans, office progress note, med management visits, et al....youre going to be in for a shocker particularly if you are in a hard science or a M.D./D.O.

I'll give you an example....you go to your ortho for a shoulder/knee/hip issue. Youre going to see a similar process or standard of care in the diagnosis and treatment of your issue. Initial office visit, exam, maybe labs if appropriate, scans, follow up, initial diagnosis made, probably physical therapy, conservative tx like joint injection, follow up, reevaluate, possible surgery.....blah blah blah.

Now....you go to a therapist, psychologist, lcsw, god forbid a qmhp....you get a talking to. The patient (who a great number lie or cant articulate shit) tells the provider what they feel. You get more talking to. They crack the DSM V and say hmmmm. It could be this.

Maybe....maybe...if its a decent size practice youll get a referral to the psychiatrist (a MD) or the nurse prac or PA for med management. Youll likely see this person every 3-6 months for refills. Youll get in and out in record time as theyve got a ton more to see.

In the mean time, its office visits with you therapist and shitty progress notes that say patient feeling better or Patient says stressed over blah blah. No diagnostics, no labs, rately drug screens, rarely pill counts, rarely a check of the patients prescription monitoring report, very rarely a referral to another medical provider.

On another note.....THIS IS WHY RED FLAG LAWS ARE BULLSHIT!
Edited: 3 days ago
1/9/02
Posts: 49653
robert bentley -
gregbrady - 

Ironically, you will get the most resistance to this change from the non-conservatives

 

the left has built their entire value system on a alot of this stuff coming our of the university system since the 60's

 

Primarily yes. I think though it might depend on the specific theory in mind.

Having said that - one of the worst things in the world of psychology, sociology, anthropology - is the dearth of conservatives taking up those fields.

Personally - I fall on the left, but I'm very much aware that conservative or "right wing" voices are also important. Particularly in fields that require checks and balances. One of the most effective ways at correcting bad ideas coming from a left-wing source would be vigorous opposition from conservative thinkers.

One of the things that bothers me the most about some of the people I share *some* political ideas with, is their prarticular form of anti-science behavior, Not only will they resist these discoveries as you mention, they'll also dismiss other discoveries purely for being inconvenient. Their general opposition to the idea that biology has *anything* to do with the brain for example. This should be almost trivial and uncontroversial - but they think that if they admit the brain evolved that it would justify awful deeds; not realizing that any such implication is squashed by recognizing the naturalistic fallacy.

I would argue thats essential in any of the "sciences" that can be influenced with bias

 

the political leanings of the physics department are irrelevant

 

the rise of the anti-science part of the left is scary. the equivalent on the right is shrinking (evangelicals)

3 days ago
3/16/11
Posts: 6430
deepu -

The replication crisis goes beyond pyschology.  Biology and it's flavours (microbiology etc) are also hit by this but to a lesser extent.  Scientific method needs to be respected always, otherwise don't call yourself a science.  I first heard about this when a paper was published (maybe even a phd thesis,  I forget) where the author(s) tried to repro 100 famous recent psych experiments.  Less than 50 were reproducable.

Danny Kahneman and Sam Harris talked about this at length. Hes a very famous psychologist who wrote a very influential book that relied heavily on a few very memorable but ultimately insufficiently scaled experiments. He took the criticism to heart and acknowledged this, which I think is the first step in solving the crises. At its heart, it's a crisis of wishful thinking and confirmation bias, but if people can be skeptical of their own work and sources, then the bar will inevitably rise. 

Here's an article that explains it better than I can:

https://retractionwatch.com/2017/02/20/placed-much-faith-underpowered-studies-nobel-prize-winner-admits-mistakes/

3 days ago
4/27/05
Posts: 1828

In. Def interesting stuff. The social sciences are not infallible as culture makes them out to be. 

Edited: 3 days ago
1/1/01
Posts: 18623
Pilgor the goat - 
deepu -

The replication crisis goes beyond pyschology.  Biology and it's flavours (microbiology etc) are also hit by this but to a lesser extent.  Scientific method needs to be respected always, otherwise don't call yourself a science.  I first heard about this when a paper was published (maybe even a phd thesis,  I forget) where the author(s) tried to repro 100 famous recent psych experiments.  Less than 50 were reproducable.

Danny Kahneman and Sam Harris talked about this at length. Hes a very famous psychologist who wrote a very influential book that relied heavily on a few very memorable but ultimately insufficiently scaled experiments. He took the criticism to heart and acknowledged this, which I think is the first step in solving the crises. At its heart, it's a crisis of wishful thinking and confirmation bias, but if people can be skeptical of their own work and sources, then the bar will inevitably rise. 

Here's an article that explains it better than I can:

https://retractionwatch.com/2017/02/20/placed-much-faith-underpowered-studies-nobel-prize-winner-admits-mistakes/

 

From your link:

 

https://replicationindex.com/2016/01/31/a-revised-introduction-to-the-r-index/

 

Abstract

Researchers are competing for positions, grant money, and status. In this competition, researchers can gain an unfair advantage by using questionable research practices (QRPs) that inflate effect sizes and increase the chances of obtaining stunning and statistically significant results. To ensure fair competition that benefits the greater good, it is necessary to detect and discourage the use of QRPs. To this aim, I introduce a doping test for science; the replicability index (R-index). The R-Index is a quantitative measure of research integrity that can be used to evaluate the statistical replicability of a set of studies (e.g., journals, individual researchers’ publications).  A comparison of the R-Index for the Journal of Abnormal and Social Psychology in 1960 and the Attitudes and Social Cognition section of the Journal of Social and Personality Psychology in 2011 shows an increase in the use of QRPs. Like doping tests in sports, the availability of a scientific doping test should deter researchers from engaging in practices that advance their careers at the expense of everybody else. Demonstrating replicability should become an important criterion of research excellence that can be used by funding agencies and other stakeholders to allocate resources to research that advances science.
Keywords: Power, Publication Bias, Significance, Credibility, Sample Size, Questionable Research Methods, Replicability, Statistical Methods

This is pretty cool, R-Index for replicability index.  I wonder how widely it will be adopted.

On another note, there was another famous psychology paper that was debunked but it was used as dogma in thousands of papers (it was an old paper).  Think of the amount of garbage generated over decades that have been based on garbage.

3 days ago
1/1/01
Posts: 10797
gregbrady - 
robert bentley -
gregbrady - 

Ironically, you will get the most resistance to this change from the non-conservatives

 

the left has built their entire value system on a alot of this stuff coming our of the university system since the 60's

 

Primarily yes. I think though it might depend on the specific theory in mind.

Having said that - one of the worst things in the world of psychology, sociology, anthropology - is the dearth of conservatives taking up those fields.

Personally - I fall on the left, but I'm very much aware that conservative or "right wing" voices are also important. Particularly in fields that require checks and balances. One of the most effective ways at correcting bad ideas coming from a left-wing source would be vigorous opposition from conservative thinkers.

One of the things that bothers me the most about some of the people I share *some* political ideas with, is their prarticular form of anti-science behavior, Not only will they resist these discoveries as you mention, they'll also dismiss other discoveries purely for being inconvenient. Their general opposition to the idea that biology has *anything* to do with the brain for example. This should be almost trivial and uncontroversial - but they think that if they admit the brain evolved that it would justify awful deeds; not realizing that any such implication is squashed by recognizing the naturalistic fallacy.

I would argue thats essential in any of the "sciences" that can be influenced with bias

 

the political leanings of the physics department are irrelevant

 

the rise of the anti-science part of the left is scary. the equivalent on the right is shrinking (evangelicals)


Definitely important for all sciences.

But with respect to proportions - the humanities have by far the worst ideological balance. That is - in reas like engineering, chemistry, physics - the political leanings are mixed-ish (with a bit more left leaning people, but plenty of conservatives). When you get into psychology though - it's something crazy like 95% left leaning.

So while important as a principle for everywhere - the variation is catastrophic in psychology and sociology.

3 days ago
1/1/01
Posts: 10798
deepu - 

The replication crisis goes beyond pyschology.  Biology and it's flavours (microbiology etc) are also hit by this but to a lesser extent.  Scientific method needs to be respected always, otherwise don't call yourself a science.  I first heard about this when a paper was published (maybe even a phd thesis,  I forget) where the author(s) tried to repro 100 famous recent psych experiments.  Less than 50 were reproducable.


Agree with all of this.

3 days ago
1/1/01
Posts: 47795

in

Edited: 3 days ago
3/16/11
Posts: 6432
deepu -
Pilgor the goat - 
deepu -

The replication crisis goes beyond pyschology.  Biology and it's flavours (microbiology etc) are also hit by this but to a lesser extent.  Scientific method needs to be respected always, otherwise don't call yourself a science.  I first heard about this when a paper was published (maybe even a phd thesis,  I forget) where the author(s) tried to repro 100 famous recent psych experiments.  Less than 50 were reproducable.

Danny Kahneman and Sam Harris talked about this at length. Hes a very famous psychologist who wrote a very influential book that relied heavily on a few very memorable but ultimately insufficiently scaled experiments. He took the criticism to heart and acknowledged this, which I think is the first step in solving the crises. At its heart, it's a crisis of wishful thinking and confirmation bias, but if people can be skeptical of their own work and sources, then the bar will inevitably rise. 

Here's an article that explains it better than I can:

https://retractionwatch.com/2017/02/20/placed-much-faith-underpowered-studies-nobel-prize-winner-admits-mistakes/

 

From your link:

 

https://replicationindex.com/2016/01/31/a-revised-introduction-to-the-r-index/

 

Abstract

Researchers are competing for positions, grant money, and status. In this competition, researchers can gain an unfair advantage by using questionable research practices (QRPs) that inflate effect sizes and increase the chances of obtaining stunning and statistically significant results. To ensure fair competition that benefits the greater good, it is necessary to detect and discourage the use of QRPs. To this aim, I introduce a doping test for science; the replicability index (R-index). The R-Index is a quantitative measure of research integrity that can be used to evaluate the statistical replicability of a set of studies (e.g., journals, individual researchers’ publications).  A comparison of the R-Index for the Journal of Abnormal and Social Psychology in 1960 and the Attitudes and Social Cognition section of the Journal of Social and Personality Psychology in 2011 shows an increase in the use of QRPs. Like doping tests in sports, the availability of a scientific doping test should deter researchers from engaging in practices that advance their careers at the expense of everybody else. Demonstrating replicability should become an important criterion of research excellence that can be used by funding agencies and other stakeholders to allocate resources to research that advances science.
Keywords: Power, Publication Bias, Significance, Credibility, Sample Size, Questionable Research Methods, Replicability, Statistical Methods

This is pretty cool, R-Index for replicability index.  I wonder how widely it will be adopted.

On another note, there was another famous psychology paper that was debunked but it was used as dogma in thousands of papers (it was an old paper).  Think of the amount of garbage generated over decades that have been based on garbage.

Yup, it's like building a house on a bad foundation. The house can never be stronger than what its built upon, and it's the same problem we see with post-modernism taking hold in the humanities with a strangely inverse outcome. 

I've done some work in mechanical systems engineering, and post-modernism (as a fundamentally skeptical idea) is a typical feedback correction circuit. Very useful stopping and correcting emergent errors in the logic, but the humanities have tried to bolster it as the fundamental nature of reality itself. Oddly, it's the opposite of what's going on here, where instead of fundamental skepticism of all truth claims, there is an overconfidence. I dont think it's a coincidence though, I think its what's actually driving the schism between humanities and STEM fields and ironically, very closely mirrors the metaphysical logic of the modern political divide. Both sides need to get their shit together, apparently. 

 

Edit: So that editing glitch is never going to be fixed? I've seen other people post the same gibberish, so it's just not me. If you dont press space after moving the placing cursor and deleting characters, it just jumbles shit up into one word. I'm sick of writing everything 10 times and then see that I missed a couple later and my post looks retarded. Jesus christ. 

3 days ago
8/26/05
Posts: 13605
^ It's because, when push comes to shove, Kahneman actually is a scientist.
3 days ago
12/17/06
Posts: 77796

I worked with a Doctor of some psychological science or some such. Must be eighteen years ago now. He was working part time in a boiler room sales office cold calling on chimney cleanings as part of some research he was doing. Something about sales psychology. One day we're on a fifteen minute smoke break and he's talking bout a sale he lost and blurts out, "all of it's bullshit anyway" in regards to his research.

Made me laugh hysterically for some reason.