Null Hypothesis Testing... Thumbs Down

 A little introduction to the history and basics of significance testing in "squishy" psychology:

https://www.sciencenews.org/article/statistical-significance-p-value-null-hypothesis-origins

"Routine bad science using routine bad methods…"

 Gelman tells us "Don’t hate the player, hate the game" (bad psychological science):

https://statmodeling.stat.columbia.edu/2021/06/16/wow-just-wow-if-you-think-psychological-science-as-bad-in-the-2010-2015-era-you-cant-imagine-how-bad-it-was-back-in-1999/

Lies, Damned Lies, and....

(via Wikipedia)

                                                    
One of the late chapters in David Spiegelhalter’s fine volume, “The Art of Statistics,” focuses on the problems and reproducibility crisis in psychology research.

This sentence gave me a bit of a startle: “In a 2012 survey of 2,155 US academic psychologists, just 2% admitted to falsifying data.” The part that gave me a gasp was the phrase, “just 2%” as if that was a small figure. I’m not surprised that 2% have falsified data; I’m surprised that that many would admit to it! Indeed, I feel sure (though am only guessing) that the majority who have done it would NOT admit to it, and that the true figure is therefore probably at least double the 2% given — that would be at least 4% who haven’t just made mistakes, or fudged a little, or spun their conclusions, but outright falsified data! 

I don’t know how many total academic psychologists there are, but depending where they are and how much they publish, 2 - 4+% represents a pretty serious problem in my mind.  And while psychology is probably especially vulnerable to such falsification this doesn’t even address how much may additionally go on in biological, medical, and physical science fields. Spiegelhalter goes on to note that 94% of those in the study admitted committing at least one of 10 "questionable research practices" looked at.  Of course there are many reasons for such a state of affairs, but ultimately none very defensible. The award-winning site "Retraction Watch" has been tracking, for years now, published scientific papers that are retracted due to various issues, including fraud -- and they never seem to run out of material! :(( (definitely a site worth following and supporting if you're not already).

I was a psychology major myself 45 years ago and complained, to deaf ears, about the sloppiness of the field, but sloppiness and fraud are almost separate issues. Still, glad to see it all attracting more attention these days.

The study Spiegelhalter cites is here (with a lot more details):

https://www.cmu.edu/dietrich/sds/docs/loewenstein/MeasPrevalQuestTruthTelling.pdf

Replicate THIS

If you missed it, last week’s NPR’s “On The Media” show included a nice segment (number 2 out of 4) on the ongoing replication problems in psychology:

http://www.wnyc.org/story/psychologys-replication-crisis

"the manipulation of noise" (quantitatively speaking)

Andrew Gelman, interesting as always, on psychology's replication crisis:
http://andrewgelman.com/2016/10/05/michael-lacour-vs-john-bargh-and-amy-cuddy/

Specifically mentioning work by Bargh, Cuddy, Bem, Lacour, and others, but applicable to a wider throng.

Let's Get Real! (I'm a tad peeved)

"...there are real issues at stake here, and there’s nothing wrong—nothing wrong at all—with people arguing about the details while at the same time being aware of the big picture."   -- Andrew Gelman

Another great follow-up from Andrew Gelman today on the whole statistics debate I referenced yesterday:
http://tinyurl.com/hlqp5b4

Do read Gelman's more formal presentation, but here, my own further informal comment:

Some of those defensive about the "crisis in replication" in psychology are arguing that the so-called "replications" (that failed) weren't actually precise replications at all, but merely rough and poor approximations... well, DUHHHH, of course there are NO TRUE REPLICATIONS in psychology... get real!: when dealing with human behavior, if you use different samples at different times on different days in different places, under varying conditions, it is not going to be an exact replication (EVEN IF you were otherwise able to duplicate the same methodological steps).  And as others have noted, IF your finding requires absolute precise duplication in order to replicate, than you are NOT doing science, which seeks to find results that generalize more widely.

Moreover, Gilbert et.al. (and other psychologists) HAVE NEVER and WILL NEVER do an adequately controlled, well-defined, unbiased, highly-generalizable experiment in social psychology using a truly random sample... it IS NOT possible!... uncontrolled independent variables are far too many, complex, poorly-defined, and likely synergistic, to permit understanding of, or definitive conclusions about, any dependent variable(s) under study, AND no human sample is ever really random (...but, that won 't prevent them from publishing such studies). Why can't we just admit out-loud to such imperfections...

Nassim Taleb is fond of dismissing such studies as BS and moving on... I'm not that harsh -- social studies can lack rigor, yet still contain glints of value and interest for further exploration, SO LONG as their results aren't presumed reliable, valid, and widely-applicable from the get-go. GOOD (and meaningful) studies in psychology are very, very, very difficult to do, and so too, good replications.

Lies, Damned Lies, and...

yeah, you know what's coming....
If statistics talk leaves you yawning, followed by a major face-plant on the desktop, no reason to continue reading....

For a few years now controversy over the use of stats in social sciences in particular (but also in biomedicine and other research areas) has been brewing. It seems to have erupted Mt. Vesuvius fashion this week.

I was myself a psychology major long ago, but never held much credence in the methodologies or empiricism that was practiced. Indeed, I'm amazed it's taken this long for the controversy (in large part over two pillars: significance-testing and replication) to bubble over; there's so much weak, dodgy psyche research out there going back a century.

Below are a sampling of the many recent posts dealing with a new set of "six principles" (in regard to p-values) the American Statistical Association just recommended:

1)  First, this NY Times piece gives some of the background to the whole debate:
http://tinyurl.com/heyzwyq

2) "Retraction Watch" interviews the Executive Director of the American Statistical Association, Ron Wasserstein, about the 'six new principles' for the use of p-values:
http://retractionwatch.com/2016/03/07/were-using-a-common-statistical-test-all-wrong-statisticians-want-to-fix-that/  (a good read)

3)  Nature reports:
http://tinyurl.com/zcl76b2

4)  from another British journal:
https://thepsychologist.bps.org.uk/replication-glass-half-full-half-empty-or-irrelevant

5)  Deborah Mayo adds some nuance to the discussion here:
http://errorstatistics.com/2016/03/07/dont-throw-out-the-error-control-baby-with-the-bad-statistics-bathwater/

6)  Andrew Gelman adds comments, with more emphasis on 'null hypothesis testing' and "the garden of forking paths":
http://andrewgelman.com/2016/03/07/29212/

7)  and from FiveThirtyEight blog, perhaps the most entertaining read:
http://fivethirtyeight.com/features/statisticians-found-one-thing-they-can-agree-on-its-time-to-stop-misusing-p-values/

8)  Some older work of Nassim Taleb, also pertinent to the discussion here:
http://www.fooledbyrandomness.com/FatTails.html

These are just a smattering of what is out there right now on this matter. Much of what transpires in psychology (and particularly social psychology), I believe, would be better characterized as "social studies" and NOT "social science" -- and social studies, by the way, are very much worth doing (despite the variables, complexity, and ambiguities involved)... they just ought not be confused with good "science," especially in regards to the generalizability or extrapolation of the findings.

When I took statistics in grad school the professor warned us on the first day of class that he hated teaching these "statistics for social scientists" courses (but was required to) because there was no way to satisfactorily teach students what they needed to know for doing research, in a semester or two. And no matter what grade we got in the class our knowledge and understanding of statistics would be inadequate; we would be turned loose on the world thinking we knew more than we did.
I've often seen mathematicians write that no one really learns or understands calculus when they first take it (again no matter what grade you get); it is only after further years of mathematics study that a deeper understanding of the calculus finally emerges. So too statistics.

Anyway, these statistical debates entail hugely important issues that we need to attend to; issues going well beyond journal research articles. Currently, statistics and probability classes are increasingly a part of secondary education, and we need some assurance of getting it right... lest we send off a whole 'nother generation with wrong ways to think about and apply statistics.
This controversy will continue for a long awhile... because it is tangled and long overdue (and... it's putting a lot of practitioners on the defensive).

Parsing the Failure To Replicate

This study in Science, involving attempts to replicate 100 prior research studies, has been making the rounds (I've linked to the story before) for finding only about 1/3 of psychology studies statistically replicable; in some places setting off a bit of a firestorm.
Virginia Mayo takes an extended, more nuanced look at the issue here:

http://tinyurl.com/nkqd7bd

Worthwhile reading if you've been following the controversy (one dispute is whether the statistical methods employed are inherently weak/poor, even in need of being discarded, or are the methods fine, but simply abused/misused often in their application).