"Crap" in Science....

 Wow!... this isn't focused on math per se, but no doubt there is math included in the subjects referenced in this piece (H/T to Ivan Oransky):

https://www.science.org/content/blog-post/how-much-published-crap-will-we-put

Spurious News/Correlations

 H/T to Nathan Yau at "Flowing Data" for this one:

https://davidepstein.bulletin.com/everything-in-your-fridge-causes-and-prevents-cancers

Statistical Significance and "Oomph"

 What models and data analysis can miss:

https://mindmatters.ai/2021/12/the-cult-of-statistical-significance-and-the-neglect-of-oomph/

Flawed Covid Research et. al.

 203 retractions... and counting:

https://retractionwatch.com/2021/12/18/weekend-reads-fraudulent-and-faulty-research-a-275k-settlement-but-resignation-for-a-professor-covid-19-ivermectin-and-beyond/

BBC on Covid Immunity...

 Natural immunity vs. vaccination:

https://www.bbc.co.uk/sounds/play/p0b98hzl

John Cook on "Fraud, Sloppiness, and Statistics"

 A LOT of scientific papers later appear to be wrong. John Cook attempts to explain the likely reasons; longer than his typical posts, but still succinct:

https://www.johndcook.com/blog/2021/12/11/fraud-sloppiness-and-statistics/

Have the Mice Had Their Coffee Yet... ;)

 Mouse studies... time-of-day may matter!:

https://www.scientificamerican.com/article/many-mouse-studies-happen-at-the-wrong-time-of-day/

To Retract Or Not To Retract (...is that the question)

 An example of Journal 'expression of concern' vs. retraction, via Retraction Watch:

https://retractionwatch.com/2021/08/23/readers-puzzle-over-marketing-journals-failures-to-retract/

Null Hypothesis Testing... Thumbs Down

 A little introduction to the history and basics of significance testing in "squishy" psychology:

https://www.sciencenews.org/article/statistical-significance-p-value-null-hypothesis-origins

Pubpeer to the Rescue...

 The power of post-publication peer review:

https://statmodeling.stat.columbia.edu/2021/06/25/this-awesome-pubpeer-thread-is-about-80-times-better-than-the-original-paper/

Can Research Integrity Ever Prevail in a Marketplace...?

 "Research integrity"... perhaps once assumed, is now threatened and very much in question given multiple pressures that abound:

https://www.tandfonline.com/doi/full/10.1080/08989621.2021.1937603

(H/T to Ivan Oransky)

Statistical Practices — the Bad Driving Out the Good

 H/T to Mike Lawler for pointing out this essay (and “belly-aching”) from Darren Dahly on common statistical (mal)practice, particularly in medicine:

https://statsepi.substack.com/p/how-bad-statistical-practices-drive

Too many great sentences in this (about the author's slog against research "bullshit") I’d love to quote, but will simply give you the opening lines:

“I am interested in research integrity and reproducibility. I believe that a lack of statistical expertise throughout the sciences is a substantial driver of problems in these areas (poor data practices being another). I feel especially strongly about this thesis as it applies to medical research.”    — Darren Dahly

"Dear p-values...."

 Taking the p-values pledge:

https://medianwatch.netlify.app/post/pvalue_pledge/

"Based on this latest experience and my exhaustion with dealing with p-values, I am making a pledge:

There will be no p-values in any paper that I co-author in the next 12 months."  -- Adrian Barnett

The Slippery Fish of Social Science...

“…when we’re doing social and behavioral science, we’re not looking for a needle in a haystack; rather, we’re trying to catch a slippery fish that keeps moving.  All this is even harder in political science, economics, or sociology. An essential aspect of social science is that it understands people not in isolation but within groups. Thus, if psychology ultimately requires a different model for each person (or a model that accounts for differences between people), the social sciences require a different model for each configuration of people (or a model that accounts for dependence of outcomes on the configuration).”

I enjoyed this statistical (or almost meta-statistical) piece from Andrew Gelman yesterday on  social/behavioral science research, where the same concerns seem to re-occur again and again over time:

https://statmodeling.stat.columbia.edu/2020/08/16/lee-cronbach-1975-beyond-the-two-disciplines-of-scientific-psychology/

"The system is broken"... and an "Amen" to Andrew Gelman

Gelman reflecting on bad research (...which has been known to exist ;)

https://statmodeling.stat.columbia.edu/2020/07/28/bad-research-scientism-the-importance-of-description-and-the-challenge-of-negativity/

Lies, Damned Lies, and....

(via Wikipedia)

                                                    
One of the late chapters in David Spiegelhalter’s fine volume, “The Art of Statistics,” focuses on the problems and reproducibility crisis in psychology research.

This sentence gave me a bit of a startle: “In a 2012 survey of 2,155 US academic psychologists, just 2% admitted to falsifying data.” The part that gave me a gasp was the phrase, “just 2%” as if that was a small figure. I’m not surprised that 2% have falsified data; I’m surprised that that many would admit to it! Indeed, I feel sure (though am only guessing) that the majority who have done it would NOT admit to it, and that the true figure is therefore probably at least double the 2% given — that would be at least 4% who haven’t just made mistakes, or fudged a little, or spun their conclusions, but outright falsified data! 

I don’t know how many total academic psychologists there are, but depending where they are and how much they publish, 2 - 4+% represents a pretty serious problem in my mind.  And while psychology is probably especially vulnerable to such falsification this doesn’t even address how much may additionally go on in biological, medical, and physical science fields. Spiegelhalter goes on to note that 94% of those in the study admitted committing at least one of 10 "questionable research practices" looked at.  Of course there are many reasons for such a state of affairs, but ultimately none very defensible. The award-winning site "Retraction Watch" has been tracking, for years now, published scientific papers that are retracted due to various issues, including fraud -- and they never seem to run out of material! :(( (definitely a site worth following and supporting if you're not already).

I was a psychology major myself 45 years ago and complained, to deaf ears, about the sloppiness of the field, but sloppiness and fraud are almost separate issues. Still, glad to see it all attracting more attention these days.

The study Spiegelhalter cites is here (with a lot more details):

https://www.cmu.edu/dietrich/sds/docs/loewenstein/MeasPrevalQuestTruthTelling.pdf

Of Drugs and Canaries

Saying “There’s a real problem in the way that clinical trials report their results,” reform-minded Ben Goldacre argues that “Statins are the canary in the cage for problems in modern medicine.”

This isn’t strictly math, but is an important read on how clinical research gets “fiddled” with:

https://cosmosmagazine.com/society/bad-pharma-and-the-canary-in-the-coalmine-for-problems-in-modern-medicine

From the piece:

"[clinical] Outcome priorities are changed; negative results are omitted; trials are foreshortened or extended to better massage the data. In the book [Big Pharma], Goldacre terms these tactics 'a quiet and diffuse scandal.'"

…and to think there was a day, not so long ago, when Big Pharma was trying to convince folks that statins (“the most commonly prescribed medication in the developed world”) ought be added routinely to municipal water supplies. 

Putting Lipstick on a Pig

"Any sufficiently crappy research is indistinguishable from fraud"... that's the gist of a recent post from Andrew Gelman taking off on Arthur C. Clarke's 3rd Law, in the realm once again, of research papers displaying poor statistical analysis (be it incompetency or deliberate deception): 

http://andrewgelman.com/2016/06/20/clarkes-law-of-research/

The post gets quite a bit of commentary in follow-up (mostly backing Gelman up): 

And in a funny bit of timing, I came to Gelman's post very shortly after seeing a political cartoon on the Web showing Paul Ryan putting lipstick on a pig drawn as Donald Trump. Just struck me as an odd juxtaposition... how often politicians put lipstick on pigs, and, so too, researchers.


ADDENDUM:  just this morning "Retraction Watch" tweets out this abstract from a John Ioannidis group indicating that the majority of randomly-controlled studies evaluating "efficacy and safety" are sponsored by industry, and, lo-and-behold, 95+% of published results favor the sponsor:

http://www.jclinepi.com/article/S0895-4356(15)00058-X/abstract

Good Science?

via WikiMediaCommons

Early in my career I did ~6 years of animal research... but couldn't stick with it as I saw too much poor/sloppy science being done. The variables are so many, so complex, so ill-defined and easily overlooked, as to make it almost surprising how often progress is actually made (or at least perceived).
Here's just one more problem:
https://www.statnews.com/2016/04/19/lab-mice-temperature/

IF temperature affects the results of experiments with lab mice, what about sounds, sights, lighting, diet, human touch, air circulation, altitude, and on and on and on... of course no one truly knows all the effects on physiology and brain chemistry of sensory inputs (anymore than anyone knows what weather events may be affected halfway around the globe eight months after a butterfly flaps its wings). Cause-and-effect, when it comes to living beings, is nothing if not chaotic.

That's not to be too harsh about such work, but simply to prompt a skeptical stance, especially toward initial, and unverified-or-unreplicated results (let alone the hype of headlines).

I've previously voiced dismay here with those who proclaim themselves "skeptics," yet who largely grant a free pass to weak science and methods published routinely in major journals (luckily, now, decades since I experienced my cynicism, such skepticism is creeping into more mainstream outlets).
There is pseudoscience, speculative science, and real or good science... and the lines blur far more than admitted. Even theoretical physics, revered in my youth, today stands accused from many quarters, of bordering on metaphysics or philosophy, and not true empiricism... I'm not judging it one way or the other, except to say that even such an accusation, from bright people, is telling.

No one said good science should be easy... or common... indeed, it is difficult and rare. Mediocre science is the norm. And wrong-headed science is not uncommon... but is correctable. The (scary) anti-science attitudes/backlash of so many Americans today is a direct result of being sold a naive bill-of-goods and never understanding the true tentative, uncertain nature of science, its strengths and too-often-unacknowledged weaknesses. Still, it remains the best, by far, we've got... and its cornerstone, by the way, is mathematics.

On Twitter I've often used the below graphic (sorry I don't know its origination), but with the suggestion that you can replace the word "success" with the word "science" and it remains true:

Bottom line, good science is incredibly complex at a time when many increasingly gravitate toward simple answers (ala the absurd rise of Donald Trump). Science, misunderstood and misused, can destroy us... yet it is also probably the ONLY thing that can save us... from ourselves.

Let's Get Real! (I'm a tad peeved)

"...there are real issues at stake here, and there’s nothing wrong—nothing wrong at all—with people arguing about the details while at the same time being aware of the big picture."   -- Andrew Gelman

Another great follow-up from Andrew Gelman today on the whole statistics debate I referenced yesterday:
http://tinyurl.com/hlqp5b4

Do read Gelman's more formal presentation, but here, my own further informal comment:

Some of those defensive about the "crisis in replication" in psychology are arguing that the so-called "replications" (that failed) weren't actually precise replications at all, but merely rough and poor approximations... well, DUHHHH, of course there are NO TRUE REPLICATIONS in psychology... get real!: when dealing with human behavior, if you use different samples at different times on different days in different places, under varying conditions, it is not going to be an exact replication (EVEN IF you were otherwise able to duplicate the same methodological steps).  And as others have noted, IF your finding requires absolute precise duplication in order to replicate, than you are NOT doing science, which seeks to find results that generalize more widely.

Moreover, Gilbert et.al. (and other psychologists) HAVE NEVER and WILL NEVER do an adequately controlled, well-defined, unbiased, highly-generalizable experiment in social psychology using a truly random sample... it IS NOT possible!... uncontrolled independent variables are far too many, complex, poorly-defined, and likely synergistic, to permit understanding of, or definitive conclusions about, any dependent variable(s) under study, AND no human sample is ever really random (...but, that won 't prevent them from publishing such studies). Why can't we just admit out-loud to such imperfections...

Nassim Taleb is fond of dismissing such studies as BS and moving on... I'm not that harsh -- social studies can lack rigor, yet still contain glints of value and interest for further exploration, SO LONG as their results aren't presumed reliable, valid, and widely-applicable from the get-go. GOOD (and meaningful) studies in psychology are very, very, very difficult to do, and so too, good replications.