Scholarly Pursuits Gone Awry

 A fun read from Andrew Gelman on questionable scholarly work and "the tipping point":

https://statmodeling.stat.columbia.edu/2021/06/04/when-does-a-misunderstanding-reach-the-point-where-it-is-recognized-to-be-flat-out-ridiculous/

The Slippery Fish of Social Science...

“…when we’re doing social and behavioral science, we’re not looking for a needle in a haystack; rather, we’re trying to catch a slippery fish that keeps moving.  All this is even harder in political science, economics, or sociology. An essential aspect of social science is that it understands people not in isolation but within groups. Thus, if psychology ultimately requires a different model for each person (or a model that accounts for differences between people), the social sciences require a different model for each configuration of people (or a model that accounts for dependence of outcomes on the configuration).”

I enjoyed this statistical (or almost meta-statistical) piece from Andrew Gelman yesterday on  social/behavioral science research, where the same concerns seem to re-occur again and again over time:

https://statmodeling.stat.columbia.edu/2020/08/16/lee-cronbach-1975-beyond-the-two-disciplines-of-scientific-psychology/

"The system is broken"... and an "Amen" to Andrew Gelman

Gelman reflecting on bad research (...which has been known to exist ;)

https://statmodeling.stat.columbia.edu/2020/07/28/bad-research-scientism-the-importance-of-description-and-the-challenge-of-negativity/

Of Brittle and Broad Theories, and Science...

“….I responded with my view that all these theories are either vague enough to be adaptable to any data or precise enough to be evidently false with no data collection needed. This was what Lakatos noted: any theory is either so brittle that it can be destroyed by collecting enough data, or flexible enough to fit anything. This does not mean we can’t do science, it just means we have to move beyond naive falsificationism.”

                           — Andrew Gelman at end of a post on his blog a short while back

"replication is central to science"

For a Sunday reflection, this from Andrew Gelman in "The Best Writing on Mathematics 2016":

"To resolve the replication crisis in science, we may need to consider each individual study in the context of an implicit meta-analysis. And we need to move away from a simplistic, deterministic model of science with its paradigm of testing and sharp decisions: accept/reject the null hypothesis and do/don't publish the paper. To say that a claim should be replicated is not to criticize the original study; rather, a replication is central to science, and statistical methods should recognize this. We should not get stuck in the mode in which a 'data set' is analyzed in isolation, without consideration of other studies or relevant scientific knowledge. We must embrace variation and accept uncertainty."

Putting Lipstick on a Pig

"Any sufficiently crappy research is indistinguishable from fraud"... that's the gist of a recent post from Andrew Gelman taking off on Arthur C. Clarke's 3rd Law, in the realm once again, of research papers displaying poor statistical analysis (be it incompetency or deliberate deception): 

http://andrewgelman.com/2016/06/20/clarkes-law-of-research/

The post gets quite a bit of commentary in follow-up (mostly backing Gelman up): 

And in a funny bit of timing, I came to Gelman's post very shortly after seeing a political cartoon on the Web showing Paul Ryan putting lipstick on a pig drawn as Donald Trump. Just struck me as an odd juxtaposition... how often politicians put lipstick on pigs, and, so too, researchers.


ADDENDUM:  just this morning "Retraction Watch" tweets out this abstract from a John Ioannidis group indicating that the majority of randomly-controlled studies evaluating "efficacy and safety" are sponsored by industry, and, lo-and-behold, 95+% of published results favor the sponsor:

http://www.jclinepi.com/article/S0895-4356(15)00058-X/abstract

The Improbability of Knowing Probability

via Gerald G/WikimediaCommons

On Oct. 21 Andrew Gelman asked on his blog, "What's the probability that Daniel Murphy hits a home run tonight?" (in a record-setting 6th straight playoff game):
http://andrewgelman.com/2015/10/21/whats-the-probability-that-daniel-murphy-hits-a-home-run-tonight/

He posted the answer as 20% and then, at the coaxing of some commenters, lowered it to 15%.
Then... later that evening, he raised the probability to 1... because of course Murphy (of the New York Mets) did just that, hit a home run in the 8th inning (playing against the Chicago Cubs, surely a major factor ;-)

And so, in a matter of hours the "probability" of something went from 20% to 15% to 100%... a nice demonstration of why, given human complexity, "probability" is often a near-meaningless concept when it comes to individual behavior and events.

Measurement

WikimediaCommons

Good, succinct post from Andrew Gelman today on statistics and "measurement," noting at start that "Statistics does not require randomness. The three essential elements of statistics are measurement, comparison, and variation." And of those, Gelman believes that "measurement" is the most slighted or "neglected" element:
http://andrewgelman.com/2015/04/28/whats-important-thing-statistics-thats-not-textbooks/

He says when it comes to measurement in research or statistics textbooks "...there’s silence, just an implicit assumption that the measurement is what it is, that it’s valid and that it’s as reliable as it needs to be."
Of course we witness this all across research... from Government economic statistics that get routinely "revised" or recomputed on a near-monthly basis, to epidemiology where statistics change sometimes with each new study sample, to even high level physics where new findings too often have to be altered or abandoned when measurements are re-taken or newly-analyzed. And perhaps psychology takes the greatest brunt of criticism as Gelman writes, "A common thread in these [psychological] studies is sloppy, noisy, biased measurement." Indeed, 'behavior' is one of the most difficult things to measure and generalize about empirically. ...But, no one ever said it was easy.

I don't think Gelman even goes far enough here. Part of the intrinsic problem of measurement is the necessity to recognize and precisely define all pertinent variables that are to be measured... in most fields, a close-to-impossible task; so measurement is relegated to a rough (and sometimes VERY rough) approximation, while still being discussed as if exact.
I suspect one of the reasons there is so much anti-science sentiment/distrust in this country is because of how often the public sees a scientific "measurement" go awry, after it had been presented as "certain" (I realize this is often more the fault of press or other science-writer coverage, than due to the scientists themselves).