Sampling Simply

One of the first things learned in intro statistics class is the importance of a "random" (supposedly "representative") sample for any scientific study. You don't even really need to "learn" it, since it's such an obvious necessary requisite to attaining useful, meaningful data. So I was dismayed early on in academia when I realized how many "studies" were being done with say 25 psyche101 students, or maybe with 50 college freshman, or a sample from any number of college students… or hey, maybe even 100 Americans -- i.e., not in any way, an actual random human sample.
Despite being a psychology major, I couldn't put much faith in most behavioral psychology studies. Along the way, I noticed epidemiology too often suffered the same sort of problem, so it was no wonder that one week a study might show coffee was good for you, and the following week, with a different sample, coffee was bad for you; even "meta" studies that combine other studies, have sampling problems. It would be 35 years after I left college before the problem got some widespread journal attention when the notion of "WEIRD" samples was put forth -- study samples being based on subjects who were primarily "Western, Educated, Industrial, Rich and Democratic." And this extends to most routine studies (behavioral, psychological, medical) involving humans.
Perhaps it is because political pollsters have become so good at predicting election outcomes based on small samples (although even there glaring mistakes happen), that so many people readily accept the results of publicized scientific "studies" as widely true.

In fact, truly random samples in human studies are a virtual impossibility (or even if you had a random sample, there'd be no way to know it for sure), which is why most such studies ought be taken with a grain of salt, instead of broadcast to the world in simplistic press headlines.
I've written before that 'skepticism' needs to extend to most scientific studies that involve human subjects -- there's plenty of junky science in reputable journals, and funded by NIH -- and that's not a slam against science, but just an honest recognition of how difficult and rare really excellent science is (especially behavioral and medical science -- although even moving outside the realm of human studies, other writers have bemoaned the near metaphysical or pseudo-scientific gloss that some current-day theoretical physics has taken on).

Anyway, I mention all of this simply because today Cathy O'Neil tackles a similar area of concern, focusing on the prevalence of males specifically in clinical studies (...sheeesh, next thing you know, females will be wanting equal opportunities in STEM fields ;-)). Give her a read here:

http://mathbabe.org/2014/09/26/women-not-represented-in-clinical-trials/

[on a side-note, Sunday I should have a new interview up at MathTango]