Sunday, November 8, 2015
Bye Bye Statistical Independence
"It's time for science to retire the fiction of statistical independence.
"The world is massively interconnected through causal chains. Gravity alone causally connects all objects with mass. The world is even more massively correlated with itself. It is a truism that statistical correlation doesn't imply causality. But it is a mathematical fact that statistical independence implies no correlation at all. None. Yet events routinely correlate with one another. The whole focus of most Big Data algorithms is to uncover just such correlations in ever larger data sets....
"A revealing problem is that there are few tests for statistical independence. Most tests tell at most whether two variables (not the data itself) are independent. And most scientists would be hard pressed to name even them. So the overwhelming common practice is simply to assume that sampled events are independent. Just assume that the data are white. Just assume that the data are not only from the same probability distribution but also statistically independent. An easy justification for this is that almost everyone else does it and it's in the textbooks. This assumption has to be one of the most widespread instances of groupthink in all of science."
-- Bart Kosko, in John Brockman's "This Idea Must Die"