Wired's "End of Theory" Six Years Later

With all this Big Data and Data Science available at our fingertips, do we need to even think anymore?

This was essentially the question asked by a June 23, 2008 Wired article, written back when Map/Reduce was so new and hot it was unheard of by most, The End of Theory: The Data Deluge Makes the Scientific Method Obsolete.

The Dilbert comic from today highlights why, when taken to an extreme, this approach is absurd. If we construct hypotheses "randomly" as the Dilbert comic suggests, progress will be too slow, costs will be too high, we'll run out of human subjects to test upon, we'll miss the boat on actually capturing customers, and the more testing on users that is done, the greater the risk of running into the ethical minefield of user manipulation.

So we need thoughtful hypothesis formation. My blog entry from January covers what makes a good hypothesis and how to go about forming one. As William Whewell declared in the early 19th century, "invention, sagacity, genius" are required at every step of the scientific method. The graphic designer in the Dilbert strip is not a genius, and Wired's 2008 suggestion that we eliminate theory is valid only if data, hypothesis formation and hypothesis testing are all free. Data might be considered free in the case of Quasi-Experimental Design where data science is performed on a data lake. But data is very expensive in the case of randomized trials, of which UX A/B testing is an example. And besides, any insight derived from quasi-experimental design needs to be verified by a randomized trial. In any case, good hypothesis formulation is never cheap.

The 2008 Wired article posits that relying on theory and relying on data are opposites. The Dilbert graphic designer apparently failed to rely on the theories within the fields of graphic design and of human psychology (remember the field of HCI?), and instead chose a "random" A/B test. The Dilbert graphic designer chose to rely on data rather than theory.

With all the bad UX A/B testing going on out there, it may then be tempting to rely on theory alone. But where does all this theory come from in the first place? From researchers conducting experiments using the scientific method. That means that UX designers conducting A/B tests are using the scientific method. Most use it badly. But if they formulate their hypotheses intelligently, and if they publish their results, then they use the scientific method well. In using the data-based technique of A/B testing, they would (if done right), actually be expanding theory -- theory that they and others can reuse in the future.

The scientific method advances human knowledge by using data to derive theory. Data without theory is fine only under 2008 Big Data theories of "collect the data and the insights come automatically."