Testing Theories with Big Data: A Super-Power Approach

In the early days of IS research, a large sample was over 100 records. In today's Big Data era, papers routinely report many thousands, and even millions of records. Large samples provide a powerful tool for testing hypotheses. Applying small-sample modeling to large samples not only wastes the "super-power" advantages but can also lead to incorrect and misleading conclusions. In particular, it is prone to super-low p-values and the inability to rely on p-values for testing hypotheses. We introduce a "super power" approach for testing hypotheses with Big Data. While the statistical literature describes the asymptotic behavior of estimators and models, those results do not seem to impact practices in IS research, probably due to their highly theoretical nature. We focus on regression models that are popular with IS researchers, but the approach generalizes to inference with other statistical models. The super-power approach encompasses the different modeling steps from hypotheses framing and study design to data visualization, model building, validation and inference. The super-power approach enables testing more pointed and more complex hypotheses, including more control variables, quantifying more subtle and rare relationships, improving robustness checking, strengthening model validity and generalizability, developing insights through analysis of subsamples, and making inferences even in the presence of some violated model assumptions.