Linear Probability Models (LPM) and Big Data: The Good, The Bad, and The Ugly

Chatla, S and Shmueli, G (2013) Linear Probability Models (LPM) and Big Data: The Good, The Bad, and The Ugly. Working Paper. Indian School of Business.

[thumbnail of forest.docx] Text

Download (21kB)
Publisher URL:


Sizes of datasets used in academic research are growing quickly, with many studies using tens and hundreds of thousands or even millions of records. Linear regression is among the most popular statistical model in social sciences research. Linear probability models, which are linear regression models applied to a binary outcome, are commonly used for various reasons, despite criticisms of such usage. We carry out an extensive study to evaluate the use of LPMs in the realm of "Big Data", where large samples and many variables are available. We evaluate performance in terms of coefficient estimation as well as predictive power. We compare performance to alternatives suggested in the literature. We find that the LPM is beneficial for descriptive modeling when the outcome is naturally binary, whereas it is beneficial for predictive modeling when the outcome is binary by discretization. We motivate and illustrate our study through an application to modeling price in online auctions, using real data from the online auction site eBay.

Item Type: Monograph (Working Paper)
Subjects: Business Analytics
Date Deposited: 31 Oct 2014 04:58
Last Modified: 26 Jul 2023 12:34

Actions (login required)

View Item
View Item