Linear Probability Models (LPM) and Big Data: The Good, The Bad, and The Ugly

Chatla, S and Shmueli, G (2013) Linear Probability Models (LPM) and Big Data: The Good, The Bad, and The Ugly. Working Paper. Indian School of Business, Hyderabad.

[thumbnail of forest.docx]


Sizes of datasets used in academic research are growing quickly, with many studies using tens and hundreds of thousands or even millions of records. Linear regression is among the most popular statistical model in social sciences research. Linear probability models, which are linear regression models applied to a binary outcome, are commonly used for various reasons, despite criticisms of such usage. We carry out an extensive study to evaluate the use of LPMs in the realm of "Big Data", where large samples and many variables are available. We evaluate performance in terms of coefficient estimation as well as predictive power. We compare performance to alternatives suggested in the literature. We find that the LPM is beneficial for descriptive modeling when the outcome is naturally binary, whereas it is beneficial for predictive modeling when the outcome is binary by discretization. We motivate and illustrate our study through an application to modeling price in online auctions, using real data from the online auction site eBay.

ISB Creiators:
ISB Creators
Chatla, S
Shmueli, G
Item Type: Monograph (Working Paper)
Uncontrolled Keywords: linear regression, binary outcome, estimation, prediction, weighted least squares, large sample, online auctions, eBay
Subjects: Business Analytics
Depositing User: LRC ISB
Date Deposited: 31 Oct 2014 04:58
Last Modified: 20 Jan 2015 10:17
Publisher URL:
Related URLs:

Actions (login required)

View Item View Item
Statistics for DESI ePrint 84 Statistics for this ePrint Item