Computer Intensive Statistical Methods

Project 1: Logistic regression and bootstrap method for real data analysis

This project covers five learning outcomes: install and use computing package; using general linear model for real data analysis; logistic regression; familiar with application of bootstrapping method; get used to writing interpretation of analysis results.

Via install.packages("ISLR"), this project considers an analysis of the data set named "Default" in "ISLR", where the response default falls into one of two categories, Yes or No. The data also include observations on income and balance.

Take Y as the categoric variable, learn the R-function glm to fit a logistic regression. You are asked to

(1) implement a logistic regression model to predict the probability of default by using income and bal- ance as the independent variables;

(2) provide interpretations of fitted model;

(3) explore bootstrap method for this logistic regression analysis.

(4) There is another data set named ‘Carseats' in "ISLR", which attempt to predict Sales (child car seat sales) Y in 400 locations based on a number of predictors (or independent variables). The Carseats data includes qualitative predictors such as Shelveloc, an indicator of the quality of the shelving loca- tion-that is, the space within a store in which the car seat is displayed-at each location. One predictor named Shelveloc takes on three possible values (Bad, Medium, and Good) which is a qualitative variable. Fit a multiple linear regression with a number of predictors, including Shelveloc and then provide an interpretation of the fitted results.

Project 2: Regression analysis of big data

This project covers five learning outcomes: least-squares regression coefficient estimate formula for the analysis of dividable ‘big' data or streaming data; aggregation and compression technique ; implementa- tion of "divide-and-conquer" algorithm in R; logarithm transformation of skewed response variable Y in regression analysis; get used to writing interpretation of analysis results.

CALIFORNIA HOUSING DATA, which was originally used by Pace and Barry (1997) (Sparse spatial autoregressions, Statistics and Probability Letters 33 291-297), is available at CMU StatLib repository. It consists of aggregated data from each of 20640 neighborhoods (1990 census block groups) in California. The response variable Y is the median house value in each neighborhood measured in units of $100,000. There are eight continuous input/independent variables, which are demographics (e.g. median income), housing density and occupancy, housing properties (e.g. number of rooms/bedrooms), and location of each neighborhood. This project aims to use aggregation and compression technique based on the "divide-and-conquer" strategy for a mean regression analysis of this data, including the interpretation of the analysis results.

(1) Properly select one independent variable from the data and carry out a polynomial regression analysis with Y as the response by using "divide-and-conquer" algorithm.

(2) Properly select more than one independent variable from the data and carry out a multiple linear regression analysis with Y as the response by using "divide-and-conquer" algorithm.

(3) Show the response variable Y (median house value) is highly skewed to right, and then apply a loga- rithm transformation of Y as the response and repeat (1) and (2).
(4) Comment on the analyses above and the "divide-and-conquer" algorithm.

Project 3: Regression analysis of big data

This project covers five learning outcomes: regression analysis of a real data; least-squares regression coefficient estimate formula for the analysis of dividable ‘big' data or streaming data; implementation of "divide-and-conquer" algorithm in R; the distributed Big Data approach; get used to writing interpreta- tion of analysis results.

The airline on-time performance data from the 2009 ASA Data Expo is used as a case study to demonstrate a logistic model fitting with a massive dataset that exceeds the RAM of a single computer. The data is publicly available and has been used for demonstration with big data analysis by several groups, which including Wang et al. (2016) (A Survey of Statistical Methods and Computing for Big Data, Statistics and Its Interface 9(4): 399-414.), Kane et al. (2013) (Scalable strategies for computing with massive data, Journal of Statistical Software 55(14).)

The data consist of flight arrival and departure details for all commercial flights within the USA, from Oc- tober 1987 to April 2008. About 12 million flights were recorded with 29 variables. A compressed version of the pre-processed data set from the big memory project is approximately 1.7GB, and it takes 12GB when uncompressed.

Wang et al. (2016) defined a binary response as 1 if a flight was late by more than 15 minutes and 0 other- wise, then carried our logistic regression analysis. Two binary covariates were created from the departure time: night (1 if departure occurred between 8pm and 5am) and weekend (1 if departure occurred on weekends and 0 otherwise). Two continuous covariates were included: departure hour (DepHour, range 0 to 24) and distance from origin to destination (in 1000 miles).

In this small project, by using flight late time (without transferring it into binary observation) as the response Y , this projects aims to use aggregation and compression technique based on the "divide-and- conquer" strategy for a mean regression analysis of this data, including the interpretation of the analysis results and comparison with logistic regression.

(1) Carry out a multiple linear regression analysis by using Y and two binary covariates and "divide-and- conquer" algorithm.

(2) Carry out a multiple linear regression analysis by using Y and two continuous covariates and "divide- and-conquer" algorithm.

(3) Carry out a multiple linear regression analysis by using Y and all four covariates and "divide-and- conquer" algorithm.

(4) Comment on the analyses above and the "divide-and-conquer" algorithm.

Attachment:- Logistic regression.rar