Contents
Chapter 1 Preliminaries: Matrix Algebra and Random Vectors ........ 1
1.1 Preliminary matrix algebra
........................................ 1
1.1.1 Trace and eigenvalues........................................
1
1.1.2 Symmetric
matrices........................................ 3
1.1.3 Idempotent matrices and orthogonal projection..................... 6
1.1.4 Singular value decomposition
........................................ 9
1.1.5 Vector di.erentiation and generalized inverse
.......................10
1.1.6 Exercises ........................................10
1.2 Expectation and
covariance........................................11
1.2.1 Basic properties
........................................11
1.2.2 Mean and variance of quadratic forms
.................................12
1.2.3 Exercises
........................................14
1.3 Moment generating functions and independence
.............................16
1.3.1 Exercises
........................................17
Chapter 2 Multivariate Normal
Distributions.....................................18
2.1 De.nitions and fundamental results
........................................18
2.2 Distribution of quadratic forms
........................................25
2.3 Exercises........................................27
Chapter 3 Linear Regression Models
........................................29
3.1 Introduction........................................29
3.2 Regression interpreted as conditional mean
....................................31
3.3 Linear regression interpreted as linear prediction
............................33
3.4 Some nonlinear
regressions........................................34
3.5 Typical data structure of linear regression models
..........................35
3.6 Exercises........................................36
Chapter 4 Estimation and Distribution Theory
..................................38
4.1 Least squares estimation (LSE) ........................................38
4.1.1 Motivation: why is LS
reasonable........................................38
Regression Analysis
4.1.2 The LS solution
........................................40
4.1.3 Exercises
........................................48
4.2 Properties of LSE
........................................50
4.2.1 Small sample distribution-free properties
.............................51
4.2.2 Properties under normally distributed
errors........................55
4.2.3 Asymptotic properties
........................................57
4.2.4 Exercises
........................................60
4.3 Estimation under linear restrictions
........................................60
4.4 Introducing further explanatory variables and related topics
...........67
4.4.1 Introducing further explanatory variables
............................67
4.4.2 Centering and scaling the explanatory variables ...................71
4.4.3 Estimation in terms of linear
prediction...............................72
4.4.4 Exercises
........................................73
4.5 Design matrices of less than full
rank........................................74
4.5.1 An example
........................................74
4.5.2 Estimability........................................74
4.5.3 Identi.ability under
Constraints........................................76
4.5.4 Dropping variables to change the model
..............................77
4.5.5 Exercises ........................................77
4.6 Generalized least squares
........................................78
4.6.1 Basic theory ........................................78
4.6.2 Incorrect speci.cation of variance
matrix.............................80
4.6.3 Exercises
........................................83
4.7 Bayesian
estimation........................................83
4.7.1 The basic
idea........................................83
4.7.2 Normal-noninformative structure
........................................84
4.7.3 Conjugate priors
........................................85
4.8 Numerical
examples........................................86
4.9
Exercises........................................90
Chapter 5 Testing Linear Hypotheses
........................................92
5.1 Linear
hypotheses........................................92
5.2 F -Test
........................................93
5.2.1 F -test........................................94
Contents VII
5.2.2 What are actually tested
........................................95
5.2.3 Examples........................................96
5.3 Con.dence ellipse
........................................ 101
5.4 Prediction and calibration........................................103
5.5 Multiple correlation coe.cient
........................................ 105
5.5.1 Variable selection
........................................ 105
5.5.2 Multiple correlation coe.cient: straight line ......................
106
5.5.3 Multiple correlation coe.cient: multiple regression ............ 108
5.5.4 Partial correlation coe.cient ........................................
110
5.5.5 Adjusted multiple correlation coe.cient ............................
111
5.6 Testing linearity: goodness-of-.t test
........................................ 112
5.7 Multiple
comparisons........................................114
5.7.1 Simultaneous inference
........................................ 114
5.7.2 Some classical methods for simultaneous intervals .............. 116
5.8 Univariate analysis of variance
........................................ 120
5.8.1 ANOVA
model........................................ 120
5.8.2 ANCOVA model
........................................ 126
5.8.3 SAS procedures for ANOVA
........................................ 127
5.9 Exercises........................................
129
Chapter 6 Variable
Selection........................................ 133
6.1 Impact of variable selection
........................................ 133
6.2 Mallows’ Cp
........................................
137
6.3 Akaike’s information criterion
(AIC)........................................139
6.3.1 Prelimilaries: asymptotic normality of MLE ...................... 140
6.3.2 Kullback-Leibler’s distance
........................................ 143
6.3.3 Akaike’s Information Criterion ........................................
144
6.3.4 AIC for linear
regression........................................147
6.4 Bayesian information criterion (BIC)
........................................ 150
6.5 Stepwise variable selection
procedures........................................ 152
6.6 Some newly proposed methods
........................................ 154
6.6.1 Penalized RSS........................................
154
6.6.2 Nonnegative garrote
........................................ 157
6.7 Final remarks on variable selection
........................................ 158
Regression Analysis
6.8
Exercises........................................
161
Chapter 7 Miscellaneous for Linear Regression
................................ 165
7.1 Collinearity
........................................
165
7.1.1 Introduction
........................................ 165
7.1.2 Examine
collinearity........................................ 166
7.1.3
Remedies........................................169
*7.2 Some remedies for collinearity
........................................ 170
7.2.1 Ridge
regression........................................170
7.2.2 Principal Component Regression
....................................... 173
7.2.3 Partial least square
........................................ 175
7.2.4 Exercises
........................................ 176
7.3 Outliers ........................................
177
7.3.1 Introduction
........................................ 177
7.3.2 Single outlier
........................................ 179
7.3.3 Multiple outliers
........................................ 181
7.3.4 Relevant
quantities........................................ 183
7.3.5
Remarks........................................185
7.4 Testing features of
errors........................................ 186
7.4.1 Serial correlation and
Durbin-Watson test ......................... 186
7.4.2 Testing heteroskeasticity and
related topics ....................... 187
7.5 Some extensions and variants
........................................ 191
7.5.1 Box-Cox model
........................................ 191
7.5.2 Modeling the variances
........................................ 192
7.5.3 A
remark........................................193
Chapter 8 Logistic Regression: Modeling Categorical Responses ... 194
8.1 Logistic regression
........................................ 194
8.1.1 Logistic regression for
dichotomous responses..................... 194
8.1.2 Likelihood function for
logistic regression........................... 196
8.1.3 Interpreting the logistic
regression.....................................198
8.2 Multiple logistic regression
........................................ 199
8.2.1 Maximum likelihood estimation
for multiple logistic regression
........................................ 200
Contents IX
8.3 Inference for logistic regression
........................................ 202
8.3.1 Inference for simple logistic regression
............................... 202
8.3.2 Inference for multiple logistic
regression............................. 204
8.4 Logistic regression for multinomial
responses................................205
8.4.1 Nominal responses baseline-category logistic regression.......206
8.4.2 Ordinal responses: cumulative logistic regression................208
8.5 Exercises........................................
210
Bibliography
........................................
212
