isye 6413 design and analysis of experiments

Data Recovery

It appears you may have used Coursicle on this device and then cleared your cookies. You can recover your data by answering these questions.

Your account no longer exists

Your user ID no longer exists. Please refresh the page. If the issue persists, please contact us at [email protected].

ISYE 6413 - Dsgn & Analy-Experiments

Prepare for your exams

• Guidelines and tips

Study with the several resources on Docsity

Earn points by helping other students or get them with a premium plan

Prepare for your exams with the study notes shared by other students like you on Docsity

The best documents sold by students who completed their studies

Summarize your documents, ask them questions, convert them into quizzes and concept maps

Clear up your doubts by reading the answers to questions asked by your fellow students

Earn points to download

For each uploaded document

For each given answer (max 1 per day)

Choose a premium plan with all the points you need

Study Opportunities

Connect with the world's best universities and choose your course of study

Ask the community for help and clear up your study doubts

Discover the best universities in your country according to Docsity users

Free resources

Download our free guides on studying techniques, anxiety management strategies, and thesis advice from Docsity tutors

From our blog

5 Problems of Design and Analysis - Experiments - Midterm Exam 2 | ISYE 6413, Exams of Systems Engineering

Material Type: Exam; Professor: Wu; Class: Dsgn & Analy-Experiments; Subject: Industrial & Systems Engr; University: Georgia Institute of Technology-Main Campus; Term: Unknown 1989;

Typology: Exams

Uploaded on 08/05/2009

10 documents

Often downloaded together

Related documents, partial preview of the text.

Lecture notes

Study notes

Document Store

Latest questions

Biology and Chemistry

Psychology and Sociology

United States of America (USA)

United Kingdom

Sell documents

Seller's Handbook

How does Docsity work

United States of America

Terms of Use

Cookie Policy

Cookie setup

Privacy Policy

Sitemap Resources

Sitemap Latest Documents

Sitemap Languages and Countries

Copyright © 2024 Ladybird Srl - Via Leonardo da Vinci 16, 10126, Torino, Italy - VAT 10816460017 - All rights reserved

Skip to content

Georgia Institute of Technology

Search form.

• You are here:
• PhD Program

Curriculum: Electives

In addition to meeting the four core area requirements, each student is required to complete five elective courses. These courses are required for getting a complete breadth in ML. These courses must be chosen from at least two of the five subject areas listed below. In addition, students can use up to six special problems research hours to satisfy this requirement. 

After core requirements are satisfied, all courses listed in the core not already taken can be used (as appropriately classified) electives. 

Course offerings are subject to change. 

Statistics and Applied Probability: To build breadth and depth in the areas of statistics and probability as applied to ML

AE 6505, Kalman Filtering

BMED 6700, Biostatistics (rarely offered)

ECE 6558, Stochastic Systems

ECE 6601, Random Processes

ECE 6605, Information Theory

ISYE 6404, Nonparametric Data Analysis

ISYE 6413, Design and Analysis of Experiments

ISYE 6414, Regression Analysis

ISYE 6416, Computational Statistics

ISYE 6420, Bayesian Statistics

ISYE 6761, Stochastic Processes I

ISYE 6762, Stochastic Processes II

ISYE 7400, Adv Design-Experiments

ISYE 7401, Adv Statistical Modeling

ISYE 7405, Multivariate Data Analysis

ISYE 8803, Statistical and Probabilistic Methods for Data Science  (rarely offered)

ISYE 8813, Special Topics in Data Science 

MATH 6263, Testing Statistical Hypotheses

MATH 6266, Statistical Linear Modeling

MATH 6267, Multivariate Statistical Analysis

MATH 7244, Stochastic Processes and Stochastic Calculus I

MATH 7245, Stochastic Processes and Stochastic Calculus II

Advanced Theory: To build a deeper understanding of foundations of ML

CS 7280, Network Science  (rarely offered)

CS 7510, Graph Algorithms  (rarely offered)

CS 7520, Approximation Algorithms  (rarely offered)

CS 7530, Randomized Algorithms  (rarely offered)

CS 7535, Markov Chain Monte Carlo Algorithms 

CS 7540, Spectral Algorithms  (rarely offered)

CS 8803 CA, Continuous Algorithms  (rarely offered)

ECE 6283, Harmonic Analysis and Signal Processing  (rarely offered)

ECE 6555, Optimal Estimation

ISYE 7682, Convexity  (rarely offered)

MATH 6112, Advanced Linear Algebra

MATH 6221, Advanced Classical Probability Theory

MATH 6241, Probability I

MATH 6580, Introduction to Hilbert Space

MATH 7338, Functional Analysis

MATH 7586, Tensor Analysis  (rarely offered)

MATH 88XX, Special Topics: High Dimensional Probability and Statistics

Applications: To develop a breadth and depth in variety of applications domains impacted by/with ML

AE 6373, Advanced Design Methods

AE 8803, Machine Learning for Control Systems  (rarely offered)

AE 8803, Nonlinear Stochastic Optimal Control

BMED 6780, Medical Image Processing

BMED 8813BHI, Biomedical and Health Informatics

BMED 8813MHI, mHealth Informatics

BMED 8813MLB, Machine Learning in Biomedicine  (rarely offered)

BMED 8823ALG, OMICS Data and Bioinformatics Algorithms  (rarely offered)

CS 6440, Introduction to Health Informatics

CS 6465, Computational Journalism  (rarely offered)

CS 6471, Computational Social Science

CS 6474, Social Computing

CS 6475, Computational Photography  (rarely offered)

CS 6476, Computer Vision

CS 6601, Artificial Intelligence

CS 7450, Information Visualization

CS 7476, Advanced Computer Vision

CS 7630, Autonomous Robots  (rarely offered)

CS 7632, Game AI

CS 7636, Computational Perception  (rarely offered)

CS 7643, Deep Learning

CS 7646, Machine Learning for Trading  (rarely offered)

CS 7650, Natural Language Processing

CSE 6141, Massive Graph Analysis  (rarely offered)

CSE 6240, Web Search and Text Mining

CSE 6242, Data and Visual Analytics

CSE 6301, Algorithms in Bioinformatics and Computational Biology  (rarely offered)

ECE 4580, Computational Computer Vision  (rarely offered)

ECE 6255, Digital Processing of Speech Signals

ECE 6258, Digital Image Processing

ECE 6260, Data Compression and Modeling

ECE 6273, Methods of Pattern Recognition with Application to Voice

ECE 6550, Linear Systems and Controls

ECE 8813, Network Security  (rarely offered)

ISYE 6421, Biostatistics  (rarely offered)

ISYE 6810, Systems Monitoring and Prognosis

ISYE 7201, Production Systems

ISYE 7204 Info Prod & Ser Sys

ISYE 7203, Logistics Systems

ISYE 8813, Supply Chain Inventory Theory

HS 6000, Healthcare Delivery

MATH 6759, Stochastic Processes in Finance

MATH 6783, Financial Data Analysis  (rarely offered)

Computing and Optimization: To provide more breadth and foundation in areas of math, optimization, and computation for ML

CS 6515, Introduction to Graduate Algorithms

CS 6550, Design and Analysis of Algorithms

CSE 6140, Computational Science and Engineering Algorithms

CSE 6643, Numerical Linear Algebra

CSE 6644, Iterative Methods for Systems of Equations

CSE 6710, Numerical Methods I  (rarely offered)

CSE 6711, Numerical Methods II  (rarely offered)

ISYE 6644, Simulation

ISYE 6645, Monte Carlo Methods

ISYE 6662, Discrete Optimization

ISYE 6664, Stochastic Optimization

ISYE 6679, Computational methods for optimization

ISYE 7686, Advanced Combinatorial Optimization

ISYE 7687, Advanced Integer Programming  (rarely offered)

Platforms: To provide breadth and depth in computing platforms that support ML and Computation

CS 6421, Temporal, Spatial, and Active Databases  (rarely offered)

CS 6430, Parallel and Distributed Databases  (rarely offered)

CS 6290, High-Performance Computer Architecture

CSE 6220, High Performance Computing

CSE 6230, High Performance Parallel Computing

Georgia Tech Resources

• Offices & Departments
• News Center
• Campus Calendar
• Special Events
• Institute Communications

Visitor Resources

• Campus Visits
• Directions to Campus
• Visitor Parking Information
• GTvisitor Wireless Network Information
• Georgia Tech Global Learning Center
• Georgia Tech Hotel & Conference Center
• Barnes & Noble at Georgia Tech
• Ferst Center for the Arts
• Robert C. Williams Paper Museum

Georgia Institute of Technology North Avenue, Atlanta, GA 30332 Phone: 404-894-2000

Notes for ISyE 6413 Design and Analysis of...

Notes for ISyE 6413

Design and Analysis of Experiments

Instructor : C. F. Jeff WuSchool of Industrial and Systems Engineering

Georgia Institute of Technology

Text book : Experiments : Planning, Analysis, and Optimization

(by Wu and Hamada; Second Edition, Wiley, 2009)

Notes for course instructors: feel free to adapt the materials here to suit the

needs of your course (latex files also available).

Unit 1 : Basic Concepts and

Introductory Regresssion Analysis

Sources : Chapter 1.

• Historical perspectives and basic definitions (Section 1.1).

• Planning and implementation of experiments (Section 1.2).

• Fisher’s fundamental principles (Section 1.3).

• Simple linear regression (Sections 1.4-1.5).

• Multiple regression, variable selection (Sections 1.6-1.7).

• Example: Air Pollution Data (Section 1.8).

Historical perspectives

• Agricultural Experiments : Comparisons and selection of varieties (and/or

treatments) in the presence of uncontrollable field conditions, Fisher’s

pioneering work on design of experiments and analysis of variance

• Industrial Era : Process modeling and optimization, Large batch of

materials, large equipments, Box’s work motivated in chemical industries

and applicable to other processing industries, regressionmodeling and

response surface methodology.

Historical perspectives (Contd.)

• Quality Revolution : Quality and productivity improvement, variation

reduction, total quality management, Taguchi’s work on robust parameter

design, Six-sigma movement.

• A lot of successful applications in manufacturing (cars, electronics, home

appliances, etc.)

• Current Trends and Potential New Areas :Computer modelling and

experiments, large and complex systems, applications to biotechnology,

nanotechnology, material development, etc.

Types of Experiments

• Treatment Comparisons : Purpose is to compare several treatments of a

factor (have 4 rice varieties and would like to see if they aredifferent in

terms of yield and draught resistence).

• Variable Screening :Have a large number of factors, but only a few are

important. Experiment should identify the important few.

• Response Surface Exploration :After important factors have been

identified, their impact on the system is explored; regression model building.

Types of Experiments (Contd.)

• System Optimization : Interested in determining the optimum conditions

(e.g., maximize yield of semiconductor manufacturing or minimize defects).

• System Robustness :Wish to optimize a system and also reduce the impact

of uncontrollable (noise) factors. (e.g., would like cars to run well in

different road conditions and different driving habits; anIC fabrication

process to work well in different conditions of humidity anddust levels).

Some Definitions

• Factor : variable whose influence upon a response variable is being studied

in the experiment.

• Factor Level : numerical values or settings for a factor.

• Trial (or run ) : application of a treatment to an experimental unit.

• Treatment or level combination : set of values for all factors in a trial.

• Experimental unit : object to which a treatment is applied.

• Randomization : using a chance mechanism to assign treatments to

experimental units or run order.

Systematic Approach to Experimentation

• State the objective of the study.

• Choose the response variable. . . should correspond to the purpose of the

– Nominal-the-best, larger-the-better or smaller-the-better.

• Choose factors and levels.

– Use flow chart or cause-and-effect diagram (see Figure 1).

• Choose experimental design (i.e., plan).

• Perform the experiment (use a planning matrix to determine the set of

treatments and the order to be run).

• Analyze data (design should be selected to meet objective sothat the

analysis is efficient and easy).

• Draw conclusions.8

Cause-and Effect Diagram

�COSMETICDEFECTS

injection pressure

injection speed

mold temperature

nozzle temperature

holding time

barrel temperature

material shot length

pre-blend pigmentation

hopper cleaning

screw & barrel cleaning

operators on shifts

operator replacements

lack of training

Figure 1: Cause-and-Effect Diagram, Injection Molding Experiment9

Choice of Response : An Example

• To improve a process that often produces underweight soap bars.

Obvious choice of response,y = soap bar weight.

• There are two sub-processes : (i) mixing, which affects soapbar density

(=y1), (ii) forming, which affects soap bar dimensions (=y2).

• Even thoughy is a function ofy1 andy2, better to studyy1 andy2 separately

and identify factors important for each of the two sub-processes.

Fundamental Principles : Replication,

randomization, and blocking

Replication

• Each treatment is applied to units that are representative of the population

(example : measurements of 3 units vs. 3 repeated measurements of 1 unit).

• Replication vs Repetition (i.e., repeated measurements).

• Enable the estimation of experimental error. Use sample standard deviation.

• Decrease variance of estimates and increase the power to detect significant

differences : for independentyi ’s,

Randomization

Use of a chance mechanism (e.g., random number generators) to assign

treatments to units or to run order. It has the following advantages.

• Protect against latent variables or “lurking” variables (give an example).

• Reduce influence of subjective bias in treatment assignments (e.g., clinical

• Ensure validity of statistical inference (This is more technical; will not be

discussed in the book. See Chapter 4 of “Statistics for Experimenters” by

Box, Hunter, Hunter for discussion on randomization distribution.)

A block refers to a collection of homogeneous units. Effective blocking : larger

between-block variations than within-block variations.

(Examples: hours, batches, lots, street blocks, pairs of twins.)

• Run and compare treatments within the same blocks. (Use randomization

within blocks.) It can eliminate block-block variation andreduce variability

of treatment effects estimates.

• Block what you can and randomize what you cannot.

• Discusstyping experiment to demonstrate possible elaboration of the

blocking idea. See next page.

Illustration: Typing Experiment• To compare two keyboardsA andB in terms of typing efficiency. Six

manuscripts 1-6 are given to the same typist.

• Several designs (i.e., orders of test sequence) are considered:

1. A,B, 2. A,B, 3. A,B, 4. A,B, 5. A,B, 6. A,B.

(A always followed byB, why bad ?)

2. Randomizing the order leads to a new sequence like this

1. A,B, 2. B,A, 3. A,B, 4. B,A, 5. A,B, 6. A,B.

(an improvement, but there are four withA,B and two withB,A. Why isthis not desirable? Impact oflearning effect.)

3. Balanced randomization: To mitigate the learning effect, randomlychoose three withA,B and three withB,A. (Produce one such plan onyour own).

4. Other improved plans?14

Simple Linear Regression : Mortality Data

The data, taken from certain regions of Great Britain, Norway, and Sweden

contains the mean annual temperature (in degrees F) and mortality index for

neoplasms of the female breast.

Mortality rate (M) 102.5 104.5 100.4 95.9 87.0 95.0 88.6 89.2

Temperature (T) 51.3 49.9 50.0 49.2 48.5 47.8 47.3 45.1

Mortality rate (M) 78.9 84.6 81.7 72.2 65.1 68.1 67.3 52.5

Temperature (T) 46.3 42.1 44.2 43.5 42.3 40.2 31.8 34.0

Objective : Obtaining the relationship between mean annual temperature and

the mortality rate for a type of breast cancer in women.

Getting Started

35 40 45 50

Temperature

Figure 2: Scatter Plot of Temperature versus Mortality Rate, Breast Cancer Data.

Fitting the Regression Line

• Underlying Model :

y= β0+β1x+ ε, ε ∼ N(0,σ2).

• Coefficients are estimated by minimizing

yi − (β0+β1xi))2

• Least Squares EstimatesEstimated Coefficients :

β1 =∑(xi − x)(yi − y)

∑(xi − x)2 , var(

∑(xi − x)2 ,

β0 = y− β1x , var(

∑(xi − x)2)

x=1N ∑xi , y=

Explanatory Power of the Model

• The total variation iny can be measured by corrected total sum of squares

CTSS= ∑Ni=1(yi − y)2.

• This can be decomposed into two parts (Analysis of Variance (ANOVA)):

CTSS= RegrSS+RSS,

RegrSS= Regression sum of squares=N

RSS= Residual sum of squares=N

(yi − yi)2.

yi = β0+ β1xi is called the predicted value ofyi at xi .

• R2 = RegrSSCTSS = 1− RSS

CTSSmeasures the proportion of variation iny explained

by the fitted model.

ANOVA Table for Simple Linear Regression

Degrees of Sum of Mean

Source Freedom Squares Squares

regression 1 β1 ∑(xi − x)2 β1 ∑(xi − x)2

residual N−2 ∑Ni=1(yi − yi)

2 ∑Ni=1(yi−yi)

total (corrected) N−1 ∑Ni=1(yi − y)2

ANOVA Table for Breast Cancer Example

regression 1 2599.53 2599.53

residual 14 796.91 56.92

total (corrected) 15 3396.44

t-Statistic

• To test the null hypothesisH0 : β j = 0 against the alternative hypothesis

H0 : β j 6= 0, use the test statistic

• The higher the value oft, the more significant is the coefficient.

• For 2-sided alternatives,p-value= Prob(

|td f |> |tobs|)

, df = degrees of

freedom for thet-statistic,tobs = observed value of thet-statistic. Ifp-value

is very small, then either we have observed something which rarely

happens, orH0 is not true. In practice, ifp-value is less thenα = 0.05 or

0.01,H0 is rejected at levelα.

Confidence Interval

100(1−α)% confidence interval forβ j is given by

β j ± tN−2, α2×s.d.(β j),

wheretN−2, α2

is the upperα/2 point of thet distribution withN−2 degrees of

If the confidence interval forβ j does not contain 0, thenH0 is rejected.

Predicted Values and Residuals

• yi = β0+ β1xi is the predicted value ofyi at xi .

• r i = yi − yi is the corresponding residual.

• Standardized residuals are defined asr is.d.(r i)

• Plots of residuals are extremely useful to judge the “goodness” of fitted

– Normal probability plot (will be explained in Unit 2).

– Residuals versus predicted values.

– Residuals versus covariatex.

Analysis of Breast Cancer Data

The regression equation is

M = - 21.79 + 2.36 T

Predictor Coef SE Coef T P

Constant -21.79 15.67 -1.39 0.186

T 2.3577 0.3489 6.76 0.000

S = 7.54466 R-Sq = 76.5% R-Sq(adj) = 74.9%

Analysis of Variance

Source DF SS MS F P

Regression 1 2599.5 2599.5 45.67 0.000

Residual Error 14 796.9 56.9

Total 15 3396.4

Unusual Observations

Obs T M Fit SE Fit Residual St Resid

15 31.8 67.30 53.18 4.85 14.12 2.44RX

R denotes an observation with a large standardized residual.

X denotes an observation whose X value gives it large leverage.

Outlier Detection

• Minitab identifies two types of outliers denoted by R and X:

R: its standardized residual(yi − yi)/se(yi) is large.

X: its X value gives large leverage (i.e., far away from majority of the X

• For the mortality data, the observation with T = 31.8, M = 67.3(i.e., left

most point in plot on p. 16) is identified as both R and X.

• After removing this outlier and refitting the remaining data, the output is

given on p. 27. There is still an outlier identified as X but notR. This one

(second left most point on p.16) should not be removed (why?)

• Residual plots on p. 28 show no systematic pattern.

Notes: Outliers are not discussed in the book, see standard regression texts.

Residual plots will be discussed in unit 2.

Prediction from the Breast Cancer Data

• The fitted regression model isY =−21.79+2.36X, whereY denotes the

mortality rate andX denotes the temperature.

• The predicted mean ofY atX = x0 can be obtained from the above model.

For example, prediction for the temperature of 49 is obtained by substituting

x0 = 49, which givesyx0 = 93.85.

• The standard error ofyx0 is given by

S.E.(yx0) = σ

∑Ni=1(xi − x)2

• Herex0 = 49, 1/N+(x−x0)2/∑N

i=1(xi − x)2 = 0.1041, and

MSE= 7.54. Consequently,S.E.(yx0) = 2.432.

Confidence interval for mean and prediction interval

for individual observation

• A 95% confidence interval for the mean responseβ0+β1x0 of y atx= x0 is

β0+ β1x0± tN−2,0.025×S.E.(yx0).

• Here the 95% confidence interval for the mean mortality corresponding to a

temperature of 49 is [88.63, 99.07].

• A 95% prediction interval for an individual observationyx0 corresponding tox= x0

β0+ β1x0± tN−2,0.025σ

where 1 under the square root representsσ2, variance of thenewobservationyx0.

• The 95% prediction interval for the predicted mortality of an individual

corresponding to the temperature of 49 is [76.85, 110.85].

Regression Results after Removing the Outlier

M = - 52.62 + 3.02 T

Constant -52.62 15.82 -3.33 0.005

T 3.0152 0.3466 8.70 0.000

S = 5.93258 R-Sq = 85.3% R-Sq(adj) = 84.2%

Regression 1 2664.3 2664.3 75.70 0.000

Residual Error 13 457.5 35.2

Total 14 3121.9

15 34.0 52.50 49.90 4.25 2.60 0.63 X

Residual Plots After Outlier Removal

50 60 70 80 90 100

Residuals versus Fitted Values

Fitted Value

Residuals versus Temperature

Figure 3: Residual Plots

Comments :No systematic pattern is discerned.

Multiple Linear Regression : Air Pollution Data

http://lib.stat.cmu.edu/DASL/Stories/AirPollutionandMortality.html

• Data collected by General Motors.

• Response is age-adjusted mortality.

• Predictors :

– Variables measuring demographic characteristics.

– Variables measuring climatic characteristics.

– Variables recording pollution potential of 3 air pollutants.

• Objective : To determine whether air pollution is significantly related to

Predictors1. JanTemp : Mean January temperature (degrees Farenheit)

2. JulyTemp : Mean July temperature (degrees Farenheit)

3. RelHum : Relative Humidity

4. Rain : Annual rainfall (inches)

5. Education : Median education

6. PopDensity :Population density

7. %NonWhite : Percentage of non whites

8. %WC : Percentage of white collar workers

9. pop : Population

10. pop/house :Population per household

11. income : Median income

12. HCPot : HC pollution potential

13. NOxPot : Nitrous Oxide pollution potential

14. SO2Pot :Sulphur Dioxide pollution potential

• There are 60 data points.

• Pollution variables are highly skewed, log transformationmakes them

nearly symmetric. The variables HCPot, NOxPot and SO2Pot are replaced

by log(HCPot), log(NOxPot) and log(SO2Pot).

• Observation 21 (Fort Worth, TX) has two missing values, so this data point

will be discarded from the analysis.

Scatter PlotsFigure 4: Scatter Plots of mortality against selected predictors

(a) JanTemp (b) Education

10 20 30 40 50 60

9.0 9.5 10.0 10.5 11.0 11.5 12.0

(c) NonWhite (d) Log(NOxPot)

0 10 20 30 40

0.0 0.5 1.0 1.5 2.0 2.5

Fitting the Multiple Regression Equation

y= β0+β1x1+β2x2+ . . .+βkxk+ ε, ε ∼ N(0,σ2).

yi − (β0+β1xi1+β2xi2+ . . .+βkxik))2

= (y−Xβ)′(y−Xβ).

• Least Squares estimates :

β = (X′X)−1X′y.

• Variance-Covariance matrix ofβ : Σβ = σ2(X′X)−1.

• The total variation iny, i.e., corrected total sum of squares,

CTSS= ∑Ni=1(yi − y)2 = yTy−Ny2, can be decomposed into two parts

(Analysis of Variance (ANOVA)):

whereRSS= Residual sum of squares= ∑(yi − yi)2 = (y−Xβ)T(y−Xβ),

RegrSS= Regression sum of squares= ∑Ni=1(yi − y)2 = β

TXTXβ −Ny2.

ANOVA Table

regression k βT

XTXβ −Ny2 (βT

XTXβ −Ny2)/k

residual N−k−1 (y−Xβ )T(y−Xβ ) (y−Xβ )T(y−Xβ )/(N−k−1)

total N−1 yTy−Ny2

(corrected)

CTSSmeasures of the proportion of variation iny

explained by the fitted model.R is called the multiple correlation coefficient.

• Adjusted R2 :

• When an additional predictor is included in the regression model,R2 always

increases. This is not a desirable property for model selection. However,R2a

may decrease if the included variable is not an informative predictor.

UsuallyR2a is a better measure of model fit.

Testing significance of coefficients :t-Statistic

• In practice, ifp-value is less thenα = 0.05 or 0.01,H0 is rejected.

• Confidence Interval : 100(1−α)% confidence interval forβ j is given by

β j ± tN−(k+1), α2×s.d.(β j),

wheretN−k−1, α2

is the upperα/2 point of thet distribution withN−k−1

degrees of freedom.

Analysis of Air Pollution DataPredictor Coef SE Coef T P

Constant 1332.7 291.7 4.57 0.000JanTemp -2.3052 0.8795 -2.62 0.012JulyTemp -1.657 2.051 -0.81 0.424

RelHum 0.407 1.070 0.38 0.706Rain 1.4436 0.5847 2.47 0.018

Educatio -9.458 9.080 -1.04 0.303PopDensi 0.004509 0.004311 1.05 0.301%NonWhit 5.194 1.005 5.17 0.000

%WC -1.852 1.210 -1.53 0.133pop 0.00000109 0.00000401 0.27 0.788pop/hous -45.95 39.78 -1.16 0.254

income -0.000549 0.001309 -0.42 0.677logHC -53.47 35.39 -1.51 0.138

logNOx 80.22 32.66 2.46 0.018logSO2 -6.91 16.72 -0.41 0.681

S = 34.58 R-Sq = 76.7% R-Sq(adj) = 69.3%

Regression 14 173383 12384 10.36 0.000Residual Error 44 52610 1196Total 58 225993

Variable Selection Methods

• Principle of Parsimony (Occam’s razor): Choose fewer variables with

sufficient explanatory power. This is a desirable modeling strategy.

• The goal is thus to identify the smallest subset of covariates that provides

good fit. One way of achieving this is to retain the significantpredictors in

the fitted multiple regression. This may not work well if somevariables are

strongly correlated among themselves or if there are too many variables

(e.g., exceeding the sample size).

• Two other possible strategies are

– Best subset regression using Mallows’Cp statistic.

– Stepwise regression.

Best Subset Regression

• For a model withp regression coefficients, (i.e.,p−1 covariates plus the

interceptβ0), define itsCp value as

Cp =RSSs2 − (N−2p),

whereRSS= residual sum of squares for the given model,s2 = mean square

error = RSS(for the complete model)df (for the complete model), N = number of observations.

• If the model is true, thenE(Cp)≈ p. Thus one should choosep by picking

models whoseCp values are low and close top. For the samep, choose a

model with the smallest Cp value(i.e., the smallest RSS value).

AIC and BIC Information Criteria

• The Akaike information criterion (AIC) is defined by

AIC = Nln(RSSN

• The Bayes information criterion (BIC) is defined by

BIC = Nln(RSSN

• In choosing a model with the AIC/ BIC criterion, we choose themodel that

minimizes the criterion value.

• Unlike theCp criterion, the AIC criterion is applicable even if the number of

observations do not allow the complete model to be fitted.

• The BIC criterion favors smaller models more than the AIC criterion.

Stepwise Regression

• This method involves adding or dropping one variable at a time from a given

model based on apartial F-statistic.

Let the smaller and bigger models be Model I and Model II, respectively.

The partialF-statistic is defined as

RSS(Model I)−RSS(Model II)RSS(Model II)/ν

whereν is the degrees of freedom of theRSS(residual sum of squares) for

• There are three possible ways

1. Backward elimination : starting with the full model and removing covariates.

2. Forward selection : starting with the intercept and adding one variable at a time.

3. Stepwise selection :alternate backward elimination and forward selection.

Usually stepwise selection is recommended.

Criteria for Inclusion and Exclusion of Variables

• F-to-remove : At each step of backward elimination, compute the partialF

value for each covariate being considered for removal. The one with the

lowest partialF , provided it is smaller than a preselected value, is dropped.

The procedure continues until no more covariates can be dropped. The

preselected value is often chosen to beF1,ν,α, the upperα critical value of

theF distribution with 1 andν degrees of freedom. Typicalα values range

from 0.1 to 0.2.

• F-to-enter : At each step of forward selection, the covariate with the

largest partialF is added, provided it is larger than a preselectedF critical

value, which is referred to as anF-to-entervalue.

• For stepwise selection, theF-to-remove andF-to-enter values should be

chosen to be the same.

(See Section 1.7)

Air Pollution Example: Best Subsets Regression

Vars R-Sq R-Sq(adj) C-p BIC S variables

4 69.7 67.4 8.3 608 35.6 1,4,7,13

5 72.9 70.3 4.3 606 34.0 1,4,5,7,13

6 74.2 71.3 3.7 607 33.5 1,4,6,7,8,13

7 75.0 71.6 4.3 609 33.3 1,4,6,7,8,12,13

8 75.4 71.5 5.4 612 33.3 1,4,5,7,8,10,12,13

Pollution Data Analysis - Stepwise RegressionStepwise Regression: Mortality versus JanTemp, JulyTemp, ...

Alpha-to-Enter: 0.15 Alpha-to-Remove: 0.15

Response is Mortality on 14 predictors, with N = 59

N(cases with missing observations) = 1 N(all cases) = 60

Step 1 2 3 4 5 6 7

Constant 887.9 1208.5 1112.7 1135.4 1008.7 1029.5 1028.7

%NonWhit 4.49 3.92 3.92 4.73 4.36 4.15 4.15

T-Value 6.40 6.26 6.81 7.32 6.73 6.60 6.66

P-Value 0.000 0.000 0.000 0.000 0.000 0.000 0.000

Educatio -28.6 -23.5 -21.1 -14.1 -15.6 -15.5

T-Value -4.32 -3.74 -3.47 -2.10 -2.40 -2.49

P-Value 0.000 0.000 0.001 0.041 0.020 0.016

logSO2 28.0 21.0 26.8 -0.4

T-Value 3.37 2.48 3.11 -0.02

P-Value 0.001 0.016 0.003 0.980

Pollution Data Analysis - Stepwise Regression(Contd)

Response is Mortality on 14 predictors, with N = 59N(cases with missing observations) = 1 N(all cases) = 60

JanTemp -1.42 -1.29 -2.15 -2.14T-Value -2.41 -2.26 -3.25 -4.17P-Value 0.019 0.028 0.002 0.000

Rain 1.08 1.66 1.65T-Value 2.15 3.07 3.16P-Value 0.036 0.003 0.003

logNOx 42 42T-Value 2.35 4.04P-Value 0.023 0.000

S 48.0 42.0 38.5 37.0 35.8 34.3 34.0R-Sq 41.80 56.35 63.84 67.36 69.99 72.86 72.86R-Sq(adj) 40.78 54.80 61.86 64.94 67.16 69.73 70.30C-p 55.0 29.5 17.4 12.7 9.7 6.3 4.3BIC 634.52 621.62 614.60 612.63 611.76 609.90 605.83

Final Model

Rival Models

Variables Cp BIC Remarks

Model 1 1,4,6,7,8,13 3.7 607 Minimum Cp

Model 2 1,4,5,7,13 4.3 606 Minimum BIC and chosen by stepwise

We shall analyze data with Model 2. (Why? Refer to the rules onpage 38 and

use the principle of parsimony.)

Analysis of Model 2

Constant 1028.67 80.96 12.71 0.000

JanTemp -2.1384 0.5122 -4.17 0.000

Rain 1.6526 0.5225 3.16 0.003

Education -15.542 6.235 -2.49 0.016

%NonWhite 4.1454 0.6223 6.66 0.000

logNOx 41.67 10.32 4.04 0.000

S = 34.0192 R-Sq = 72.9% R-Sq(adj) = 70.3%

Regression 5 164655 32931 28.45 0.000

Residual Error 53 61337 1157

Total 58 225993

Residual Plot

850 900 950 1000 1050

Fitted Values

Residual versus Fitted Values

Figure 6 : Plot of Residuals

Comments on Board

A Branc - ISyE

ISyE 6416: Computational Statistics Lecture 1: Introductionyxie77/isye6416/Lecture1.pdf · 2015-01-06 · ISyE 6416: Computational Statistics Lecture 1: Introduction Prof. Yao Xie

SIMULATION OUTPUT ANALYSIS - ISyE Homesman/courses/Mexico2010/Module09-Output... · SIMULATION OUTPUT ANALYSIS Dave Goldsman School of ISyE, Georgia Tech, Atlanta, Georgia, USA May

Word Count: 6413

Fir s t i n I t s C l a s - ISyE Home | ISyE | Georgia ... · program in ISyE measures up in comparison to others. I think you will find the piece interesting, informative, and maybe

ISYE 2028 Chapter 8 Solutions

ISyE 6416: Computational Statistics Lecture 1: Introductionyxie77/isye6416/Lecture1.pdf · 2015-01-06 · ISyE 6416: Computational Statistics Lecture 1: Introduction ... What this

6413 User Manual

M01 FALE REA 04GLB 6413 U01 - English Center

Superstition Springs March Elementary 480-641-6413 ...

Notes for ISyE 6413 Design and Analysis of Experiments - H. Milton

Lecture 8: Deep Learning - ISyE Hometzhao80/Lectures/Lecture_8.pdf · Lecture 8: Deep Learning Tuo Zhao Schools of ISyE and CSE, Georgia Tech

6413 Operational Guide VI

Fall 2013 ISyE Alumni Mag

Unit 8: Robust Parameter Design - ISyE Home | ISyE ...jeffwu/isye6413/unit_08_12spring.pdf · Unit 8: Robust Parameter Design Source : Chapter 11 (sections 11.1 - 11.6, part of sections

Lecture 7: Unsupervised Learning - ISyE Hometzhao80/Lectures/Lecture_7.pdf · Lecture 7: Unsupervised Learning Tuo Zhao Schools of ISyE and CSE, Georgia Tech. CS7641/ISYE/CSE 6740:

ISyE 6201: Manufacturing Systems Instructor : Spyros ... · ISyE 6201: Manufacturing Systems Instructor : Spyros Reveliotis ... ISYE 6201 Spring 2007 Homework 4 Solution 2 A. Question

Derailment of Freight Train 6413