Multiple Regression in R
This R Guide for multiple regression hypothesis testing was created by Jason A. French. Email him with any comments and suggestions.
Before we begin, if you’re using this R Guide I’m assuming some previous knowledge:
 You’re familiar with how to enter basic commands. If not, there are plenty of tutorials regarding this, many of which are written by my adviser, Bill Revelle. I don’t want to redo what he has more successfully done.
 The dataset was successfully entered into R. If not, once again Bill comes to the rescue with his guide. I’m generally a fan of exporting my data as a .csv file and using the
read.csv()
function (.e.g,data < read.csv(mydata.csv)
). However, if you’re a fan of Excel, you’ll find Bill’sread.clipboard()
function in thepsych
package helpful.  You understand basic equation notation. Essentially,
Data = Model + Error
. The parameters (or variables) in our model are often denoted by β, and the error by ε.  You understand why we test hypotheses using two models. If I want to look at the impact of Ability on Grades despite differences in family SES, I have to control for SES. So our most basic model would be Grades ~ SES, and we test whether Grades ~ SES + Ability is a better model,
Now that we’re up to speed, I’ve organized this tutorial as a series of realistic situations that one might encounter. Part 1 shows how to specify your models to do hypothesis testing and does not require R. The second and third parts demonstrate how to use R to test for an interaction, curvilinear effect, and finally test whether a set of variables are significant.
Part 1: Hypothesis Testing
Using multiple regression in R involves testing specific models. Unless your models are setup correctly, the significant difference between two models may not lead to the inference you think it does! Unfortunately, the US learned this the hard way. The fear of fat, which reached its height during the 1980s and 1990s, was due to a faulty regression submitted to the FDA. The researcher tested whether levels of fat and sugar predicted rates of cardiovascular disease but never partialled out the effect of sugar (which was the real problem). Thus, due to the witch hunt against fat, food manufacturers offered lowfat varieties of everything. However, there are only so many things to get taste and calories from. When you remove the fat from a food, you have to add a lot of sucrose/fructose to make up for the lack of flavor. Thus, rates of heart disease continued to climb because of bad regression (see Lustig’s 2009 colloquium).
Consider the following situation involving judgements about political candidates. Here, I specify what the various models would be. You can plug in your own variables from your data to follow along.
Political Candidates
A political scientist is interested in factors influencing judgments of political candidates. To explore these issues, he uses some national survey data collected from a randomly drawn sample of 989 voting age citizens. Data on each of the following five variables are available from each of these survey respondents:
CAND
: The respondent’s judgment of the liberalness/conservativeness of a particular candidate . Respondents make their judgments of the candidate on a seven point scale, with 1 defined as “very conservative” and 7 defined as “very liberal.”
SELF
: Respondent’s judgment of his or her own liberalness/conservativeness on the same seven point scale as for CAND.
LIKE
: Respondent’s judgment of liking for the candidate. Judgments are on a 100 point scale with 0 indicating the respondent very much does not like the candidate and 100 indicating the respondent very much likes the candidate.
TRUST
: Respondent’s judgment of the candidate’s trustworthiness on a sevenpoint scale where 1 indicates “very untrustworthy” and 7 indicates “very trustworthy.”
Know
: A composite measure of the respondent’s political knowledge, ranging from 0 to 10, with a score of 10 indicating that the respondent is very knowledgeable about political issues and a score of 0 indicating that the respondent knows nothing about politics.
Based on these data, I conduct analyses to answer each of the following questions. For each question, I indicate the compact and augmented models that should be compared to answer the question and state the null hypothesis being tested.
In addition, indicated the degrees of freedom associated with the F* that would be used to evaluate each hypothesis. Some of these questions require we construct new variables out of the five that were measured.
<<<<<<< HEAD:source/regression/index.markdown Are judgments of liking for the candidate predictable from the respondent’s self judgment of liberalness/conservativeness?
Model C: Like_{i} = β_{0} + ε_{i}
Model A: Like_{i} = β_{0} + β_{1} Self_{i} + ε_{i}
Model C: Like_{i} = β_{0} + ε_{i}
Model A: Like_{i} = β_{0} + β_{1} Self_{i} + ε_{i}
F^{*}_{{1,987}} = H_{0}: β_{1} = 0
Tutorials:source/tutorials/regression.markdown
F^{*{1,987}} = H_{0}: β_{1} = 0
<<<<<<< HEAD:source/regression/index.markdown Are judgments of the candidate’s trustworthiness related to candidate evaluation even when controlling for the respondent’s own political persuasion?
Model C: Like_{i} = β_{0} + β_{1} Self_{i} + ε_{i}
Model A: Like_{i} = β_{0} + β_{1} Self_{i} + β_{2} Trust_{i} + ε_{i}
Model C: Like_{i} = β_{0} + β_{1} Self_{i} + ε_{i}
Model A: Like_{i} = β_{0} + β_{1} Self_{i} + β_{2} Trust_{i} + ε_{i}
F^{*{1,986}} = H_{0}: β_{2} = 0
Tutorials:source/tutorials/regression.markdown
F^{*{1,986}} = H_{0}: β_{2} = 0
<<<<<<< HEAD:source/regression/index.markdown Are judgments of how the candidate is evaluated predictable from trustworthiness judgments less strongly among politically knowledgeable respondents?
Model C: Like_{i} = β_{0} + β_{1} Trust_{i} + β_{2} Know_{i} + ε_{i}
Model A: Like_{i} = β_{0} + β_{1} Trust_{i} + β_{2} Know_{i} + β_{3} (Know_{i})(Trust_{i}) + ε_{i}
Model C: Like_{i} = β_{0} + β_{1} Trust_{i} + β_{2} Know_{i} + ε_{i}
Model A: Like_{i} = β_{0} + β_{1} Trust_{i} + β_{2} Know_{i} + β_{3} (Know_{i})(Trust_{i}) + ε_{i}
F^{*}_{{1,985}} = H_{0}: β_{3} = 0
Tutorials:source/tutorials/regression.markdown
F^{*{1,985}} = H_{0}: β_{3} = 0
<<<<<<< HEAD:source/regression/index.markdown Do those respondents whose political views are relatively extreme in either a liberal or conservative direction, like the candidate less well than those who are more moderate, even controlling for the fact that liberals prefer the candidate to those who are more conservative?
Model C: Like_{i} = β_{0} + β_{1} Self_{i} + ε_{i}
Model A: Like_{i} = β_{0} + β_{1} Self_{i} + β_{2} Self^{2i} + ε_{i}
Model C: Like_{i} = β_{0} + β_{1} Self_{i} + ε_{i}
Model A: Like_{i} = β_{0} + β_{1} Self_{i} + β_{2} Self^{2}_{i} + ε_{i}
F^{*}_{{1,986}} = H_{0}: β_{2} = 0
Tutorials:source/tutorials/regression.markdown
F^{*{1,986}} = H_{0}: β_{2} = 0
<<<<<<< HEAD:source/regression/index.markdown On average, do respondents assimilate judgments of the candidate’s position to their own position? That is, the more liberal respondents are, the more they judge the candidate to be liberal. The more conservative respondents are, the more conservative they judge the candidate to be.
Model C: Cand_{i} = β_{0} + ε_{i}
Model A: Cand_{i} = β_{0} + β_{1} Self_{i} + ε_{i}
Model C: Cand_{i} = β_{0} + ε_{i}
Model A: Cand_{i} = β_{0} + β_{1} Self_{i} + ε_{i}
F^{*}_{{1,987}} = H_{0}: β_{1} = 0
Tutorials:source/tutorials/regression.markdown
F^{*{1,987}} = H_{0}: β_{1} = 0
<<<<<<< HEAD:source/regression/index.markdown Does the extent to which respondents assimilate judgments of the candidate’s position to their own depend on the degree to which they like or dislike the candidate? Those who relatively like the candidate assimilate the candidate’s position to their own more strongly than those who like the candidate less well.
Model C: Cand_{i} = β_{0} + β_{1} Self_{i} + β_{2} Like_{i} + ε_{i}
Model A: Cand_{i} = β_{0} + β_{1} Self_{i} + β_{3} Like_{i} + β_{3} (Self_{i})(Like_{i}) + ε_{i}
F^{*{1,985}} = H_{0}: β_{3} = 0
Is liking for the candidate predictable from knowledge of the absolute difference between the respondent’s judgment of his or her own political position and his or her judgment of the candidate’s political position?
First, create a new variable: Diff_{i} = Self_{i}  Cand_{i}
Model C: Cand_{i} = β_{0} + ε_{i}
Model C: Cand_{i} = β_{0} + β_{1} Self_{i} + β_{2} Like_{i} + ε_{i}
Model A: Cand_{i} = β_{0} + β_{1} Self_{i} + β_{2} Like_{i} + β_{3} (Self_{i})(Like_{i}) + ε_{i}
F^{*{1,985}} = H_{0}: β_{3} = 0
1.7
Is liking for the candidate predictable from knowledge of the absolute difference between the respondent’s judgment of his or her own political position and his or her judgment of the candidate’s political position? First, create a new variable: Diff_{i} = Self_{i}  Cand_{i}
Model C: Cand_{i} = β_{0} + Error_{i}
Model A: Cand_{i} = β_{0} + β_{1} Diff_{i} + ε_{i}
F^{*}_{{1,987}} = H_{0}: β_{1} = 0
Tutorials:source/tutorials/regression.markdown
Model A: Cand_{i} = β_{0} + β_{1} Diff_{i} + ε_{i}
F^{*{1,987}} = H_{0}: β_{1} = 0
Part 2: Testing for Interactions
The Effect of Books and Attendance on Grades
Download this example dataset. In this section, we’ll look at the influence of Books and Attendance on one’s Grades. However, one might hypothesize that the number of books you read has a larger impact on your grade if you bother attending lecture more often. If we’re hypothesizing that people who attend class get more out of the books versus the students who don’t attend and use them to cram at the end of the quarter or semester, we need to test for an interaction of Books x Classes, over and above the main effects of Books
and Classes
on our dependant variable, Grades
.
Make sense?
1 2 3 4 5 6 7 8 9 10 11 12 13 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 

1


1 2 3 4 5 6 7 8 9 10 

In sum, we have signficant main effects of both Books (F(1,36) = 14.57, p < 0.001) and Attendance (F(1,36) = 5.22, p < .03). However, this relation between Grades and Books depends on how many classes you attend (F(1,36) = 4.43, p < .05).
Graphing the Interaction
Sometimes graphs can help us conceptualize what this interaction means. I will graph the interaction as a series of simple relationships, which we’ll derive using a bit of math. As our main equation is Gradê = (61.60 + 1.333Attend_{0}) + (4.155 x (0.735 x Attend_{0}))Books_{0}, we’ll plug in values for Attend to change the simple slope.
I will graph Grade
by the centered Books
at 1) the centered Attend
, 2) one standard deviation below from the average Attend
(i.e., 4.278 days missed), and 3) one standard deviation above from the average Attend
(4.278 days attended). The three lines correspond to the three equations below:
<<<<<<< HEAD:source/regression/index.markdown
If we plug in three representative values of Attend_{0}…
Gradê = (61.60 + 1.333Attend_{0}) + (4.155(0.735Attend_{0}))Books_{0}
Tutorials:source/tutorials/regression.markdown
Attend_{0} = 4.278
Gradê = (61.60 + 1.333 x 4.278) + (4.155 + (0.735 x 4.278))*Books_{0}
<<<<<<< HEAD:source/regression/index.markdown
Attend_{0} =0
Gradê = = (61.60 + 1.3334.278) + (4.155 + (0.7354.278))*Books_{0}
Tutorials:source/tutorials/regression.markdown
Gradê = (61.60 + 1.333 x 0) +(4.155+(0.735 x 0)) x Books_0
<<<<<<< HEAD:source/regression/index.markdown
Attend_{0} = 4.278
Gradê = (61.60 + 1.3330) +(4.155+(0.7350))*Books_{0}
Tutorials:source/tutorials/regression.markdown
Gradê = (61.60 + 1.333 x 4.278) + (4.155 + (0.735 x 4.278)) x Books_{0}
<<<<<<< HEAD:source/regression/index.markdown
Interpreting the interaction
======= Gradê = (61.60 + 1.3334.278) + (4.155 + (0.7354.278))*Books_{0}
Tutorials:source/tutorials/regression.markdown
For the people who attended fewer classes than average, the number of textbooks they read was rather weakly related with their grade. On the other hand, for the people who attended more classes than average, the number of textbooks they read had a strong positive relationship with their grade.
1 2 3 4 5 6 7 

Testing for Curvilinear Effects
Next we test if either Books
or Classes
has a curvilinear (quadratic) effect in the prediction of Grade
. To help the reader, I provide graphs of these results.
<<<<<<< HEAD:source/regression/index.markdown
Let’s do Grades ~ Attend first
Let’s do Grades ~ Attend first
Gradê = b_{0} + b_{1} Attend + b_{2} Attend^{2}
Tutorials:source/tutorials/regression.markdown
Gradê = b_{0} + b_{1} Attend + b_{2} Attend^{2}
1 2 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 

1


1 2 3 4 5 6 7 8 9 

1 2 3 

Attendance does not have a curvilinear relationship with Grades, p = n.s.
Next, we’ll test Grades ~ Books
<<<<<<< HEAD:source/regression/index.markdown
Gradê = b_{0} + b_{1} Books + b_{2} Books^{2}
Gradê = b_{0} + b_{1} Books + b_{2} Books^{2}
Tutorials:source/tutorials/regression.markdown
1 2 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 

1


1 2 3 4 5 6 7 8 9 

1 2 3 

Books do not have a curvilinear relationship with Grades, p = n.s.
The Final Countdown: Testing sets of parameters
<<<<<<< HEAD:source/regression/index.markdown Download the file here. I test to see whether the set of predictors “Burglary” and “Auto”(Theft) add to the prediction of “Murder” once “Assault” and “Robbery” are taken into account (i.e., “over and above Assault and Robbery”).
Model C: Murder̂ = b_{0} + b_{1} Assault + b_{2} Robbery
Model C: Murder̂ = b_{0} + b_{1} Assault + b_{2} Robbery
Model A: Murder̂ = b_{0} + b_{1} Burglary + b_{2} Auto + b_{3} Assault + b_{4} Robbery
Tutorials:source/tutorials/regression.markdown
Model A: Murder̂ = b_{0} + b_{1} Burglary + b_{2} Auto + b_{3} Assault + b_{4} Robbery
1 2 3 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 

1 2 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 

1


1 2 3 4 5 6 7 8 9 

The probability of the Fvalue corresponds to the probability that I would get had I calculated the proportion of reduced error (i.e., PRE) by hand:
SSC = 408.56 [error for Model C above]
SSA = 368.48 [error for Model A as above]
PRE = (SSC – SSA)/SSC
PRE = (408.56 – 368.48)/(408.56) = 0.098
Because the observed PRE is less than the critical PRE (of .125), we fail to reject the null, and conclude that burglary and auto as a set of predictors do not significantly add to the prediction of murder above and beyond assault and robbery.