CS 100: Studio 7

Confidence Intervals and Significance Tests: Two Sides of the Same Coin

October 27 and 28, 2021

Introduction

The topic of this week’s studio is approval ratings. In the first part, you will investigate Presidential approval ratings, and in the second part, those of Congress. These ratings summarize polls conducted among U.S. adults. Your job will be to test the significance of the polls’ results, using confidence intervals and significance testing—two sides of the same coin.

Upon completion of all tasks, a TA will give you credit for today’s studio. If you do not manage to complete all the assigned work during the studio period, do not worry. You can continue to work on this assignment until Sunday, October 31, at 6 PM. Come by TA hours any time before then to show us your completed work and get credit for today’s studio.

Objectives

By the end of this studio you will be able to: Compute a confidence interval Conduct a significance test

Data

This week’s studio does not involve a data set, just the results of a few polls. The numbers used in Part 1 are taken from a 2019 Gallup presidential job approval center, located here. The numbers in Part 2 are taken from a 2013 study carried out by Public Policy Polling about Congressional approval, located here.

Part 1: Presidential Approval Ratings

As you are probably aware, Donald J. Trump was the former president of the United States. To get a sense of how he fared while he was in office, we relied on job approval polls, conducted on a sample of U.S. adults. On August 14, 2019, Gallup conducted just such a poll, and found that among 1500 U.S. adults, 41% approve of President Trump’s job performance, and 54% disapprove. To quantify how closely the poll results can be expected to mirror the opinions of the population as a whole, Gallup also reported the poll’s margin of error (MoE): 3 points. Polling agencies don’t generally report their confidence levels, only their margins of error, but with a little trial and error, you can recover a confidence interval from a MoE. In this exercise, you will verify that Gallup used a 95% confidence interval to calculate the MoE of Trump’s approval rating. What that means is, if this poll were repeated multiple times, the proportion of the population that approves of Trump would fall within the range [0.38, 0.44] 95% of the time.

If we assume that “approve” and “disapprove” are the only available answers to the poll (we apologize to those who have no opinion), each participant’s response can be viewed as a Bernoulli trial, and we can use a binomial distribution to model their collective responses. By the central limit theorem, the estimates 0.41 (41% approval rating) and 0.54 (54% disapproval rating) are normally distributed, with standard error \(\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}\), where \(\hat{p}\) is the sample proportion of successes (either 0.41 or 0.54). So the 95% confidence interval is computed as \(\hat{p} \pm 1.96 \sqrt{\frac{\hat{p} (1-\hat{p})}{n}}\).

Practice: Write a couple of lines of R code to determine the 95% confidence interval for a sample proportion. Do this by first creating variables phat and n, and assigning those variables values (e.g., 0.41 and 1500). Then input the relevant formulas, using variables instead of hard-coded values. (You will generalize 1.96 from a hard-coded constant to a variable later in this problem.)
We would like to use the code you just wrote to compute a 95% confidence interval both for Trump’s approval rating and his disapproval rating. In other words we want to run your code twice, assuming two different sample proportions. Rather than repeat code, embed your code in a function that takes as input a sample proportion and a sample size, and returns the 95% confidence interval as a vector.

Hint: The last line of your function should be something like c(lower, upper), where lower and upper are variables you define in your function (denoting the lower and upper bounds of the confidence interval the function computes).

Another Hint: As 1500 is the sample size of all polls in this question, feel free to assign n the default value of 1500 in the function header, like this: function(phat, n = 1500).

Use your function to report a confidence interval that estimates the proportion of the population that approves of Trump, and a confidence interval that estimates the proportion of the population that disapproves. Do the confidence intervals overlap? If not, we can conclude (at the 95% confidence level) that Trump’s approval and disapproval ratings are different. If they do overlap, we cannot say with confidence that his approval and disapproval ratings are different.

Do Trump’s approval and disapproval ratings overlap at the 99% confidence level? Generalize your function to take as input a third parameter, namely a confidence level \(\alpha\), and calculate the confidence interval at that level.

Hint: Use the qnorm function.

Here are the approval ratings for the past four presidents after their first 10 months in office, as reported here. Using your function, calculate the 95% confidence intervals for Obama, Bush Jr, Clinton, and Bush Sr. Do you observe any relationship between the confidence intervals around their approval ratings and their re-elections?

President	Approval	Disapproval	Sample Size
Obama	40%	53%	1500
Bush Jr	60%	37%	1500
Clinton	46%	44%	1500
Bush Sr	71%	19%	1500

Following the hypothesis testing procedure developed in the lecture slides, perform a one-sided difference of sample proportions significance test between Trump’s approval rating and Clinton’s. At a 5% significance level, are their ratings different? What is the relevant \(p\)-value? (Time permitting, calculate the \(p\)-value corresponding to some of the other former presidents’ ratings as well.)

Hint: This standard error for a difference of sample proportions between Trump and Clinton is the square root of the standard error of the proportion who support Trump, squared, plus the standard error of the proportion who support Clinton, squared. That is, if \(s_T\) is the standard error of the proportion who support Trump, and \(s_C\) is the standard error of the proportion who support Clinton, the standard error of their difference is \(\sqrt{s_T^2 + s_C^2}\).

Part 2: Congressional Approval Ratings

In 2013, Public Policy Polling conducted a survey of 830 people asking whether they preferred Congress or some other things. The results can be found here.

Please select two outcomes from this report to test. You’ll be testing whether there is a statistically significant difference between the proportion of people who prefer Congress and the proportion of people who prefer the alternative. (As above, we apologize, but you need not consider the people who cannot make up their mind.)

To perform these tests, the strategy is as follows: First, choose a confidence level (e.g., 95%). Then, compute a difference of sample proportions confidence interval at that level. Finally, observe whether the confidence interval contains 0. If it does not, you can conclude that the difference in sample proportions is statistically significant, at the assumed confidence level. (If it does, you cannot conclude anything.)

To compute a difference of sample proportions confidence interval, you first compute the difference between the proportion of people who support Congress and the proportion of people who prefer the alternative. Then find the standard error. Finally, calculate the margin of error by multiplying the standard error by the critical value corresponding to your confidence level. Put these pieces together to build your confidence interval, and then conclude whether the difference is significant or not.

Time permitting, perform two difference of sample proportions significance tests that mimic the tests you just performed with confidence intervals.

End of Studio

When you are done please call over a TA to review your work, and check you off for this studio. If you do not finish within the two hour studio period, remember to come to TA office hours to get checked off.