The Search for Significance: A Crash Course in Statistical Significance Using ACS 2007

If we told you the American Community Survey (ACS) found that 26 percent of Hoosier women between the ages of 35 and 44 had a bachelor's degree or more compared to just 23 percent of men, how can you know if that is a real difference in educational attainment (that is, a statistically significant finding) or just a result of random sampling error? This article provides a brief tutorial on calculating statistical significance for those who want to accurately use ACS data without becoming statisticians.¹

Margins of Error

As with any survey, margins of error are critical—particularly as the size of the population in question decreases (because that typically increases the margin of error). A large margin of error makes the survey estimate less reliable, which can negatively affect your analysis and comparisons. The ACS reports the margin of error for the 90 percent confidence level. Therefore, if we look at the first row in Table 1, we can say that we're 90 percent confident that the number of Hoosier men between the ages of 25 and 34 is between 422,281 and 427,919 (that range—which is the estimate plus or minus the margin of error—is known as the confidence interval). In other words, there's only a 10 percent chance that the actual number of men in that age group falls outside of that range.

Table 1: Educational Attainment and Confidence Intervals for Indiana Men and Women, 2007

One might think that this is all the information we need to determine statistical significance: As long as the confidence intervals of two numbers don't overlap, we're good to go, right? Unfortunately, it is a bit more complex than that, and the Census Bureau discourages the use of confidence intervals alone to determine a value's statistical significance. Instead, we should calculate z-scores, which are standardized figures that allow us to make comparisons.

Three Steps to Determining Significance

The first step in determining statistical significance is to convert the margin of error into a standard error.  This calculation varies depending on if we are using numbers directly from published ACS tables or if we've done some intermediate calculations on our own, such as calculating a percentage. Since our data do not contain any derived estimates, all we need to do for this step is divide the margin of error value by 1.645.²

The second step is to calculate the z-score itself (see Table 2). If we let A represent the male estimates, use B for the female estimates and use SE(A) and SE(B) for the standard errors of those respective estimates, the formula is as follows:

Table 2: Comparing Male and Female Educational Attainment Z-Scores for Indiana, 2007

Here's an important note for Excel users: When downloading percentage data from American FactFinder, it will format the values as percents (22.8%), which Excel stores in decimal form (0.228). The margins of error, however, are stored as regular numbers (0.9). As one can imagine, mixing those two formats yields utterly meaningless z-scores. Therefore, always make sure to convert any percentages to numeric format (22.8) so they are in the same units as the margin of error before calculating the z-score.

The third step is to use the z-score to determine if the difference between the genders is significant or if random chance can explain the difference. Table 3 provides the z-score thresholds with their corresponding confidence level. Essentially, as the absolute value of the z-score becomes larger,  the more confident we are that a real difference in the estimates exists. Looking back at Table 2, we find that nearly all of the values are significant at the 99 percent level, which means that we're 99 percent sure that the difference is not due to random chance.

Table 3: Z-Scores and Levels of Significance

For more information, download the Census Bureau's instructions on statistical testing and ACS available at  www.census.gov/programs-surveys/acs/guidance.html.

Notes

Rachel Justis, Geodemographic Analyst
Indiana Business Research Center, Kelley School of Business, Indiana University