> =?:;<G Pbjbjَ -,]jjj@8 Lll"dd(HJJJJJJ$#%FnindpHHrTH#&ҧO:
Making Dummies Work: Comparing the Effects of Dummy Variables in Linear and Non-linear Causal Models
Andy Wroe
Department of Government
University of Essex
Wivenhoe Park
Colchester
Essex CO4-3SQ
Phone: 01206 873809
E-mail: wroeac@essex.ac.uk
February 1999
Introduction
There is an important debate within political science surrounding the use of best-practise statistical techniques. One group of scholars argues that the data should determine which techniques are used. Another group argues that we should use the techniques that best allow us to answer our questions.
Even a cursory glance at the major political-science journals reveals a contemporary obsession with voting behaviour. In most voting behaviour articles, political scientists attempt to explain why individuals or groups vote the way they do. Because most vote choices involve a Yes-No (1-0) decision, we should, according to the data-driven school of thought, employ non-linear probability techniques when investigating voting decisions. However, the question-driven scholars argue that this approach is ineffective because non-linear techniques do not allow us to investigate those questions that most interest us. This article examines the efficacy of both schools of thought in relation to a specific but not uncommon voting behaviour problem. That is, should we use linear or non-linear techniques when employing a multi-stage, recursive causal model where the dependent variable is dichotomous and many of the independent variables are dummies?
A) The Causal Model and Coding of the Variables
Because the world is complex, we develop theories about how some part of the world works, and then express these theories in models. However, we cannot rely on statistical techniques alone when constructing models because all statistical techniques make prestatistical assumptions. Of central importance to political scientists is the question of cause and effect; Davis argues that [m]ost methodologists agree that causal order is a substantive or empirical problem to be solved by our knowledge about how the real world works, [and] not by statistical gyrations.
Here I present a causal model to explain vote choice on Proposition 187, the illegal immigration initiative, which was passed by California voters in 1994. Borrowing heavily from Bartle, and Miller and Shanks, the model is a multi-stage, bloc recursive model [designed] to arrange a variety of putative causes or explanations in a single, cumulative descriptive array. Each bloc contains a number of variables whose current values were acquired at approximately the same point in time, at least in comparison to the variables in other stages. Each bloc also contains a general theme; variables are related to each other in the sense that they share the same kind of content and may influence vote through similar processes or mechanisms. The model is multi-staged because fixed or long-term variables are entered first into the model followed by less fixed or more short-term variables that are more proximate to the dependent variable, vote. The logic of this temporal sequence is that fixed characteristics, such as race or party identification, may cause less fixed characteristics, such as leadership evaluations, which may in turn influence the vote decision. The model is bloc recursive because, while variables entered in an earlier bloc may cause those entered in a later bloc, they cannot themselves be caused by variables entered afterwards. Furthermore, it is assumed that there is no causal link between variables entered in the same bloc. Thus, we rule out the possibility of reciprocal causation or feedback. Figure 1 summarises the causal model.
Figure 1. The Causal Model (!= causal direction)
!Bloc 1: Fixed Personal Characteristics
Age + Race + Gender + Place of Birth
!Bloc 2: Acquired Social Characteristics
Religious Affiliation + Level of Education + Place of Residence
!Bloc 3: Life-style Characteristics
Family Income
!Bloc 4: Party and Ideological Positions
Party Identification + Ideological Position
!Bloc 5: Personal and General Evaluations
Retrospective Personal Financial Evaluations + Current State Evaluations
!Bloc 6: Evaluations of State and National Politicians
Rating of Governor Wilson s Performance + Support for Clinton
!Bloc 7: Vote in Other Races
Vote for Governor + Vote for US Senator + Vote on Prop. 184
! Vote on Proposition 187
Of particular interest are the total and direct effects of each variable on vote choice. Miller and Shanks argue that the total effect of a variable is that suggested by its coefficient in its bloc of entry. Thus, for example, the total effect of race is captured by the race coefficients in Bloc 1 because this is the bloc in which race is entered into the model. The direct effects of all the variables are captured by their coefficients in the full model (containing all the variables in Blocs 1-7). Here, all other variables are controlled, and the coefficients therefore represent each variables direct effect on vote choice; that is, that portion of the total effect that is unmediated by any other explanatory variable in the model.
The dependent variablevote on Proposition 187is coded 1 if the respondent voted Yes to exclude illegal immigrants from receiving education, health and welfare benefits, and 0 if the respondent voted No. Most of the explanatory variables included in the causal model are nominal in nature. The value (0, 1, 2etc.) ascribed to a category functions only as a label or name (white, black, Latinoetc.), and no mathematical assumptions can be made about relationships between the categories. The explanatory variables are therefore re-coded into a series of dummy variables with mutually exclusive and exhaustive categories, scoring 1 if the attribute (e.g. Latino) is present and 0 if not (e.g. all non-Latinos). When introducing dummy variables into a regression equation, we must exclude one dummy otherwise perfect multicollinearity will result and the regression will not run. The excluded dummy serves as the reference to which the included dummies are compared and interpreted; and so long as we adjust our interpretation of regression coefficients to be consistent with the underlying measurement properties of our independent variables, we are on solid statistical ground.
A full list of dummy variables included in each stage of the model is provided in Appendix A. However, for the sake of brevity and clarity, I will focus on just one explanatory variable in this paperrace. Race, located in the first bloc of the causal model, is re-coded into a series of dummieswhite, Asian, African American, Latino, Native American and otherwith white excluded as the reference. It is hypothesised that vote on 187 will be racially bifurcated; Latinos will be least likely to vote Yes because of a widespread feeling within the Latino community that Proposition 187 was, in effect, a nativist attack on their presence in the Golden State.
It is important here to introduce some helpful terminology to ensure the following discussion is intelligible. The reference refers to the one excluded dummy variable to which the included dummy variables are compared. The constant refers to the collection of dummy variables that are excluded in any bloc of the causal model. A prior variable refers to a variable that appears in a bloc located before any variable under discussion; a parallel variable describes a variable that appears in the same bloc as the variable being discussed; and an intervening variable refers to a variable that appears in a bloc between the variable under discussion and the vote choice. Finally, probability model refers to all statistical models analyzing event probability. In other words, a probability model is a model with a 1-0 dependent variable.
B) A Non-linear Model
The dichotomous nature of the dependent variable requires a probability model to estimate the effects of the independent variables on vote choice. In this case, data-driven scholars would advise the use of a non-linear probability modellogit or probit, for example. Why should we not use a linear probability model? King puts the case forcefully:
[U]sing dichotomous dependent variables in [linear] regressioncan yield predicted probabilities greater than one or less than zero, heteroskedasticity, inefficient estimates, biased standard errors, and useless test statistics. Of more importance is that a linear model applied to these data is of the wrong functional form; in other words, it is conceptually incorrect.
Following the statistical best-practise advice of the data-driven scholars, the causal model is first estimated using the logistic-regression non-linear probability model. Each bloc of variables is entered into the logistic-regression equation in turn, and the relevant estimates for the race dummies are reported in Tables 1.1, 1.2, 2.1 and 2.2.
Table 1.1. Logistic Regression: The Importance of RaceWhite Reference
VariableEstimate TypeBivariateBloc 1Blocs
1-2Blocs
1-3Blocs
1-4Blocs
1-5Blocs
1-6Blocs
1-7AsianPred-
Exp()
P(Y=1)
MPP-.50*
.61
.48
-.12-.50*
.61
.48
-.13-.39*
.68
.39
-.10-.34
.71
.40
-.08-.35
.71
.52
-.09-.39
.68
.48
-.10-.35
.71
.37
-.09-.51*
.60
.66
-.10BlackPred-
Exp()
P(Y=1)
MPP-.62*
.54
.45
-.15-.59*
.55
.46
-.15-.74*
.48
.31
-.18-.71*
.49
.31
-.17-.06
.94
.59
-.02-.07
.93
.56
-.02.18
1.19
.50
.04.28
1.32
.81
.05LatinoPred-
Exp()
P(Y=1)
MPP-1.57*
.21
.24
-.36-1.48*
.23
.26
-.35-1.78*
.17
.14
-.35-1.80*
.17
.13
-.35-1.49*
.23
.26
-.35-1.49*
.23
.24
-.34-1.37*
.25
.18
-.28-1.28*
.28
.47
-.29NativePred-
Exp()
P(Y=1)
MPP.69*
2.00
.75
.15.73*
2.07
.76
.15.77*
2.17
.68
.19.80*
2.23
.67
.19.99*
2.69
.81
.20.89*
2.42
.77
.19.85
2.34
.66
.20.92
2.52
.89
.13OtherPred-
Exp()
P(Y=1)
MPP-.16
.85
.56
-.04-.12
.89
.58
-.03.03
1.03
.50
.01.04
1.04
.49
.01.12
1.13
.64
.03.11
1.11
.60
.02-.05
.95
.44
-.02-.04
.96
.76
.00Constant (White)Pred-
P(Y=1).41*
.60.43*
.61-.04
.49-.08
.48.44
.61.32
.58-.17
.461.16*
.76Notes: Pred- = estimated score; Exp() = odds ratio; P(Y=1) = predicted probability (of Yes vote); MPP = marginal predicted probability; * = statistical significance at .05 level.
Table 1.2. Logistic Regression: The Importance of Race Asian Reference.
VariableEstimate TypeBivariateBloc 1Blocs
1-2Blocs
1-3Blocs
1-4Blocs
1-5Blocs
1-6Blocs
1-7WhitePred-
Exp()
P(Y=1)
MPP.50*
1.65
.60
.12.50*
1.64
.61
.13.39*
1.48
.49
.10.34
1.40
.48
.08.35
1.42
.61
.09.39
1.44
.58
.10.35
1.41
.46
.09.51*
1.67
.76
.10BlackPred-
Exp()
P(Y=1)
MPP-.11
.89
.45
-.03-.10
.91
.46
-.02-.36
.70
.31
-.08-.37
.69
.31
-.09.29
1.34
.59
.07.32
1.38
.56
.08.52
1.68
.50
.13.79*
2.20
.81
.15LatinoPred-
Exp()
P(Y=1)
MPP-1.07*
.35
.24
-.24-.98*
.37
.26
-.22-1.39*
.25
.14
-.25-1.47*
.23
.13
-.27-1.14*
.32
.26
-.26-1.10*
.33
.24
-.24-1.03*
.36
.18
-.19-.77*
.46
.47
-.19NativePred-
Exp()
P(Y=1)
MPP1.19*
3.29
.75
.271.22*
3.39
.76
.281.16*
3.20
.68
.291.14*
3.13
.67
.271.34*
3.81
.81
.291.28*
3.58
.77
.291.19*
3.30
.66
.291.44*
4.20
.89
.23OtherPred-
Exp()
P(Y=1)
MPP.34
1.41
.56
.08.38
1.46
.58
.10.42
1.51
.50
.11.37
1.45
.49
.09.47
1.60
.64
.12.50
1.64
.60
.12.29
1.34
.44
.07.48
1.61
.76
.10Constant (Asian)Pred-
P(Y=1)-.09
.48-.07
.48-.43
.39-.42
.40.09
.52-.08
.48-.52
.37.65
.66Notes: Pred- = estimated score; Exp() = odds ratio; P(Y=1) = predicted probability (of Yes vote); MPP = marginal predicted probability; * = statistical significance at .05 level.
Table 2.1. Logistic Regression: Changing the Constant in Blocs 1-7 from Feinstein to Huffington Vote White Reference
VariableEstimate TypeBlocs 1-7
(Feintein Vote)Blocs 1-7
(Huffington Vote)AsianPred-
Exp()
P(Y=1)
MPP-.51*
.60
.66
-.10-.51*
.60
.83
-.06BlackPred-
Exp()
P(Y=1)
MPP.28
1.32
.81
.05.28
1.32
.92
.03LatinoPred-
Exp()
P(Y=1)
MPP-1.28*
.28
.47
-.29-1.28*
.28
.69
-.20NativePred-
Exp()
P(Y=1)
MPP.92
2.52
.89
.13.92
2.52
.95
.06OtherPred-
Exp()
P(Y=1)
MPP-.04
.96
.76
.00-.04
.96
.89
.00Constant (White)Pred-
P(Y=1)1.16*
.762.10*
.89Notes: Pred- = estimated score; Exp() = odds ratio; P(Y=1) = predicted probability (of Yes vote); MPP = marginal predicted probability; * = statistical significance at .05 level.
Table 2.2. Logistic Regression: Changing the Constant in Blocs 1-7 from Feinstein to Huffington Vote Asian Reference
VariableEstimate TypeBlocs 1-7
(Feinstein Vote)Blocs 1-7
(Huffington Vote)WhitePred-
Exp()
P(Y=1)
MPP.51*
1.67
.76
.10.51*
1.67
.89
.07BlackPred-
Exp()
P(Y=1)
MPP.79*
2.20
.81
.15.79*
2.20
.92
.10LatinoPred-
Exp()
P(Y=1)
MPP-.77*
.46
.47
-.19-.77*
.46
.69
-.13NativePred-
Exp()
P(Y=1)
MPP1.44*
4.20
.89
.231.44*
4.20
.95
.13OtherPred-
Exp()
P(Y=1)
MPP.48
1.61
.76
.10.48
1.61
.89
.07Constant (Asian)Pred-
P(Y=1).65
.661.59*
.82Notes: Pred- = estimated score; Exp() = odds ratio; P(Y=1) = predicted probability (of Yes vote); MPP = marginal predicted probability; * = statistical significance at .05 level.
i. Odds Ratios
The SPSS programme automatically generates the estimated coefficients (labelled Pred- in the tables) and the odds ratios (labelled Exp()). The logistic-regression coefficients are not directly interpretable. However, the odds ratios which are derived from the coefficients do have meaning. They are calculated by exponentiating in other words, taking the anti-log of, with base e the raw coefficients. Odds ratios give the odds of an event occurring versus not occurring, per unit change in the explanatory variable, other things being equal. In our example, therefore, they represent the odds of voting Yes compared to the reference because race is coded as a series of dummy variables.
In a simple bivariate regression, where the only explanatory variables are the family of race variables minus the reference (in this case, whitesee Column 3 of Table 1.1), Latinos have an Exp() of .21. This means that the odds of Latinos voting Yes on 187 are .21 times as high as the reference in this case whites where 1 represents the same odds. In other words, Latinos are only about one-fifth as likely to vote Yes as whites. We also see that Native Americans, with an odds ratio of 2.00, are twice as likely as whites to vote Yes. It should be noted that no odds ratios are reported for any of the reference categories. Because the reference is the base category to which all other variables are compared, it is not ascribed an odds ratio because there is nothing to compare it to. We can, however, estimate the odds of the reference category voting Yes compared to any dummy variable. To do this we simply change the sign of the dummy variable s odds ratio and exponentiate. For example, in Table 1.1, the odds of an Asian voting Yes compared to a white person is ePred- = e-.50 = .61. The odds of a white voting Yes compared to an Asian is e-Pred- = e.50 = 1.65. Column 3, Row 3 of Table 1.2 confirms this calculation.
The odds ratio has some attractive statistical properties. It is directly interpretable from the regression output. Moreover, the interpretation is clear because 1 is the threshold. Thus an odds of greater than 1 indicates an increased chance of a Yes vote compared to the reference, and an odds of less than 1 indicates a decreased chance. More importantly, however, the odds ratio controls for all other variables in the model. This can be seen clearly from Tables 2.1 and 2.2. On the first run of the full model (containing all variables in Blocs 1-7), with whites as the reference, Californians who voted for Dianne Feinstein in the Senate contest are included in the constant. In this case, as reported in Column 3 of Table 2.1, Latinos are .28 or less than a third as likely to vote Yes as whites. On the second run of the full model, retaining whites as the reference, Californians who voted for Michael Huffington in the Senate race are included in the constant. It is clear from Column 4 of Table 2.1 that odds of voting Yes are the sameLatinos are still less than a third as likely as whites to vote Yes on Proposition 187. However, it is worth repeating the point that odds ratios must be interpreted relative to the reference. If we change the race reference to Asian for example, we see from Tables 2.1 and 2.2 that Latinos odds ratios change.
It is certainly the case that odds ratios have some desirable statistical properties and are readily interpretable. Can they, however, be used to examine the impact of the explanatory variables on vote choice? As an illustrative example, we shall look at the odds of Latinos and African Americans voting Yes compared to whites when parallel and intervening variables are added to the causal model. Table 1.1 reports the relevant estimates. As previously noted, in the simple bivariate model Latinos are .21 or one-fifth as likely as whites to vote Yes. When we control for the Bloc 1 parallel variables Latinos are still about one-fifth as likely as whites to vote Yes. This decreases slightly as the Bloc 2 and 3 intervening variables are introduced into the model before increasing again with the introduction of the Bloc 4, 5, 6 and 7 variables. In general, however, the introduction of these controls has little effect on the odds of Latinos voting Yes compared to whites. This indicates that race, in this case being Latino as compared to white, did have an impact on vote choice. This tentative conclusion is reinforced by the statistical significance of the Latino Pred- estimates across all blocs of the causal model. When using dummy variables in this way, statistical significance does not mean that we can reject the null hypothesis that being Latino has no impact on voting decisions. It means, rather, that Latinos voted in a statistically significant different way from whites.
However, while Latinos were significantly less likely to vote Yes than whites, the odds ratios do not allow us to estimate the absolute impact of Latinoness on their vote choices. In other words, we cannot use the odds ratios to assess how important being Latino was for Latinos when making their voting decision on Proposition 187; we cannot do so because of the comparative nature of the statistic. This means that we cannot make inferences about the total and direct impact of the race dummy variables on vote choice. What we can say is that the odds ratios and their statistical significance across blocs show that Latinos were significantly less likely than whites to vote Yes. Moreover, this difference appears to a racial difference because the difference remains significant when controlling for all variables in the causal model; in other words, its effect is unmediated by other variables in the model.
The racial bifurcation argument is reinforced by the estimates presented in Table 1.2. Here Asian is the reference to which all odds are compared. It is clear that Latinos odds of voting Yes are only about a third those of Asians, across most blocs of the causal model, and that the differences between the two groups are statistically significant. It is worth noting that the coefficients are larger and therefore the difference in odds is greater between Latinos and whites than they are between Latinos and Asians a finding that makes intuitive sense. This reinforces the earlier point that odds ratios can only be interpreted in comparison to a reference category; they offer no absolute or objective estimate of a Yes vote, nor to what extent the explanatory variable in question affected the vote choice.
Examining the odds ratios for African Americans tell another interesting story about racial bifurcation but leads to the same conclusions about the usefulness of odds ratios as above. In a simple bivariate logistic regression, blacks are only half as likely as whites to vote Yes. This remains the case even with the introduction of parallel and intervening variables in Blocs 1, 2 and 3. The difference disappears with the entry of the first political variablesparty identification and ideological predispositionin Bloc 4. In fact, the introduction of more political evaluations in Blocs 6 and 7 changes the sign of the coefficients; now the odds ratios indicate that blacks are now more likely than whites to vote Yes controlling for these intervening variables although the difference is not statistically significant.
On the one hand, the logic of the causal model indicates that blacks were less likely than whites to vote Yes because they are more liberal on political matters. On the other hand, however, the causal model also indicates that blacks may be more liberal precisely because of their race. In sum, we can say that being black did have an effect on vote choice; however, most of this effect was indirect because it was mediated by the political variables. Thus, race played a significant and direct role for Latinos even with controls but not African Americans when we compare them to whites.
Of course, Table 1.2 tells the same story about African Americans. However, most of the coefficients are insignificant because Asian is now the reference category, and the difference between the black and Asian vote is not as great as it is between the black and white vote.
In conclusion, we see that odds ratios can be used to compare a dummy variable to its reference, and we can use the significance level to judge whether the difference is statistically relevant. We can also make some inferences about the underlying causal process by examining how the odds ratios change when additional variables are introduced into the model. However, we cannot use the odds ratios to make judgements about the absolute importance or impact of each dummy variable on vote choices. I would also argue that odds ratios, while they have appealing mathematical qualities, are not such an intuitively appealing measure of association as predicted probabilities when examining voting decisions.
ii. Predicted Probabilities
As its name suggests, a predicted probability estimates the probability of an event occurring. It is probably the most used logistic-regression estimate. The predicted probability is intuitively attractive because in non-linear probability models, unlike the odds ratio, it is bounded by the 1-0 range of the dependent variable. The formula for the probability of an event occurring is:
K
" k xk
Prob(Y=1) = ek=1 . [1]
K
" k xk
1 + ek=1
Therefore, in a simple bivariate logistic regression, with whites as the reference category (see Column 3, Table 1.1) the probability that white Californians will vote Yes is:
Probwhite(Y=1) = (e.41)/(1 + e.41) = 1.51/2.51 = .60
And the probability that Latinos will vote Yes is:
Problatino(Y=1) = (e.41 1.57)/(1 + e.41 1.57) = .31/1.31 = .24
A predicted probability has some desirable properties aside from its easy and attractive interpretation. Unlike the odds ratio statistic, a predicted probability estimate is invariant with regards the reference category. In other words, the predicted probability estimates for each dummy in a bloc will always be the same no matter which reference we employ. The probability of African Americans voting Yes in Table 1.1, where the reference is white, is identical (in any bloc of the causal model) to the probability of African Americans voting Yes in Table 1.2, where Asian is the reference. This is because the predicted-probability estimate represents an absolute rather than comparative chance of a Yes vote.
Does this attractive property aid in the estimation of the dummy variables impact across blocs? The simple answer is no. A predicted probability is invariant in respect to its own reference, yet it is not invariant in respect to the other references that make up the constant term, and therefore any comparison across blocs is meaningless because the constant term changes across blocs. For example, the predicted probability of African Americans voting Yes in Bloc 1 (either Table 1.1 or 1.2) is .46. This estimate actually represents the probability of a Yes vote for an African-American female, aged 50-59, who was born in the US but had one or both parents born outside the US. This is because these gender, age and place-of-birth characteristics constitute the constant term. The predicted probability of African Americans voting Yes when the Bloc 2 intervening variables are introduced is .31. This estimate now represents the probability of a Yes vote for an African-American female, aged 50-59, born in the US but with one or both parents born outside the US, with no religious affiliation, some college education but no degree, living in the rest of northern California. We cannot compare the Bloc 2 African American with her Bloc 1 counterpart. The differences in the constant terms across blocs make any comparison nonsensical; there is simply nothing to compare and therefore any attempt to do so is meaningless.
iii. Marginal Predicted Probabilities
If the changing constant term is responsible for predicted probabilities lack of comparative value across blocs, an intuitively attractive solution may be to remove the constant from the estimation. We can do this by using marginal predicted probabilities (MPPs). A MPP estimates the marginal effect of an independent variable on the probability of an event occurring. The standard way to calculate the marginal effect is by taking partial derivatives, using the formula:
dP = P(1 P)k
dxk
where d is the partial derivative, P is the probability that Y = 1, and 1 P is the probability that Y = 0. When using dummy variables, however, derivatives only provide an approximate measure of the marginal probability effect of a variable. A more simple and effective technique for obtaining the marginal effect of a dummy variable is computed by taking the difference of the predicted probability conditional on each of the two categories in the dummy variable. Thus we take the two predicted probabilities of the relevant dummies and subtract them. To find the MPP of a Latino voter compared to the reference white voter (see Table 1.1), we subtract the Latino probability estimate from the white probability estimate. Therefore the MPP of being Latino is .26 .61 = -.35.
The obvious objection to the MPP estimate is that, as with the odds ratio, it is only relevant vis--vis the reference. One response to this objection is that if the reference is the same across blocs then the trends of the MPP estimates have comparative value; they have value because the reference is always the same and because we have removed the confounding effects of other variables in the constant term. Thus, if we compare the MPP estimates for Latinos across blocs in Tables 1.1 and 1.2 we see that although the actual MPPs are different for Latinos in the same blocs in the two tables, the trend in the estimates is the same. From this observation, it could be argued that MPPs might provide a useful heuristic device in non-linear causal models. This is true, but only within very narrow parameters, however. The objections are outlined below.
There are two fundamental criticisms of the non-linear models MPP statistic, both of which result from the nature of the predicted-probability statistic from the MPP itself is derived. The predicted-probability estimate, unlike the odds ratio, is non-linear and non-additive. I will deal with the non-linear problem first.
In logistic regression, the probability estimate is forced within the 1-0 bounds of the dependent variable by equation [1] above. The sigmoid functional form of the logistic-regression curve means that a large coefficient at or near the (non-linear) tails of the distribution translates into a small MPP. Conversely, a smaller coefficient at or near the (linear) centre of the function may translate into a larger MPP. Columns 9 and 10 of Table 1.1 provide a good example of this. The coefficient for Native Americans in Blocs 1-6 is .85 which exponentiates to an odds ratio of 2.34. In Blocs 1-7, the coefficient increases in size to .92, and the odds ratio also increases to 2.52. However, the MPP estimate falls from .20 in Blocs 1-6 to .13 in Blocs 1-7. The answer to this counter-intuitive result lies in the predicted-probability scores. In Blocs 1-6, the probability of a Native American voting Yes is .66, and therefore falls within the linear part of the sigmoid curve. However, in Blocs 1-7, the probability of a Native American voting Yes is .89, and therefore falls within the non-linear part of the curve. The larger coefficient is squeezed while the smaller coefficient is not. Therefore, the MPP estimates are not comparable across blocs because they are non-linear.
Linked to this is the criticism that predicted-probability estimates, and therefore the MPP derivatives, are non-additive. A predicted probability estimates the chance of a Yes vote for a given individualcall her D. Consider that Ds hypothetical probability of a Yes vote in Blocs 1-2 is .45. This probability is a function of a number of factorsAfrican American, female, old, good education, and no religious affiliation, for examplethat added together produce a probability of .45. The factors of this individual are not independent of one another. If we change any of her characteristics then the probability will change, but it will not change independently of the other factors. The change depends upon the specific covariate pattern at which it is evaluated. The same change for a different person may therefore produce a different probability and MPP. Probabilities are therefore non-additive, and comparison is problematic. A good example of the non-additive nature of the non-linear MPP estimate is its sensitivity to the categories included in the constant termas Table 2.1 demonstrates. When the full model is run, with Californians who voted for Dianne Feinstein in the Senate contest included in the constant, the MPP of Latinos compared to whites voting Yes is -.29. However, if we change the constant and include Huffington rather than Feinstein voters, the MPP decreases to -.20. Thus, the effect of any variable is not independent of other variables in the model; comparison is therefore problematic because the additivity assumption is violated.
The non-additive nature of the non-linear predicted-probability and MPP estimates also suggests that there is a general flaw in the interpretation of non-linear models in political science. The difficulties in estimating the impactrather than the statistical significanceof explanatory variables on vote choice were noted above. The widely used solution to these problems is to chose an average voter, and then to test whether a change in a particular explanatory produces a different choice on the dependent variable. Despite the average label, however, the voter will have a particular set of characteristics which influence how a change in the explanatory variable is translated into a change on the dependent variable. The difference in the Latino MPP estimates between Feinstein and Huffington voters used above emphasises this point; for different voters, the same explanatory variable will have a different effect. Therefore, the non-additive nature of the predicted probabilities and the MPPs suggests that the use of the average voter is a spurious heuristic device. This criticism becomes even more damaging if the problem of non-linearity is operationalised. In other words, any comparison at the non-linear tails of the sigmoid curve exacerbates the non-comparative nature of the predicted probabilities and the MPP estimates.
iv. Summary
In this section, we have examined three estimates derived from the non-linear probability model. Of the three, only the odds ratio has the mathematical properties required to make statistically sound inferences across the causal model. However, the interpretation of the odds ratio forced us to question the utility of its estimates. The predicted-probability estimates and the marginal-predicted-probability estimates, while enjoying easy and intuitively attractive interpretations, were shown to lack comparative value. Of particular significance was the non-linear and non-additive nature of the predicted probabilities.
There is one potential solution to the non-linear problem, however. If the probabilities fall within the linear section of the sigmoid curve (between .25 and .75), the non-linear problem is irrelevant. However, if this is the case, why use a non-linear probability model? As long as the predicted probabilities fall within the 1-0 range of the dependent variable, it would appear that linear probability models suffice.
Moreover, there is no solution to the non-additivity problem of the non-linear model, while the linear model assumes additivity. The linear probability model also offers additional attractions. The coefficient is directly interpretable from the regression output. It represents the marginal effect of the explanatory variable on the dependent variable, with all other variables controlled for. Thus, unlike in the logistic function, the linear models probabilities are linear and additive, and are directly interpretable from the statistical packages output.
Why, the question-driven scholars would argue, use non-linear functions when linear alternatives offer more for a lower cost? If the Ford Escort gets us where we want to go, why drive the Ferrari? Should we abandon our Ferrari for the Escort when we know the Escort can get us there? The question-driven scholars surely would answer Yes. The remainder of this paper will examine the causal model utilising a linear OLS regression to estimate the impact of the explanatory dummy variables on vote choice.
C) A Linear Model
Considering the strength of feeling among the data-driven scholars surrounding the use of best-practise statistical techniques, it is striking that the results of the OLS regression are very similar to those produced by the logistic regression. For example, if we compare Tables 1.1 and 3.1 it is clear that the bivariate predicted probabilities (P[Y=1]) are identical for all the dummy variables in both the linear and non-linear models. When the Bloc 1 controls are introduced, the difference between the linear and non-linear probabilities is never greater than 1 percentage point. As more controls are introduced, the differences between the two models increase slightly. However, the predicted-probability scores rarely differ by more than 5 percentage points. Furthermore, despite protestations that dichotomous dependent variables produce inefficient OLS estimates and useless confidence tests, we see that the statistical significance of the race dummy variables in the linear model almost perfectly mirrors the non-linear model across all blocs of the causal model. The only differences are that the Asian dummy is attributed significance in Bloc 5 of the OLS model but not in the logistic regression, and the Native American dummy is given significance in Bloc 6 of the linear model but not in the non-linear model. All the other significances from Tables 1.1 and 3.1 are identical. The similarities between the size of the predicted probabilities and the statistical significance of the coefficients suggest that both models will tell equally good stories about race and Proposition 187.
Table 3.1. Linear Regression: The Importance of Race White Reference
VariableEstimate TypeBivariateBloc 1Blocs 1-2Blocs 1-3Blocs 1-4Blocs 1-5Blocs 1-6Blocs 1-7AsianPred-/MPP
P(Y=1)-.12*
.48-.12*
.48-.09*
.40-.08
.41-.07
.52-.08*
.49-.07
.38-.08*
.62BlackPred-/MPP
P(Y=1)-.15*
.45-.15*
.45-.17*
.32-.16*
.33-.01
.58-.02
.55.03
.48.04
.74LatinoPred-/MPP
P(Y=1)-.36*
.24-.34*
.26-.38*
.11-.38*
.11-.28*
.31-.28*
.29-.24*
.21-.20*
.50NativePred-/MPP
P(Y=1).15*
.75.15*
.75.14*
.63.15*
.64.16*
.75.15*
.72.14*
.59.13
.83OtherPred-/MPP
P(Y=1)-.04
.56-.03
.57.01
.50.01
.50.02
.61.02
.59-.01
.44.00
.70Constant
(White)Pred-
[=P(Y=1)]
.60*
.60*
.49*
.49*
.59*
.57*
.45*
.70*Notes: Pred- = estimated ; MPP = marginal predicted probability; P(Y=1) = probability that Y =1; * = statistical significance at .05 level.
Table 3.2. Linear Regression: The Importance of RaceAsian Reference
VariableEstimate TypeBivariateBloc 1Blocs 1-2Blocs 1-3Blocs 1-4Blocs 1-5Blocs 1-6Blocs 1-7WhitePred-/MPP
P(Y=1).12*
.60.12*
.60.09*
.49.08
.49.07
.59.08*
.57.07
.46.08*
.70BlackPred-/MPP
P(Y=1)-.03
.45-.02
.46-.08
.32-.08
.33.06
.58.06
.55.10*
.49.13*
.75LatinoPred-/MPP
P(Y=1)-.24*
.24-.22*
.26-.29*
.11-.31*
.10-.21*
.31-.20*
.29-.17*
.22-.11*
.51NativePred-/MPP
P(Y=1).27*
.75.28*
.76.23*
.63.22*
.63.23*
.75.23*
.72.20*
.59.21*
.83OtherPred-/MPP
P(Y=1).09
.57.09
.57.09
.49.08
.49.09
.61.10
.59-.06
.45.08
.70Constant
(Asian)Pred-
[=P(Y=1)]
.48*
.48*
.40*
.41*
.52*
.49*
.39*
.62*Notes: Pred- = estimated ; MPP = marginal predicted probability; P(Y=1) = probability that Y =1; * = statistical significance at .05 level.
Table 4.1. Linear Regression: Change Constant from Feinstein to Huffington White Reference
VariableEstimate TypeBlocs 1-7
(Feinstein Vote)Blocs 1-7
(Huffington Vote)AsianPred-/MPP
P(Y=1)-.08*
.62-.08*
.79BlackPred-/MPP
P(Y=1).04
.74.04
.91LatinoPred-/MPP
P(Y=1)-.20*
.50-.20*
.67NativePred-/MPP
P(Y=1).13
.83.13
1.00OtherPred-/MPP
P(Y=1).00
.70.00
.87Constant
(White)Pred-
[=P(Y=1)]
.70*
.87*Notes: Pred- = estimated ; MPP = marginal predicted probability; P(Y=1) = probability that Y=1; * = statistical significance at .05 level.
Table 4.2. Linear Regression: Change Constant from Feinstein to Huffington Asian Reference
VariableEstimate TypeBlocs 1-7
(Feinstein Vote)Blocs 1-7
(Huffington Vote)WhitePred-/MPP
P(Y=1).08*
.70.08*
.86BlackPred-/MPP
P(Y=1).13*
.75.13*
.91LatinoPred-/MPP
P(Y=1)-.11*
.51-.11*
.67NativePred-/MPP
P(Y=1).21*
.83.21*
.99OtherPred-/MPP
P(Y=1).08
.70.08
.86Constant
(Asian)Pred-
[=P(Y=1)]
.62*
.78*Notes: Pred- = estimated ; MPP = marginal predicted probability; P(Y=1) = probability that Y=1; * = statistical significance at .05 level.
i. Predicted Probabilities
As with the non-linear model, we should not use the predicted probabilities to inform our causal story. Unlike the logistic-regression probabilities, the OLS probabilities are linear and additive; however, they are extremely sensitive to the constant. The difference between the Bloc 6 and Bloc 7 probabilities for all dummy variables in Table 3.1 emphasises how sensitive probabilities are vis--vis the constant term. The inclusion of the Bloc 7 controls increases the probabilities substantially.
However, unlike the non-linear model, the linear models marginal predicted probabilities allow us to make useful causal inferences.
ii. Marginal Predicted Probabilities
It was noted above that the non-linear and non-additive nature of the logistic-regression MPP estimates makes comparison across blocs problematic. In OLS regression, however, the MPP estimates are linear and additive, and, because the confounding effect of the constant is removed, the MPP estimates allow us to compare across blocs. Why are the OLS MPP estimates linear and additive? In the linear probability model, the MPP estimates do not have to be constructed from the predicted-probability estimates, as in the non-linear model. Rather, in the linear model, the coefficients are directly interpretable as each dummy variable s marginal effect on vote choice. Because the linear model s coefficients are by definition linear and additive, the MPP estimates must be linear and additive by association. Tables 4.1 and 4.2 illustrate this conclusion. If we change the constant to include Californians who voted for Huffington rather than Feinstein in the Senate race, the MPP estimates do not change. This is in contrast to the non-linear model (see Tables 2.1 and 2.2) where the MPP estimates change if the constant is altered. Of course, as in the non-linear model, the linear models MPP estimates must always be interpreted vis--vis the reference and will change if the reference is changed (compare Tables 3.1 and 3.2). However, so long as the reference is held constant across the blocs of the causal model, we can use linear regressions MPP estimates to make causal inferences about the effects of the dummy variables on vote choice.
In many ways, the linear models MPP estimates are analogous in character to the non-linear models odds ratios. Both are additive, and therefore both have comparative value across blocs because they are not dependent on the variables included in the constant term. Moreover, both must be interpreted vis--vis the reference category. I would argue, however, that the OLS MPP estimates are more intuitively appealing than the logistic-regression odds ratios especially when the dependent variable is a 1-0 dichotomous variable representing a vote choice. The MPP estimates can be interpreted as the percentage-point difference that a dummy variable has on vote choice vis--vis the reference. Odds ratios, however, are not bounded by the dependent variable and can take on any value from 0 to infinity. Furthermore, scores ranging from 0 to 1 (the threshold) are linear and scores above one are multiplicative. Without doubt, some people who find the concept of odds and odds ratios intuitively appealing. However, whichever method and estimate the researcher uses, it is incumbent upon the researcher to convey the most intelligible story to the reader.
Before we move onto joint hypotheses testing and predictive efficacy tests, it is necessary to make one final note about the efficacy of MPP estimates in linear models. This section has shown that MPP estimates in linear causal models have great potential when we are interested in questions of causality. However, researchers may be reluctant to employ linear models when the dependent variable is discrete. While this may be the case, researchers should consider utilising dummy variables MPP estimates in linear models when the dependent variable is of an interval-level nature. In a classic linear regression, with dummy variables and a continuous dependent variable, the MPP statistic offers an important heuristic device in the examination of questions of cause and effect. Moreover, in this situation, the criticisms that King noted above do not apply. Thus, we have a best-practise heuristic to examine the causal effect of dummy variables.
D) Joint Hypotheses Testing and Predictive Efficacy in Linear and Non-linear Causal Models
Estimating the effect of explanatory dummy variables in both linear and non-linear models is problematic. However, it is possible in both models to make some inferences about the statistical significance and impact of a group or groups of dummy variables. We can, for example, keep race coded as a series of dummies and still make inferences about the family of race dummy variables by using joint-hypotheses tests and by examining the predictive efficacy of race.
i. Joint-Hypotheses Tests
We can use joint-hypotheses tests across blocs of a causal model to test the null hypothesis on a group or family of dummy variables (e.g. racecomprising white, Asian, Latino, African American, Native American and others). This is important because, while it may well be the case that one dummy (e.g. Latino) is significantly different from its reference (e.g. white), the family of dummy variables (e.g. race) may well make no significant impact on vote choices. Joint-hypotheses tests therefore provide a very convenient measure of the significance of a nominal-level variable that has been re-coded into a series of dummy variables. We can compare the significance of a group of dummy variables across blocs. This, in turn, enables us to estimate how the intervening variables in the causal model affect the significance of a group of variables.
There are two ways to test the statistical significance of a group of dummy variables in logistic regression. The simpler test involves two runs of the model and a comparison of chi-square values. This is known as the change in chi-square test ("C2). The unconstrained model that is, the model including the group of target variables is run first. The constrained model with the target variables excluded is run second. The chi-square value of the constrained model is subtracted from the chi-square value of the unconstrained model. This is simply written as:
"C2 = unconstrained C2 constrained C2 [2]
The resulting "C2 value is then tested for statistical significance, with the degrees of freedom counted as the number of variables excluded in the constrained model. If the chi-square value is statistically significant at some arbitrary, predefined level, we can reject the null hypothesis that the group of dummy variables has no impact on vote choice. The results for race, reported below in Table 5.1, allow us to reject the null hypothesis and confirm that it did, as expected, have a significant impact on Californians voting decisions across all blocs of the causal model.
The calculation of a joint-hypotheses test in linear regression is similar but slightly more complex than in the logistic regression case. We use an incremental F test rather than a "C2 test. The formula for the incremental F test is:
F = (unconstrained R2 constrained R2)/(unconstrained df constrained df) [3]
(1 unconstrained R2)/(unconstrained N unconstrained df 1)
where R2 is the coefficient of determination (the percentage of the dependent variables variance explained by the model), df is degrees of freedom (number of variables), and N is the number of cases. If the incremental F is significant at some arbitrary and predefined level, where the degrees of freedom equals the number of variables excluded in the constrained model, we can reject the null hypothesis for the constrained (excluded) variables. We can use the incremental F test in exactly the same way as the "C2 test described above. The statistical significance of race (presented in Table 5.2), measured using the incremental F test, mirror those of the "C2 test across all blocs of the causal model. We can therefore reject the null hypothesis for race using either linear or non-linear models.
Table 5.1. Logistic Regression: The Statistical Significance of Race
ModelStatisticBivariateBloc 1Blocs 1-2Blocs 1-3UnconstrainedC-square (df)225.3 (5)262.4 (14)*666.6 (32)*667.6 (36)*ConstrainedC-square (df)-91.0 (9)*466.4 (27)*475.0 (31)*Difference"C2 (df)-171.4 (5)*200.2 (5)*192.6 (5)*Blocs 1-4Blocs 1-5Blocs 1-6Blocs 1-7UnconstrainedC-square (df)1243.0 (41)*1152.7 (44)*1386.1 (49)*1578.2 (52)*ConstrainedC-square (df)1142.7 (36)*1055.8 (39)*1314.7 (44)*1531.2 (47)*Difference"C2 (df)100.3 (5)*96.9 (5)*71.4 (5)*47.0 (5)*Notes: dependent variable = Proposition 187; unconstrained models contain dummy race variables; constrained models exclude dummy race variables; difference = unconstrained constrained model; - = not applicable; * = statistical significance at .05 level; (df) = degrees of freedom.
Table 5.2. Linear Regression: The Statistical Significance of Race
StatisticBloc 1Blocs 1-2Blocs 1-3Blocs 1-4Blocs 1-5Blocs 1-6Blocs 1-7Incremental F for Race35.131*42.051*40.635*21.857*21.461*16.761*11.164*Notes: * = statistical significance d" .05
ii. Predictive Efficacy
Predictive efficacy& refers to the ability of one s model to generate accurate predictions of a case s status on the dependent variable. Unfortunately, there is no one generally accepted measure of predictive efficacy for non-linear models. The first one that I propose here is based on the Classification Table (CT) produced automatically by the SPSS logistic-regression command. The CT compares the cases actual score on the dependent variable with the score predicted by the model, and reports the percentage of cases correctly predicted. We construct the change in predictive efficacy score for a group of variables in a similar way to the joint-hypothesis tests. In the first instance, the unconstrained model is run with the target variables included. The constrained model is then run, with the target variables excluded. Subtracting the constrained models percent of cases correctly predicted from the unconstrained models gives a percentage-point difference in the target (constrained) variables contribution to the predictive accuracy of the model at which ever causal stage the calculation is made. The non-linear models predictive efficacy scores are reported below in Table 6.1.
Table 6.1. Logistic Regression: How Race Improves the Predictive Efficacy of the ModelUsing SPSS Classification Table.
ModelBivariateBloc 1Blocs 1-2Blocs 1-3Unconstrained60.98%61.2467.1667.20Constrained-57.7564.5164.29Difference-3.492.652.91Blocs 1-4Blocs 1-5Blocs 1-6Blocs 1-7Unconstrained73.5373.3476.6080.56Constrained72.4572.3876.0880.04Difference1.08.96.52.52Notes: dependent variable = Proposition 187; unconstrained models contain dummy race variables; constrained models exclude dummy race variables; difference = unconstrained constrained model; - = not applicable.
Table 6.2. Logistic Regression: How Race Improves the Predictive Efficacy of the ModelUsing Cox and Snells Pseudo R2.
ModelBivariateBloc 1Blocs 1-2Blocs 1-3Unconstrained.046.055.136.141Constrained-.019.097.102Difference-.036.039.039Blocs 1-4Blocs 1-5Blocs 1-6Blocs 1-7Unconstrained.252.250.3031.00Constrained.233.231.2891.00Difference.019.019.014.00Notes: dependent variable = Proposition 187; unconstrained models contain dummy race variables; constrained models exclude dummy race variables; difference = unconstrained constrained model; - = not applicable
Table 6.3. Linear Regression: How Race Improves the Predictive Efficacy of the ModelUsing R2
ModelBivariateBloc 1Blocs 1-2Blocs 1-3Unconstrained.047.055.138.143Constrained-.019.098.103Difference-.036.040.040Blocs 1-4Blocs 1-5Blocs 1-6Blocs 1-7Unconstrained.263.261.322.416Constrained.244.241.307.406Difference.019.020.015.010Notes: dependent variable = Proposition 187; unconstrained models contain dummy race variables; constrained models exclude dummy race variables; difference = unconstrained constrained model; - = not applicable.
Table 6.1 shows that in the first stage of the causal model race increases the predictive efficacy of the model by nearly three-and-a-half percentage points. Following the logic of Miller and Shanks, we can interpret this score as the apparent total effect of race on vote choice. As the Bloc 2 and 3 intervening variables are introduced, we see that the impact of race remains quite stable at around three percentage points. However, the introduction of the Bloc 4 political variables reduces the improvement that race provides to the predictive accuracy of the model. In other words, party identification and ideological position mediate the impact of race. We also see from Bloc 6 that the impact of race is further mitigated by evaluations of state and national politicians. In the full model, race improves the predictive efficacy of the model by just half of one percentage point. We can interpret this figure as the direct impact of race on vote choice, and, although the impact may seem small, we see from Bloc 7 of Table 5.1 that it is a statistically significant impact. The logic of the causal order suggests that race may be responsible for individuals positions on other non-political variables (in Blocs 2 and 3) and on the political variables (in Blocs 4 to 7). We can conclude that race was a significant total and direct determinant of vote choice, although most of its effect was indirect because it is mediated by other variables in the causal model.
A similar test for OLS regression employs the coefficient of determination, R2. The results of this test, reported in Table 6.3, are similar to those reported in Table 6.1 but different enough to warrant attention. To make the estimates from Tables 6.1 and 6.3 comparable, we must first multiply the linear-regression estimates by 100. When this is done, it is clear that the non-linear CT impacts are around double those derived from the linear R2 measure. They are different because they measure different things. If we employ a non-linear pseudo R2 measure in place of the CT measure, we see that the difference between the non-linear and linear models disappears. This reinforces the general point of this paper that the two models produce estimates and statistical significances that are very similar. There is very little to chose between the two, despite the protestations of the data-driven scholars.
Unfortunately, there is no one generally accepted or correct measure of predictive efficacy. However, I agree with Demariss conclusion:
In sum, it may not be prudent to rely on only one measure for assessing predictive efficacy in logistic regressionparticularly in view of the lack of consensus on which measure is most appropriate. Perhaps the best strategy is to report more than one measure for any given analysis. If the model has predictive power, this should be reflected in some degree by all of the measure.
Conclusion
The use of dummy variables in multivariate analyses is problematic whether we employ linear or non-linear models. The problems are exacerbated when examining questions of causality.
In both linear and non-linear models, predicted probabilities are too sensitive to the constant term to have value across blocs of a causal model. We can remove the confounding effects of the constant by examining the MPP scores. However, in non-linear models the MPPs are non-linear and non-additive which makes interpretation and comparison problematic. We should therefore be extremely cautious when using probabilities in non-linear models, whether of a causal or non-causal nature. As this paper has shown, the use of the average-voter method to test the impact of a variable on vote choice in non-linear models is highly suspect because of the non-linear and non-additive nature of these models. The odds ratio must therefore be employed in non-linear models, even though the interpretation is not as intuitively attractive as the MPP estimate.
In the linear model, the MPP estimate bypasses the problems of non-linearity and non-additivity. Moreover, I would argue it offers an intuitively attractive interpretation. It must be stressed, however, that in both linear and non-linear models we can only interpret the dummy vis--vis its reference. We therefore cannot make inferences about the impact of a single dummy variable (e.g. Latino) in isolation. Estimating the impact and significance of a group or family of dummy variables (e.g. race) is possible, however, if we employ joint hypothesis and predictive accuracy testing. This allows us to examine the overall effect of a group of dummy variables on vote choice as intervening variables are entered into the causal model.
Should we use linear or non-linear probability models when the dependent variable is dichotomous and the explanatory variables are dummies? As this paper has shown, there is no definitive answer to this question because there are problems and advantages with both models. I would argue, however, that while we should always try to use best-practise statistical techniques, we should also be guided by the questions that we want to answer. We should strive to be as scientific as possible, but we should not lose sight of why we want to be scientific; of course, we want the best possible estimates and therefore answers, but if we cannot answer our questions, even the best estimates are useless.
Appendix A: Survey Questions and Dummy Variables in the Causal Model
Bloc 1: Fixed Personal Characteristics
Age:
How old are you? (1) 18-24; (2) 25-29; (3) 30-39; (4) 40-49; (5) 50-59; (6) 60-64; (7) 65+
Re-coded into seven dummies; Respondents aged 50-59 excluded as reference
Race:
What is your racial or ethnic background? (1) White (Non-Hispanic); (2) Black; (3) Latino; (4) Asian; (5) American Indian; (6) Other
Re-coded into seven dummies; Whites excluded as reference
Gender:
Are you: (1) Male; (2) Female?
Re-coded into two dummies; Females excluded as reference
Place of Birth:
Which one of the following describes you? (1) I was born outside of the US; (2) I was born in the US; one/both of my parents werent; (3) I was born in the US and so were both my parents
Re-coded into three dummies; Respondents born in the US, with one/both parents werent excluded as reference
Bloc 2: Acquired Social Characteristics
Religious Affiliation:
What religion do you consider yourself? (1) Protestant; (2) Roman Catholic; (3) Other Christian; (4) Jewish; (5) Other religion; (6) No religion
Re-coded into six dummies; Respondents with no religion excluded as reference
Level of Education:
What was the last grade of regular school that you completed? (1) I did not graduate from high school; (2) I have a high school diploma; (3) I attended college but did not graduate; (4) I have a college degree; (5) I studied at graduate school; (6) I have a graduate degree
Re-coded into six dummies; Respondents who attended college but did not graduate excluded as reference
Place of Residence:
No question. Residence identified by pollster depending on where exit poll taken. The categories are: (1) Los Angeles County (excluding San Fernando Valley); (2) Orange County; (3) San Diego; (4) Kern County; (5) Rest of southern California; (6) Bay Area; (7) Central Valley; (8) Rest of northern California; (9) San Fernando Valley
Re-coded into nine dummies; Respondents residing in rest of northern California excluded as reference
Bloc 3: Life-style Characteristics
Family Income:
If you added together the yearly income of all the members of your family living at home last year, would the total be: (1) Less than $20,000; (2) $20,000 to under $40,000; (3) $40,000 to under $60,000; (4) $60,000 to under $75,000; (5) $75,000 or more?
Re-coded into five dummies; Respondents with family income between $40,000 and $60,000 excluded as reference
Bloc 4: Party and Ideological Positions
Party Identification:
Regardless of registration, do you generally consider yourself a: (1) Democrat; (2) Independent; (3) Republican; (4) Another party member?
Re-coded into four dummies; Independents excluded as the reference
Ideological Position:
In most political matters, do you consider yourself: (1) Liberal; (2) Middle-of-the-road; (3) Conservative?
Re-coded into three dummies; Respondents who considered themselves as middle-of-the-road excluded as the reference
Bloc 5: Personal and General Evaluations
Retrospective Personal Financial Evaluations:
Financially speaking, are you better off, worse off or about the same as you were four years ago? (1) Better off; (2) Worse off; (3) About the same
Re-coded into three dummies; Respondents who considered themselves about the same excluded as reference
Current State Evaluations:
Do you think things in California are generally: (1) Going in the right direction; (2) Seriously off on the wrong track?
Re-coded into two dummies; Respondents who thought California was seriously off on the wrong track excluded as reference
Bloc 6: Evaluations of State and National Politicians
Rating of Governor Wilsons Performance:
Do you approve or disapprove of the way Pete Wilson is handling his job as governor? (1) Approve; (2) Disapprove
Re-coded into two dummies; Respondents who disapproved excluded as reference
Support for Clinton:
If Bill Clinton were running for reelection today, how would you vote? (1) Definitely for him; (2) Probably for him; (3) Probably against him; (4) Definitely against him; (5) Not sure
Re-coded into five dummies; Respondents not sure excluded as reference
Bloc 7: Vote in Other Races
Vote for Governor:
In the election for governor, did you just vote for: (1) Kathleen Brown, the Democrat; (2) Pete Wilson, the Republican; (3) Some other candidate? (4) I did not vote for Governor
Re-coded into two dummies. GovBrown = score 1 if the respondent voted for Brown and 0 if voted for Wilson (Other categories re-coded as missing data). GovWilson = score 1 if voted for Wilson and 0 if voted for Brown (Other categories re-coded as missing data). Respondents who voted for Wilson excluded as the reference
Vote US Senator:
In the election for US Senator, did you just vote for: (1) Dianne Feinstein, the Democrat; (2) Michael Huffington, the Republican; (3) Some other candidate? (4) I did not vote for US Senator
Re-coded into two dummies. FeinVote = score 1 if respondent voted for Feinstein and 0 if voted for Huffington (Other categories excluded as missing data). HuffVote = score 1 if voted for Huffington and 0 if voted for Brown (Other categories excluded as missing data). Respondents who voted for Feinstein excluded as reference
Vote on Proposition 184:
How did you just vote on Proposition 184 (Three Strikes sentencing)? (1) Voted for; (2) Voted against; (3) Didnt vote
Re-coded into one dummy. P184_no = score 1 if respondent voted against and 0 if voted for (Other categories re-coded as missing data). Respondents who voted for excluded as the reference
I would like to thank John Bartle, Antony Lyons, Eric Tanenbaum, Hugh Ward and John Upton for their comments on this paper.
All statistical texts that I am aware of recommend the use of either logit or probit when the dependent variable is dichotomous. See, for example, Aldrich and Nelson, Linear Probability, Logit and Probit Models, Sage University Paper series on Quantitative Applications in the Social Sciences, 07-045 (Beverly Hills, California: Sage Publications, 1994), passim; Kennedy, Peter, A Guide to Econometrics, third edition (Cambridge, Massachusetts, 1992), pp. 229-246; Liao, Tim Futing, Interpreting Probability Models: Logit, Probit, and Other Generalized Linear Models, Sage University Paper series on Quantitative Applications in the Social Sciences, 07-101, (Thousand Oaks, California: Sage Publications, 1994), passim; Demaris, Alfred, Logit Modeling: Practical Applications, Sage University Paper series on Quantitative Applications in the Social Sciences, 07-086, (Newbury Park, California: Sage Publications, 1992), passim; Pindyck, Robert S., and Daniel L. Rubinfeld, Econometric Models and Economic Forecasts, third edition, (New York: McGraw-Hill, Inc., 1991), pp. 248-286; King, Gary, How Not to Lie With Statistics: Avoiding Common Mistakes in Quantitative Political Science, American Journal of Political Science, August 1986, pp. 666-687.
Two good examples of what can be achieved by not using best-practise statistical techniques are Miller, Warren E., and J. Merrill Shanks, The New American Voter, (Cambridge, Massachusetts: Harvard University Press, 1996), and Bartle, John, Models of the 1992 British General Election, Unpublished Ph.D. Thesis (Essex, England: University of Essex).
Sullivan, John L., in Davis, James A., The Logic of Causal Order, Sage University Paper series on Quantitative Applications in the Social Sciences, Vol. 07-055, (Beverly Hills, California: Sage Publications, 1985), p. 5. For a concise review of the causal-modelling literature, see Bartle, Models of the 1992 British General Election, pp. 145-148.
Davis, The Logic of Causal Order, pp. 11.
The causal model employed here is based upon those used by Miller and Shanks, The New American Voter, and Bartle, Models of the 1992 British General Election.
Miller and Shanks, The New American Voter, pp. xi.
Miller and Shanks, The New American Voter, p. 15.
Miller and Shanks, The New American Voter, p. 189.
Of course, the assumption of unidirectional causality is probably unrealistic in practise. The assumption is made for the sake of simplicity and transparency. Bartle, however, has made the intriguing point that perhaps all causation in the real world is unidirectional, and that non-recursive modelling actually picks up flaws in the data rather than any feedback. Thus, he suggests that the solution is better data-collection techniques, not statistical gyrations (Private correspondence).
The first three bloc titles where first utilised by Bartle, Models of the 1992 British General Election.
Miller and Shanks, and Bartle attach a series of caveats to their interpretation of the ATE coefficients. For a detailed discussion of this point, see The New American Voter, pp. 551-552, and Models of the 1992 British General Election, pp. 134-137.
All the data in this paper are from the Los Angeles Times 1994 California General Election Exit Poll.
See Hardy, Melissa A., Regression With Dummy Variables, Sage University Paper Series on Quantitative Applications in the Social Sciences, 07-093 (Newbury Park, California: Sage Publications, 1993), pp. 11-12.
In some examples, Asian is excluded to make a statistical or methodological point.
See Davis, The Logic of Causal Order, pp. 16-22.
Liao, Interpreting Probability Models, pp. 1-2. See, also, Aldrich and Nelson, Linear Probability, Logit, and Probit Models, pp. 12-15.
See, infra text, footnote 1.
King, How Not to Lie With Statistics, p. 681.
Liao, Interpreting Probability Models, p. 16.
For a fuller exposition of odds ratios desirable properties, see Liao, Interpreting Probability Models, pp. 14-15.
We could, for example, use the history of the New Deal coalition to support this argument.
The African-American case is a good example of why political scientists should use multivariate causal models. If all the variables were entered into the logistic regression in one run, all the interesting causal inferences would be lost. Without causal modelling, all we could say is that blacks are more likely to vote Yes than whites when we control for other factors. Yet we know from the exit poll results that blacks were less likely to vote Yes than whites. We cannot satisfactorily account this change (Blocs 1-7) unless causal modelling techniques are employed.
As noted earlier, in the linear probability models, the predicted probabilities are not bounded by the 1-0 nature of the dependent variable.
For the derivation of this formula, see Liao, Interpreting Probability Models, p. 12.
Liao, Interpreting Probability Models, p. 19.
Demaris, Logit Modeling, pp. 48-49.
Demaris, Logit Modeling, p. 49.
For the more complex test, involving just one run of the model, see Aldrich and Nelson, Linear Probability, Logit, and Probit Models, pp. 60-61.
See Demaris, Logit Modeling, pp. 52, 56-60; and Aldrich and Nelson, Linear Probability, Logit, and Probit Models, pp. 59-60.
For the original form from which formula [3] is derived, see Hardy, Regression With Dummy Variables, pp. 24-25.
Demaris, Logit Modeling, p. 26.
For a review of some of the different options, see Demaris, Logit Modeling, pp. 53-60; and Aldrich and Nelson, Linear Probability, Logit, and Probit Models, pp. 56-59.
Aldrich and Nelson, Linear Probability, Logit, and Probit Models, pp. 57-58, suggest their own pseudo R2 measure. The formula is c/(N + c) where c is the chi-square statistic and N is the sample size. Using this formula for the full model (Blocs 1-7) produces an unconstrained R2 of .3227 and a constrained R2 of .3156. This yields a "R2 statistic of .0071 which is close to that of the OLS regression. Demaris reports (Logit Modeling, p. 56) that Aldrich and Nelson s measure performs best across a range of different models.
Demaris, Logit Modeling, p. 56.
PAGE 1
PAGE 2
il ,-._`
K
fg#
$
qr$&tuBC!!""T&U&&&''%(&())++`,h,V-X-d-..>0L01123356777CJ5CJmHCJ,mHH*j0J6U6
j0JU0Jj0JOJQJUOJQJCJCJMijklmw
!$
!$$dhx$dhxijklmw
./b
K
L
^`bjD^D2l v!#''''x()I+K+++++++++++++++
[
./b
K
L
^dhxdhx
!
!
!^`bjD^D2l v!#''dhdhx
dhxdhdhx'''x()I+K+++++++++++++++++++dh$$dhdhxd+++++++
,,&,.,:,B,J,T,`,h,p,z,,,,,,,,,,,,,,,----&-0-<-D-L-V-X-d-r---------------...$...8.@.H.R.\.d.l.v.~..............
///$/2/:/B/L/Z/b/j/
a+++
,,&,.,:,B,J,T,`,vppppphhhhh$dh$dh$$$l
t
&!`,h,p,z,,,,,,,,,,,,,,,----&-0-<-D-L-V-$dh$$dh$V-X-d-r----------vppppphhhhhh$dh$dh$$$l
t
&!------...$...8.@.H.R.\.d.l.v.~........$dh$.......
///$/2/:/B/L/Z/b/j/t//////$dh$dh$$$$l
t
&!j/t//////////////0
00"0*020<0>0L0Z0h0v0~000000000000000111"1*121<1F1N1V1^1h1p1x111111111111111222"2*242<2D2L2V2^2f2n2x22222222222222223
c/////////0
00"0*020<0>0L0Z0h0v0~000Ҽdh$$$$l
t
&!$dh$0000000000000111"1*121<1F1N1V1^1h1p1x111$dh$1111111111111222"2*242<2D2L2V2^2Ҭdh$$$$l
t
&!$dh$^2f2n2x22222222222222223"303:3B3Ҝdh$$$$l
t
&!$dh$3"303:3B3L3T3^3f3p3x333333333305555666&626:6F6N6Z6b6n6v666666666666667
777&7.767>7H7P7X7`7j7r7z777777777777777788"8,848<8F8P8X8`8j8t8|8888888
aB3L3T3^3f3p3x333333333305555666dh$$dhx$$$l
t
&!$dh$6&626:6F6N6Z6b6n6v6666p$$l
t
,D\t!dh$666666666667
777&7.767>7H7P7X7`7j7r7z7777$dh$dh$7777777777777nhhdh$$$l
t
,D\t!$dh$788"8,848<8F8P8X8`8j8t8|8888888888888888$dh$dh$8888888888999"9,949<9>9L9Z9h9v9~9999999999999:
:::*:2:::D:R:Z:b:l:z::::::::::::::;;;";.;8;@;H;T;^;f;n;z;;;;;;;;;;;;;;;<<<$<,<.<:<J<X<f<n<v<
c8999"9,949<9>9L9Z9h9v9~99999999999dh$$$$l
t
,D\t!$dh$7>9L9::.<:<==L>@@@VAAAAZBhBBBrC~CCD^DFhGGGGlHzHHIIIJ0JrJOO*P+P8W:WFWLWNWVWWWWWWWYY)^/^8c>c s!sss*yFy{ {{{{{{{{{{{{|||4|>*>*H* 6>*H*6>*H*6H*OJQJH*
j0JU65CJCJT999:
:::*:2:::D:R:Z:b:l:z:::::::::$$$l
t
,D\t!$dh$::::::;;;";.;8;@;H;T;^;f;n;z;;;;;;;;;;;$dh$dh$;;;;;<<<$<,<.<:<J<X<f<n<v<<<<<<<<Ҥdh$$$$l
t
,D\t!$dh$v<<<<<<<<<<<<<<<<<=== =*=2=:=B=L=T=\=d=n=v=~============>>> >(>2>:>B>J>L>??@@@@@A$AHAJAVAdArAAAAAAAAAAAAAAABBB&B.B6B>BHBPBXBZBhBvBBBBB
a<<<<<<<<<<=== =*=2=:=B=L=T=\=d=n=v=~=$dh$~============>>> >(>2>:>B>J>L>??ژx$dh$dh$$$$l
t
,D\t!?@@@@@A$AHAJAVAdArAAAAAAAF$$l
t\8$dh$$$dh$$xAAAAAAAAABBB&B.B6B>BHBPBXBdh$F$$l
t\8$dh$XBZBhBvBBBBBBBBBBBBBBCC$C,C4C>CFCNC$$dh$dh$$$l
t8BBBBBBBBBBCC$C,C4C>CFCNCVC`ChCpCrC~CCCCCCCCCCCCCCD&D4D@DHDTD\D^DEFFFFF G6GZG\GhGvGGGGGGGGGGGGGGGHH"H,H6H>HFHPHZHbHjHlHzHHHHHHHHHHHHHHI
aNCVC`ChCpCrC~CCCCCCCCCCCCCCD&D4D@DHDdh$$$l
t8$dh$HDTD\D^DEFFFFF G6GZG\GhGF$$l
t\!<-dh$$xx$$l
t8$dh$hGvGGGGGGGGGGGGGGGHH"H,HF$$l
t\!<-$dh$dh$,H6H>HFHPHZHbHjHlHzHHHHHHHHHHHHHHIIdh$$$l
t!<-$dh$II$I2I:IFIPIXI`IlIvI~IIIIIIIIIIIIIIJJJ0J>JLJTJ\JhJpJrJKLPWQTVX;X]adg9ill%orrtgv*yFyGy{{{(|8||||~K~~~~$JK$& ($}Ò~*67My
^I$I2I:IFIPIXI`IlIvI~IIIIIIIIIIIIIIJ$$l
t!<-$dh$dh$JJJ0J>JLJTJ\JhJpJrJKLP;X]adglrtgv*ydhxdh$$$l
t!<-$dh$*yFyGy{{{(|8||||~K~~~~$JK$ ($}Òxdhxdhxd4|8|x|z|~||||||~~$~%~(~/~0~3~~~~~~~~~$J$%"$(46 *6y*<$̻Zh(Ծ|fhv<JCJ5CJ5
j0JU>*6>*OJQJH*H*6H*66H*UÒ~*67My*<Xlzʺ$dh$dh$dh$dhx
!dhxdhx*<Xlzʺ$08DLX`jr|̻$,8@JR\dlt|ļ̼ؼ(0<DPXZh~½̽Խ
(>
a$08DLX`nhhbdh$dh$$$l
t
,"$dh$`jr|̻nhdh$$$l
t
,"$dh$̻$,8@JR\dlt|ļ̨dh$$$$l
t
,"$dh$dh$ļ̼ؼ(0<DPXZh~ʄdh$$$$l
t
,"$dh$$dh$½̽Խ
(>LV^hpxpdh$$$$l
t
,"$dh$>LV^hpx¾ʾҾԾ&(24>@JLVXbdnpz|LM(0:BLT\dlt~"*2:BLT^fhv
`¾ʾҾԾ&(24>@JPdh$$$$l
t
,"$dh$JLVXbdnpz|LMdh$dh$dhxx$$$l
t
,"$dh$(0:BLTnxhhbdh$dh$$$l
t
,"$dh$T\dlt~n|hdh$$$l
t
,"$dh$"*2:BLT^fhv̨dh$$$$l
t
,"$dh$dh$
&2:<J`nx"08@HPX`hpx *,68BDNPZ\x02D`t$,
`
&2:<J`nxʈdh$$$$l
t
,"$dh$$dh$"08@HPX`hdh$$$$l
t
,"$dh$`hpx *Pdh$$$$l
t
,"$dh$\2D.:2>dz6`n
.jD_Jy\^`@A&'(789JL>*H*6>*>*H*
j0JU6H*6OJQJ5CJ5CJU*,68BDNPZ\x02D`tdh$dh$xx$$$l
t
,"$dh$$,.:]F$$l
t\S $dh$dh$dh$F$$l
t\S
,.:P^fnv~&02>Tbjrz2VXdz(6BJV^`n
.<P
a:P^fnv~&02ҸҬҤdh$$$l
tS $dh$dh$2>Tbjrzdh$xx$$l
tS $dh$dh$2VXdzdh$F$$l
t\S $dh$dh$
(6BJVdh$$$l
tS $dh$dh$F$$l
t\S V^`n
.<PR\ذؤdh$$$l
tS $dh$PR\^hjC_`UJyqZJc ,@Tbv ":VZn "6J^rt.Jd~
\\^hjC_`UJyZdhxdhxx$$l
tS $dh$Jc ,@Tbv,\$$l
tֈ_
Wdh$dhddL[]^wxy"$VWRTV@Bz|~ ":NRrt.BFw(*~JK
j0JUOJQJCJ5CJ5CJ6H*H*66>*>*V ":VZn $$l
t_
Wdh$ "6J^rt.Jd~ެD<dh$ $$l
t_
W
34w$$$ $$l
t_
Wdh$
34w(*~'1;EFTZ`flmy
()7
](*@]"$$l
t4| $$$r$$l
tִ4|
*~Q$$l
tr_
"cdh$dhEFTlmy()7KLXijuK L N y z
*
+
7
K
L
W
k
l
MN6
j0JUCJH*6CJCJCJ5CJ5Xt$$l
t_
"cdh$Q$$l
tr_
"c'1;EFTZ`flmy٨ٜٔt$$l
t_
"cdh$
()7<~Q$$l
tr_
"cdh$$l
t_
"cdh$7<AFKLXZ_dijuw|N T ^ e o y z
%
*
+
7
<
A
F
K
a<AFKLXZ_dijuw|xt$$l
t_
"cQ$$l
tr_
"cdh$ߨߌ߄|dh$$$l
t_
"cN T ^ e o y dh$$l
t_
"cdh$y z UxQ$$l
tr_
"cdh$Q$$l
tr_
"c
t٨ٌ$$l
t_
"cdh$
%
*
+
7
<
A
F
K
L
W
\
a
f
k
l
ABلـ
d$$l
t_
"cdh$K
L
W
\
a
f
k
l
AB[J !!f!g!m!n!!!."/"8"9"X"Y"""""`#a#####$$$$$$%%&&&&&&''M(N(q(r((())))**0*1***++++++++#,$,S,T,,,
_NO&'(K g!m!/"8"""###$$%&&N(q(r(()**0*+++#,$,S,R-n-d....//00002244.6/666U7788)8@888v9|999F:L:{::Q;v;;;<5<<6OJQJ6>*5OJQJOJQJ
j0JU6H*X[J !!f!g!m!n!!!."/"8"9"X"Y""""dhxdhx""`#a#####$$$$$$%%&&&&&&''M(N(q(r(((())))**0*1***++++++++#,$,S,T,,,Q-R-n-o--,Q-R-n-o---c.d.....7/8/////U0V0000000112222334444r5s5.66;<P>}> ?U???ABC~CQDDDcEEEEYFFHII
J4JVJJjKKKLMOOOP
PPPPPPPPPP
Y--c.d.....7/8/////U0V00000001122223334444r5s5.66;<P>}> ?U???ABC~CQDDDcEEEEYFF<<<<=4=>A>P>Q>Y>r>}>~>>>>? ?!?5?K?U?V?j????????AAABBBBBBCCC?CPC~CCCCQDRDDDDDDDDE(EVEcEdEjEpEEEEEEEEEEE,FKFYFZFFFGGdHhHHHIIIIIIIJ
JJ
j0JU6aFHII
J4JVJJjKKKLOOO
PPPPPPPPh&`#$JJ&J4J5J?JMJVJWJJJJJJK0K\KjKkKKKKKKKKL=LKLpLLLLLLMMM/M0M^M_MMMMMMN4N6N8NNNOOOOOPPPP P
PP
PPPPPPP0JmH0J
j0JUH*
j0JU6G+
0&P . A!"#$%
[(@(NormalCJmH :@: Heading 1$dh@&5CJF@F Heading 2$<@&56CJOJQJ@@@ Heading 3$<@&CJOJQJ:@: Heading 4$dh@&6<A@<Default Paragraph Font2@2
Footnote TextCJ8&@8Footnote ReferenceH*BC@BBody Text Indentdhx*B@"* Body TextCJ<P@2<Body Text 2dhx5CJ,@B,Header
!&)@Q&Page Number<R@b<Body Text Indent 2p(U@q( Hyperlink>*B*,_f#
q
tBT% !*6<Q{RDXXfmltϮn@LZ
!"#~e"O' \ P
#x5T+V(<{Z,
74|LN<JP^'+`,V--./01^2B3667789:;<~=?AXBNCHDhG,HIJ*yÒ`̻ļJT`*:2V\*<y
"(-3FP
+j/38v<BI>,P
7K
,P
!!rvKY< B k p u}
!###$$$$$$$e%i%l%o%&&*&-&0&&&&&''''((()L)P)S)V))))*****e+i+l+o+,",%,(,,,--1-I-L-'.0.4.>.x............4/8/;/>/}////////0060:0R0U0/181<1F1111111111112=2A2D2G22222222233@3D3\3_3R4V444A5O56677::::C;G;=>"?,?BBvEzEKGUG>NGNXXXXXXXXXX+Y4YYY4Z>Zddddddddjj$m+mnnjpqpttBvKvvvvwwwzzzz{{]|l|||eoÊ#nrԌ،8<jnIMtxڑޑ#'ޒ*3FP^b
BFgk"+/9mv˕ϕ%)PT|?AIKxzIPbkboajcm'yx<EYc%RW[a*5qxmsHNBHqx:>u|
=Anuw|}#(.4OVX]^f{=DFKLT
2 7 = C
#%*+3>X[
}Uz ((I-M-R0V0\3`34444i6l6777788H<L<z==>>0@4@AABBC CDDHH!Q$QSSUU&V*VWXXXXXXXYYYYZ2Z4Z?Z^=___``ddddddggBlFl.o2oppssvv$w(wMw\wNySy4~8~Ն׆_aјԘ xz!%|LQ9B8= y:i@X[=jn=\
;>X[./C:\TEMP\AutoRecovery save of ess_pap_MPP_12.asd./C:\TEMP\AutoRecovery save of ess_pap_MPP_12.asd..C:\Wroe\Ph.D. Work\Articles\ess_pap_MPP_12.doc./C:\TEMP\AutoRecovery save of ess_pap_MPP_12.asd./C:\TEMP\AutoRecovery save of ess_pap_MPP_12.asd./C:\TEMP\AutoRecovery save of ess_pap_MPP_12.asd./C:\TEMP\AutoRecovery save of ess_pap_MPP_12.asd.9C:\Wroe\Publications\Working Documents\ess_pap_MPP_13.doc./C:\TEMP\AutoRecovery save of ess_pap_MPP_13.asd.9C:\Wroe\Publications\Working Documents\ess_pap_MPP_13.docN BqX>"uf,D`cʵ0o(.0o(.0o(.0o(.cqX>N,D@A\d$%&')+,-./0136W7W8W:W;WEWFKLOPRSWXYdelopq____________?
?Z@@@@@@@4@@,@.@0@2@6@:@<@>@@@B@D@F@J@@@R@@@V@@@b@@@j@@@p@@@t@@@z@|@@@@@@@4@@@<@@@`@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@N@@GTimes New Roman5Symbol3&Arial"qh2f2f2f!%/p5!20dIntroduction..Oh+'0h
$0
<HPX`
Introductionontr.trtrNormalc.rm3rmMicrosoft Word 8.0@2~@#O@!O@%ѧO!%՜.+,D՜.+,8hp|
d/
IntroductionTitle 6>
_PID_GUIDAN{117DA7F4-0462-11D2-B241-D29B9472720A}
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~
!"#$%&'()+,-./013456789>Root Entry F`!O CҧO@1Table%WordDocument-,SummaryInformation(*DocumentSummaryInformationelop8velope #12Envelop21CompObj#10Envelope #9Note114QuartoFj)B4 (JIS)A5A4 SmallA4A3ExecutementLegalLedgerTabloidLetter SmallLetter - |mpH >pȿ
FMicrosoft Word Document
MSWordDocWord.Document.89q