| Home |
| 1-Overview || 2-FAQ || 3-Primary Findings || 4-Actual Performance | | 5-Funding Variables |
| 6-Teacher Data || 7-Race || 8-OSRC || 9-Closing Statement || Appendix-Top Performing Districts |
Randy L. Hoover, Ph.D.
The fundamental purpose of this study was to examine what forces and factors may be affecting district-level performance on the Ohio Proficiency Tests and to attempt to determine to what degree these variables shape district-level performance. To this end, two categories of variables were used: school variables and non-school variables. School variables are those forces and factors that schools can control and adjust such as class size, per pupil expenditure, and teacher salary among many others. Non-School variables are forces and factors over which schools have no control such as mean family income, property values, and poverty levels among many others.
As briefly discussed previously, the primary finding is that OPT performance is affected most significantly by non-school variables representing the lived experiences of the children attending the school district. The lived experiences of children come from and happen within the advantagement- disadvantagement of their environments. These experiences of real-life are non-school variables that clearly shape how, what, and whether a child learns.
The term "Presage Factor" was chosen to indicate the data used collectively as a measure of the non-school variables that serve as the indicator of the degree of advantagement- disadvantagement experienced in the lives of the districts school children. The term was chosen because the word "presage" means to predict, foresee, or foreshadow, which is what knowledge of basic living conditions within the district allows us to do with OPT performance when we can mathematically quantify elements of those basic living conditions.
The graph below shows the power of the Presage Factor as a measure of advantagement- disadvantagement in predicting district OPT performance. The "Y" axis represents the mean percent of a district's students passing across the four sections of the 4th, 6th, 9th, and 12th grade 1997 Ohio Proficiency Tests: %Passing = [(%4Math + %4Reading + %4Writing + %4Citizenship + %6Math + %6Reading + %6Writing + %6Citizenship + %9Math + %9Reading + %9Writing + %9Citizenship + %12Math + %12Reading + %12Writing + %12Citizenship)/16].
The "X" axis represents the Presage Factor expressed in raw scores. The presage score is a measure of the degree of social-economic disadvantagement-advantagement derived from EMIS data that combines the percent of the student population of the school district for Aid to Dependent Children, percent enrolled in the Free or Reduced Lunch Program, percent listed by the State of Ohio in Economic Disadvantagement, and Mean Family Income. The formula or algorithm for the Presage Factor is: Presage Score = (Mean Family Income/1000) - (%FreeReducedLunch + %ADC + %EcoDis).
From the data analysis represented in the graph below, we find that performance across the 593 Ohio districts included in this study is associated with non-school environmental conditions of advantagement-disadvantagement to the extent of r = 0.80. This is an extremely high correlation and clearly brings the validity of OPT into serious question.
Interpretation of the correlation coefficient of r=0.80 tells us that, conservatively, the non-school related effects of advantagement-disadvantagement defined by the Presage Factor determine 64% of OPT performance. It is important to note that this 64% determination is restricted to the effects of the Presage Factor and, by definition, does not include other possible advantagement-disadvantagement effects outside the realm of those included in the Presage Factor.
Indeed, the idea that advantagement-disadvantagement limited to the scope of the Presage Factor determines 64% of OPT performance is a conservative interpretation of the overall power of social-economic living conditions because it may well be excluding other significant non-school forces and factors. There is a real possibility that there are still social-economic effects beyond the range of those comprising the Presage Factor, though extremely powerful in its own predictive power. For more possible insights to additional non-school variable effects beyond those within the scope of the Presage Factor, see the sections on "Actual District Performance: Controlling for the Presage Factor" and "Percent African-American and Percent White as Variables Across Presage Score, Percent Passing, and Actual Performance."
Because of the discovery that OPT performance is overwhelmingly determined by the social-economic living conditions that the students of the district experience growing up, the inescapable conclusion is that OPT is not a valid measure of either school or teacher effectiveness and should not be used for accountability assessment. The OPT is invalid because the results of this study show that it does not measure what it claims to measure: Student performance on the OPT is, at best, academically meaningless. It is highly biased against economically disadvantaged students and highly biased in favor of economically advantaged students.Using z-Scores for Graphs:
A z-score (often called a "standard score") is a transformation of a raw score into standard deviation units. Using z-scores allows us to immediately know how far above or below the mean is any given score, thus allowing us to visualize how extreme the score is relative to all other scores. The mean of any z-score distribution is always zero. Using z-scores does not alter the distribution of scores in any way and does not affect the analysis or the findings. Converting to z-scores is a linear transformation and does not change the results of the data analysis in any way other than to make the data more understandable.
The advantage of the z-score is found in allowing us to understand one score relative to other scores. For example the Presage score as a raw score for Youngstown City School District is -173.08, which does not tell us how extreme the disadvantagement is. The Presage z-score for Youngstown is -3.82, which tells us that it is 3.82 standard deviations below the State average, thus allowing us to see that Youngstown's students are very deeply in social-economic disadvantagement. Likewise, the presage score for Indian Hill Exempted School District is 164.76, a figure that alone tells us little about the meaning of the score. However, the z-score for Indian Hill is 4.37, which tells us that it is a very advantaged district.
Most simply put, standard deviation describes how a set of scores is distributed around the mean (average) of the set. For use in this study, basic knowledge of standard deviation is helpful in reading and understanding the z-scores. Z-scores tell us how many standard deviations above or below the mean a score is. Z-scores above the mean are positive numbers and those below are negative numbers.
Z-scores greater than 1.0 or lower than -1.0 tell us that these scores are significantly more extreme than those within 1.0 and -1.0. In the case of reasonably normal distributions such as with the data in this study, approximately 68% of the scores will fall within the 1.0 and -1.0 range of the first standard deviation, and 95% of the scores will fall within the limits of the second standard deviation. Scores in the second standard deviation are more extreme than those in the first standard deviation, and those in the third standard deviation may be thought of as being very extreme. Thus, the example of Youngstown given above as having a Presage z-score of -3.82 tells us that it is a case of children living in extremely disadvantaged environments relative to what is typical within the State of Ohio.
The following graph is a z-score version of the previous graph showing the relationship between percent passing and the presage score. Both percent passing and the presage scores have been transformed into z-scores. You will note that the graph is virtually identical to the previous one and has exactly the same correlation coefficient (r=0.80). However, because we now have z-scores to view, we can easily see the categories near, above, or below the mean for each district.
In addition, categories of advantagement-disadvantagement have been added to the graph using the z-score divisions of standard deviation. The center column "Middle Class" is divided down the middle by the mean (average) for the state. Using the z-score divisions for standard deviations above and below the mean, we can then classify levels of advantagement-disadvantagement based upon those mathematical divisions, thus making it more clear as to just how the different districts can be seen to compare with each other. Youngstown City and Indian hill districts that were used previously as examples of z-scores are both circled on the graph, showing the z-score significance visually.
Though categorical descriptors have not been added to the x-axis, we can still see how far above or below the state mean the various districts fall. If we were to create a grid by marking off the z-score standard deviations for the percent passing 1997, we would see that districts cluster in very similar ways where passing and presage scores have similarly high or low z-scores. This grouping is simply another way of seeing how districts with higher levels of advantagement cluster with higher levels of percent passing as low advantaged districts cluster with low percent passing. Once again, note how Youngstown City and Indian Hill are respectively low-low and high-high within the clusterings that are shaped by the data as arrayed by z-score graphing.Data Supporting the Presage Factor Significance:
What is somewhat unusual is that the variables combined through the calculus of the presage formula yield a more powerful predictive correlation than do any one of the individual variables used in the formulation. However fortuitous, it is important and illuminating to understand the significant degree to which district test performance is predicted by the individual variables of Free/Reduced Lunch enrollment, ADC, Economic Disadvantagement, and Mean Family Income. The following four graphs visually represent these component variables used in the presage formula. I believe they help us understand the gravity of using tests such as OPT where the bias is so clearly shown.
The graph of percent enrolled in the free/subsidized lunch program shows an inverse correlation of r=0.73, which should be considered an extremely significant correlation. It is an inverse correlation simply because as the percent enrolled in the program increases, district performance drops. The primary evidence the finding provides is to validate the association of test performance with a specific measure of advantagement-disadvantagement.
The State of Ohios own measure of economic disadvantagement also shows significant correlation with district OPT performance. As with the previous graph, the correlation is inverse, telling us that as the percent of economic disadvantagement goes up, district test performance goes down.
The graph of mean income provides us with both a significant correlation and a telling view of the mean income data itself. The correlation between the mean income of a district and OPT performance is r=0.58. Though lower than the correlations seen in the previous findings, r=0.58 is still a highly significant correlation coefficient. In terms of the coefficient of determination (r2), we find mean family income conservatively determining about 33% of district OPT performance.
However, the distribution is somewhat curvilinear. A curvilinear distribution is one in which the distribution points have a visible curvature of some sort. The curvilinearity is visible in the mean income graph as the array of points can be seen to bend to the right toward the quadrant formed by the above-average mean income and above-average district performance area of the graph.
Two findings can be drawn from the curvilinear spread. The first finding is the statistical reality that because the data array is clearly curvilinear, the correlation coefficient is underestimating the degree of association between the two variables. This means that the correlation coefficient of r=0.58 is most likely considerably lower than the actual degree of correlation. In other words, though r=0.58 is a relatively high correlation, it belies the reality of there being actually a higher correlation than seen due to the curvilinearity.
The second finding is serendipitous to the study but both relevant and interesting taken within the context of OPT and the effects of non-school variables on district performance. In examining the curved nature of the data, we can see implicit evidence of how mean income changes dramatically as we move from the upper middle class to the upper classes.
Because income distribution is the primary determiner of relative advantagement-disadvantagement disparity, the decision was made to examine how the continuum of advantagement-disadvantagement has been shaped by mean income changes over the past ten years and how it may have exacerbated the extremes of poverty and wealth affecting the lived experiences of Ohios children.
In other words, because we can think of the presage score as representing a point on the continuum of advantagement-disadvantagement and because the range (length) of that continuum represents the scope of disparity in living conditions, we can examine how that scope may have changed over the past several years. The relevance of this side-bar analysis to this study is to provide a context for better understanding who is intrinsically advantaged and who is intrinsically disadvantaged by OPT and how those may have changed as a function of the distribution of wealth over the past few years. The following graph shows how district mean family income has changed over the years 1987 through 1998.
This graph shows how district mean family income changed from 1987 to 1998 in terms of the presage scores. The most striking finding is that income increased far greater for the wealthiest districts than for the less wealthy ones. Indeed, when the graph is examined closely, we see that increases in family income are relatively slight from the extremely disadvantaged upward through the middle class until we reach the upper end of the middle class and into the advantaged class, where it changed dramatically.
The most contrasting districts have been identified on the graph to better understand the extremes of the advantagement- disadvantagement continuum. As would be expected given the power of the Presage Factor, the mean percent passing for the 5 districts with the greatest increase in mean family income is 91.4%; the mean percent passing for the six districts with the least change in mean family income is 52.9%.
What the comparison in the above paragraph tells us is that OPT is very tightly tied to an explicit association with wealth. The degree to which the association with wealth is a function of living conditions and the lived experiences of the districts children is told in the elements that comprise the Presage Factor and their individual contributions shown in this section above. However, the question also arises as to the degree of local financial contribution to the local districts funds given the wealth available to commit funds. Findings regarding funding variables will be examined briefly in Section Five, after examination of district performance controlling for the effects of the Presage Factor in Section Four.