| Home |
| 1-Overview || 2-FAQ || 3-Primary Findings || 4-Actual Performance | | 5-Funding Variables |
| 6-Teacher Data || 7-Race || 8-OSRC || 9-Closing Statement || Appendix-Top Performing Districts |

Forces and Factors Affecting Ohio Proficiency Test Performance:
A Study of 593 Ohio School Districts

Randy L. Hoover, Ph.D.

Section Four
Actual District Performance: Controlling for the Presage Factor

      An interesting way to examine district performance is to look at it controlling for the effects of the non-school forces and factors that comprise the Presage Factor. The concept of actual district performance reflects the idea that once we are able to establish the effects of the Presage Factor on district performance, we then are able to compare the predicted rate of passing with the actual rate of passing given the presage score for the district. In this sense, we are controlling for the effects of advantagement-disadvantagement for each of the 593 Ohio school districts and seeing OPT performance through a very different lens than does the State of Ohio.
      In other words, since we know the power of the Presage Factor’s effect (r=0.80) and that most conservatively it accounts for 64% of the test performance, we can then examine district performance controlling for the Presage Factor’s effects by comparing the predicted passing rate to the actual passing rate. We then compare those performances.
      The following is a graphing of what I term "actual" performance because it shows how districts are performing with the social-economic factors contained the Presage factor removed. Essentially, it is a graph that indicates how far districts are above or below the regression line shown in the primary graph of "Advantagement-Disadvantagement as a Predictor of District Performance." (See Section Three.) The distance above or below the regression line of the aforementioned graph is termed a "residual" and represents the difference between where we would expect a district to fall based upon the predictive power of the Presage Factor and where the district actually falls.
  • The upper left quadrant represents districts that are performing average or above average and have average or below average levels of advantagement.
  • The upper right quadrant represents districts performing average or above average and have average or above average advantagement.
  • The lower left quadrant represents districts that are performing average or below average and have average or below average advantagement.
  • The lower right quadrant represents districts performing average or below average and have average or above average advantagement.
  • The greater the distance above or below the x-axis (the horizontal red line), the more the district is performing respectively beyond or below what would be expected given the presage score of the particular district.
  • Districts falling between +1 and -1 on the x-axis are all within one standard deviation of the mean and may be considered as having average performance that is about where we would expect them to perform.
  • Any district above the +1 mark above the x-axis is performing significantly better than average and better than would be expected. Likewise, any district below the -1 mark below the x-axis is performing significantly lower than average and lower than would be expected.
      This graph shows that when the district OPT performance residuals (actual performance) from the primary graph are themselves compared to Presage levels, there is no correlation whatsoever. This is one of those rare cases when a low or zero-order correlation is good. What is shown is that actual district performance (performance determined by controlling for Presage Factor effects) is, indeed, free of any and all Presage Factor effects as we would expect. This graph offers us a view from which we can examine district performance without having to be misled by Presage Factor effects. However, several caveats must be made to avoid misunderstanding actual performance.
      The term "actual" is restricted to describing performance controlling only for the Presage Factor. There should be no claim made that this performance has any substantial validity beyond simply correcting for the bias of social-economic advantagement-disadvantagement as defined by the Presage Factor. However, we can and do argue that it does give a much more accurate view of district performance than does the format used by the State of Ohio.
      Though we are now able to view district performance free of Presage Factor effects yet through the lens of OPT, this study does not in any way wish to imply that OPT should ever be used as any measure of district, teacher, or pupil accountability whatsoever. Indeed, this study does not address the severe psychological effects of OPT on Ohio’s public school children. Likewise, this study does not address the pedagogical effects of OPT in terms of detrimental effects on empowering curriculum and instruction in Ohio’s schools. Carefully conducted studies of the psychological effects and the pedagogical effects of OPT are vital before any consideration is given to using any form of high stakes testing to draw conclusions about school district performance.
      Some of the performance differentials between the OPT as reported by the State of Ohio and scores adjusted for the bias of the Presage Factor are striking. For example, Youngstown City Schools ranks 581 out of 593 districts in percent passing the 1997 OPT. However, when correcting for the bias of advantagement-disadvantagement, Youngstown’s rank is 58 out of 593. Youngstown City School District is in the top 10% in the rankings of actual performance. Indian Hill Exempted School District is 16th in percent passing 1997 OPT, but falls to 581 when correcting for advantagement-disadvantagement.
      In the case of Youngstown City, we have a district that is steeped in disadvantagement as defined by the Presage Factor, and in the case of Indian Hill we have a district steeped in advantagement. We know through the primary finding that the presage effects predict a low rate of passing for Youngstown City and a high rate of passing for Indian Hill. However, by calculating the difference between the predicted rate of passing and the actual rate of passing, we can see the degree to which they are performing above or below expectations established by the power of the non-school variables of the Presage Factor. Thus, we now have a new way to assess district performance, one that compares districts without the bias inherent in the non-school variables contained in the Presage Factor.
      Some districts show little or no change when correcting for Presage Factor OPT bias. For example, South Range Local School District is 15th in percent passing the 1997 OPT, yet is 3rd in actual performance when correcting for advantagement-disadvantagement. In other words, South Range’s performance is high in both systems.
      The importance of what this aspect of the analysis shows us is that viewing or ranking districts without considering the social-economic bias of the OPT results in many districts being extremely over-rated or extremely under-rated. In this manner, the stakeholders of the state in general and the stakeholders within the local districts in particular are often being given monumentally misleading assessment information. Unfortunately, this misleading information is used to drive public praise or public criticism of Ohio’s local schools. Indeed, many Ohioans are keenly interested in their public schools but are relying on invalid information to make informed decisions directly affecting the lives of both adults and children.
      The problem resulting from failure to understand or correct for OPT bias is compounded through the format of the Ohio School Report Card. Because so many of the performance standards comprising the OSRC are directly dependent upon the percentages of students passing the various tests, the fundamental and significant bias of the tests carries over directly into the OSRC ratings. Section 9 of the study deals briefly with the validity of the Ohio School Report Card as affected by the findings about OPT performance.
            Understanding the Next Sections:
      We have now completed the basis for understanding the presentation and findings regarding the additional EMIS variables used in this study of forces and factors affecting OPT district performance. In the following sections, these additional variables are presented using three perspectives on the data:
The variable in relation to the Presage Factor.
The variable in relation to percent passing the 1997 tests.
The variable in relation to actual performance as defined in this section.
      Using these three perspectives gives us a triangulation that illuminates the role of the particular variable beyond simply its association with passing rates that are biased by the loading of the non-school effects of advantagement-disadvantagement. Thus the interpretation of each of the following variables is intended to provide deeper insight into the significance and meaning of the variable as it affects or is related to the context of OPT performance. The following variables are examined and interpreted in the subsequent sections of the study:
• Section 5: Federal, State, and Local Funding
• Section 6: Teachers
• Section 7: African-American and White
| Home |
| 1-Overview || 2-FAQ || 3-Primary Findings || 4-Actual Performance | | 5-Funding Variables |
| 6-Teacher Data || 7-Race || 8-OSRC || 9-Closing Statement || Appendix-Top Performing Districts |