| Home |

| 1-Overview || 2-FAQ || 3-Primary Findings || 4-Actual Performance | | 5-Funding Variables |

| 6-Teacher Data || 7-Race || 8-OSRC || 9-Closing Statement || Appendix-Top Performing Districts |

Forces and Factors Affecting Ohio Proficiency Test Performance:

A Study of 593 Ohio School DistrictsRandy L. Hoover, Ph.D.

Section 2

Frequently Asked Questions and Explanation of Terms

The following are very brief summaries of key elements of the study. Each item presented below is explained in greater depth within the text of the study itself.

•What did the study involve?Briefly stated, this research study involved the examination of 593 Ohio school districts across 40 variables using 16 sets of OPT scores for each school district. All data were collected from EMIS online data banks and the data were analyzed using statistical methods such as regression analysis and correlation analysis. Both school and non-school variables were used.

•What is the purpose of this study?The purpose of the study was to attempt to identify both school and non-school variables most significantly associated with district test performance in order to illuminate the degree to which OPT is a valid and reasonable mechanism for assessing school performance in terms of academic achievement and educator accountability. Similarly, an attempt was made to isolate and examine any variables found to be likely significant in contributing to actual district performance.

•What is the difference between a school variable and a non-school variable?

School variables are those forces and factors that schools can control and adjust such as class size, per pupil expenditure, and teacher salary among many others. Non-school variables are forces and factors over which schools have no control such as mean family income, property values, and poverty levels among many others.

•What are the primary findings?The study found that OPT district test performance is most strongly connected to the living conditions and the lived experiences of the students in terms of economic, social, and environmental factors. District test performance was found to correlate extremely high with advantagement-disadvantagement: The greater the wealth of the students of the school district, the better the district OPT performance. In this study, the term "Presage Factor" is used to indicate the social, economic, and environmental variables of advantagement-disadvantagement.

•What do the primary findings tell us about the OSRC?The findings also show the Ohio School Report Card to be equally as invalid as OPT performance. This finding is not too surprising when we consider that OPT performance is the primary element that drives OSRC ratings. In other words, if OPT does not carry significant validity, then OSRC will not either because it is primarily a function of district OPT performance.

•What exactly is the Presage Factor?The Presage Factor is a combination of the Ohio Department of Education’s online EMIS variables that represent measures of advantagement-disadvantagement. It combines the following EMIS measures: percent ADC, percent enrolled in the subsidized school lunch program, percent economically disadvantaged, and mean family income. These variables are combined in a very straightforward manner using a simple calculus to derive a scaled measure of advantagement-disadvantagement. Section Three gives the precise formula for calculating the Presage Factor.

•What is meant by advantagement-disadvantagement?Advantagement-disadvantagement is intended to represent the continuum of social-economic forces and factors that are indicated by the Presage Factor. They are the forces and factors that shape the lived experience of all children. The knowledge, culture, values, attitudes, and meanings that children bring to school are a largely shaped by their lived experiences. This particular term is not used the same way as the terms "educationally disadvantaged" or "educationally advantaged." These terms refer to how schooling itself, through its practices and processes, is structured to reward or punish students for the knowledge, values, and cultural meanings they bring to school.

•What are linear regression and statistical correlation?Linear regression is used to examine the relationship between two variables such as the Presage Factor and the percent passing the OPT. Basically it allows us to perceive how the change in one set of variables relates to corresponding change in the other set of variables. Statistical correlation then allows us to determine the strength of the relationship between the two sets of variables. The correlation used in this study is called "Pearson's correlation" or "Pearson's r."

It is this correlation that tells how significant the association is between the sets of variables. Correlation analysis yields what is called the "correlation coefficient" or "r." The range of "r" is from -1.0 to 1.0. The closer that "r" is to -1.0 or 1.0, the stronger the relationship between the two sets of variables being analyzed. For example, where r=1.0, the correlation is perfect... where r=0.0, there is no relationship whatsoever. In cases where "r" is negative, the correlation is said to be inverse, meaning that as the value of one variable increases, the value of the other decreases. (See the graph of percent passing and percent ADC for an example of an inverse correlation.) In cases where "r" is positive, as the value of one variable increases so does the value of the other variable.

In social science research, a perfect correlation is rarely, if ever, found. Indeed, correlations approaching either r=-0.50 or r=0.50 are usually considered relatively significant. It is suggested that you consult a good statistics text for better understanding of the details and assumptions involved with regression analysis and correlation. It needs to be noted that the primary finding of this study regarding the relationship between advantagement-disadvantagement and OPT district performance is r=0.80, a significantly high correlation by any statistical standards.

•What are residuals?

A residual is the difference between what the linear regression predicts a given value will be and what the value actually is based upon the line generated by the mathematics of linear regression. It is essentially the mathematical distance of a data point above or below the regression line. In the case of this study, district residuals from the Presage Score/Percent Passing regression are used to postulate actual performance. Doing this gives us some idea of performance controlling for the Presage Factor.

•What exactly is a z-Score and why use it?

A z-score (often called a "standard score") is a transformation of a raw score into standard deviation units. Using z-scores allows us to immediately know how far above or below the mean is any given score, thus allowing us to visualize how extreme the score is relative to all other scores. The mean of any z-score distribution is always zero. Using z-scores does not alter the distribution of scores in any way and does not affect the analysis or the findings. Converting to z-scores is a linear transformation and does not change the results of the data analysis in any way other than to make the data more understandable.

The advantage of the z-score is found in allowing us to understand one score relative to other scores. For example, the Presage score as a raw score for Youngstown City School District is -173.08, which does not tell us how extreme the disadvantagement is. The Presage z-score for Youngstown is -3.82, which tells us that it is 3.82 standard deviations below the State average, thus allowing us to see that Youngstown's students are very deeply in social-economic disadvantagement.

•What exactly is standard deviation?

Most simply put, standard deviation describes how a set of scores is distributed around the mean of the set. For use in this study, basic knowledge of standard deviation is helpful in reading and understanding the z-scores. Z-scores tell us how many standard deviations above or below the mean a score is. Z-scores greater than 1.0 or lower than -1.0 suggest more significant scores beyond those within 1.0 and -1.0. In the case of reasonably normal distributions such as with the data in this study, approximately 68% of the scores will fall within the 1.0 and -1.0 range of the first standard deviation and 95% of the scores will fall within the limits of the second standard deviation. Scores in the third standard deviation may be thought of as being extreme. Thus, the example of Youngstown given above as having a Presage z-score of -3.82 tells us that it is a case of children living in extremely disadvantaged environments.

•How significant or powerful are the findings?

The correlation between the measure of advantagement-disadvantagement (Presage Factor) and OPT performance are extremely high (r=0.80). Indeed, these findings about this relationship are about as high as are ever found in social science research. . . the findings are very significant both statistically, conceptually, and practically.

•Can OPT scores be raised through school interventions?

The question as to whether OPT scores can be raised can certainly be answered in the affirmative, though it is not considered within the study. However, any educational imperative to raise scores must not be based on an invalid test nor must it be directed toward any form of high stakes testing. Instead, it must be driven by the vision of empowerment, the idea that what students are taught in schools must be personally experienced by the students. Knowledge must be taught in such a manner that it is felt as relevant and usable in the mind of the learner. To empower learners requires constructing learning activities that become personally felt lived experiences for the students in the classrooms, not abstract rote exercises over facts and ideas that the students perceive as meaningless and irrelevant. The usability of academic knowledge must be taught by the teachers and must be experienced by the students if we are to empower learners and raise scores significantly.

•What do the findings tell us about the validity of the OPT as an assessment of academic achievement?

The findings tell us that OPT performance is in no manner a valid measure of academic achievement: The OPT measures almost exclusively only the quality of life in which the students of the district live.

•What do these findings suggest about the validity of the Ohio School Report Card?

The findings tell us that the Ohio School Report Card, because it is almost entirely based upon OPT performance, is a totally invalid assessment of actual school district performance and should not be used. OSRC is extremely misleading, and the general public should be outraged about its use. Likewise, the State Legislature and Governor should be held accountable for misleading the citizens of Ohio and using state monies for such an invalid assessment of school district performance.

•What do the findings tell us about accountability on the part of districts, administrators, teachers, and Ohio's public school pupils?

Accountability is the least understood term in the American political lexicon. For true accountability to be invoked, we must understand that valid accountability is a function of the decision latitude and amount of performance control vested in those being held accountable. In other words, it is wrong to hold districts, administrators, teachers, or students accountable for a test that measures variables over which they have absolutely no control. This study finds beyond the shadow of any doubt that the OPT is not a measure of virtually anything related to in-school variables; it is a measure of non-school variables, forces, and factors. Therefore, to hold those associated with schools accountable for OPT performance is absurd and wrong. It is tantamount to holding the TV weather person accountable for today’s weather.

•From this study, is it possible to assess with some degree of validity the actual levels of Ohio school district performance?

The answer here is both yes and no. It is "yes" in terms of knowing that the Presage Factor is so very powerful that if we control for its effects, we begin to get a much clearer and certainly much more valid picture of how each district is actually performing. It is "no" in the sense that this performance even controlling for the Presage variables still is primarily based upon the OPT itself. To assess school district performance using the OPT would be foolish and wrong in that it is the public school student who suffers most from the test. In other words, why hurt and mislead the children and parents of Ohio to assess district performance using an invalid instrument.

•When will copies of the findings be made public?

The study itself was officially released to the public and media as of 12:01 AM, February 27, 2000. On April 27, 2000 the findings will be presented at Ohio's Teaching Learning Conference 2000 in Columbus. The presentation is scheduled for 8:45 am, tentatively in Room C214 in the Columbus Convention Center. Copies of the final study will be available at this presentation and will be available online March 1, 2000 at http://cc.ysu.edu/~rlhoover/OPT .

| Home |

| 1-Overview || 2-FAQ || 3-Primary Findings || 4-Actual Performance | | 5-Funding Variables |

| 6-Teacher Data || 7-Race || 8-OSRC || 9-Closing Statement || Appendix-Top Performing Districts |