Introduction to the OPT Study and the Findings: FAQ

Introduction to the OPT Study and the Findings:
FAQ

These data are for use only by students enrolled in the graduate classes of Randy L. Hoover, Ph.D. Any use outside the reasonable limits of classroom assignments or any use by others is strictly forbidden without explicit written permission of the author.

©R.L. Hoover, 1999

Return to the Table of Contents

Frequently Asked Questions:
What did the study involve?
What is the purpose of this study?
What is the difference between a school variable and a non-school variable?
What are the primary findings?
What exactly is the Presage Factor?
What are linear regression and statistical correlation?
What are residuals?
What is a z-score and why use it?
What is meant by standard deviation?
How significant or powerful are the findings?
Can OPT scores be raised through school interventions?
What do the findings tell us about the valididty of the OPT as an assessement of academic achievment?
What do these findings suggest about the validity of the Ohio School Report Card?
What do the findings tell us about the accountabilty of districts, administrators, teachers, and pupils?
Is it possible using OPT to assess with some validity the actual levels of Ohio school district performance?
When will the results be made public?

What did the study involve?
      Briefly stated, this research study involved the examination of 593 Ohio school districts across 40 variables using 16 sets of OPT scores for each school district. All data were collected from EMIS online data banks and the data were analyzed using statistical methods such as regression analysis and corrlation analysis. Both school and non-school variables were used.

What is the purpose of this study?
      The purpose of the study was to attempt to identify both school and non-school variables most significantly associated with district test performance.

What is the difference between a school variable and a non-school variable?
      School variables are those things that schools can control and adjust such as class size, per pupil expenditure, and teacher salary among many others. Non-School variables are things over which schools have no control such as mean family income, property values, and poverty levels among many others.

What are the primary findings?
      The study found that OPT district test performance is most strongly connected to the living conditions of the students in terms of economic, social, and environmental factors. District test performance was found to correlate extremely high with advantagement-disadvantagement: The greater the wealth of the students of the school district, the better the OPT performance. In this study the term "Presage Factor" is used to indicate the social, ecomomic, and environmental variables of advantagement-disadvantagement.

What exactly is the Presage Factor?
      The Preage Factor is a combination of EMIS variables that represent measures of advantagement-disadvantagement. It combines the following EMIS measures: percent ADC, percent enrolled in the subsidized school lunch program, percent economically disadvantaged, and mean family income. These variables are combined in a very straightforward manner using a simple calculus to derive a scaled measure of advantagement-disadvantagement. The precise forumula may be found in the final study document when it is released.

What are linear regression and statistical correlation?
      Linear regression is used to examine the relationship between two variables such as the Presage Factor and the percent passing the OPT. Basically it allows us to perceive how the change in one set of variables relates to corresponding change in the other set of variables. Statistical correlation the allows us to determine the strength of the relationship between the two sets of variables. The correlation used in this study is called "Pearson's correlation" or "Pearson's r."
      It is this correlation that tells how significant the association is between the sets of variables. Correlation analysis yeilds what is called the "correlation coefficient" or "r." The range of "r" is from -1.0 to 1.0. The closer that "r" is to -1.0 or 1.0, the stronger the relationship between the two sets of variables being analyzed. For example, where r=1.0 the correlation is perfect... where r=0.0 there is no relatinship whatsoever. In cases where "r" is negative, the correlation is said to be inverse meaning that as the value of one variable increases, the value of the other decreases. (See the graph of percent passing and percent ADC for an example of an inverse correlation.) In cases where "r" is positive, as the value of one variable increases so does the value of the other variable.
      In social science research, a perfect correlation is rarely, if ever, found. Indeed, correlations approaching either -0.50 or 0.50 are usually considered relatively significant. It is suggested that you consult a good statistics text for better understanding of the details and assumptions involved with regression analysis and correlation. It needs to be noted that the primary finding of this study regarding the relationship between advantagement-disadvantagement and OPT district performance is r=0.80, a signifcantly high correlation by any statistical standards.

What are residuals?
      A residual is the difference between what the linear regression predicts a given value will be and what the value actually is based upon the line generated by the mathematics of linear regression. It is essentially the mathematical distance of a data point above or below the regression line. In the case of this study, district residuals from the Presage Score/Percent Passing regression are used to postulate actual performance. Doing this give us some idea of performance controlling for the Presage Factor.

What exactly is a z-Score and why use it?
      A Z-score (often called a "standard score") is a transformation of a raw score into standard deviation units. Using z-scores allows us to immediately know how far above or below the mean is any given score, thus allowing us to visualze how extreme the score is relative to all other scores. The mean of any z-score distribution is always zero. Using z-scores does not alter the distribution of scores in any way. Converting to z-scores is a linear transformation and does not change the the results of the data analysis at all.
      The advantage of the z-score is in allowing us to understand one score relative to other scores. For example the Presage score as a raw score for Younstown City School District is -173.08 which does not tell us how extreme the disadvantagement is. The Presage z-score for Youngstown is -3.82 which tells us that it is 3.82 standard deviations below the State average, thus allowing us to see that Youngstown's students are very deeply in extreme poverty.

What exactly is standard deviation?
      Most simply put, standard deviation desribes how a set of scores are distributed around the mean of the set. For use in this study, basic knowledge of standard deviation is helpful in reading and understanding the z-scores. Z-scores tell us how many standard deviations above or below the mean a score is. Z-scores greater than 1.0 or lower than -1.0 suggest more significant scores beyond those within 1.0 and -1.0. In the case of reasonably normal distributions such as with the data in this study, approximately 68% of the scores will fall within the 1.0 and -1.0 range of the first standard deviation and 95% of the scores will fall within the limits of the second standard deviation. Scores in the third standard deviation may be thought of as being extreme. Thus, the example of Youngstown given above as having a Presage z-score of -3.82 tells us that it is a case of children living in extreme poverty environments.

How significant or powerful are the findings?
      The correlation between the measure of advantagement-disadvantagement (Presage Factor) and OPT performance are extremely high (r=0.80). Indeed, these findings about this relationship are about as high as are ever found in social science research. . . the findings are very significant both statistically, conceptually, and practically.

Can OPT scores be raised through school interventions?
      The question as to whether OPT scores can be raised can certainly be answered in the affirmative. However, any educational imperative to do so must not be based on an invalid test nor must it be directed toward any form of high stakes testing. Instead, it must be driven by the vision of empowerment, the idea that what students are taught in schools must be personally experienced by the students. Knowledge must be taught in such a manner that it is felt as relevant and usable in the mind of the learner. To empower learners requires constructing learning activities that become personally felt lived experiences for the students in the classrooms, not abstract rote exercises over facts and ideas that the students perceive as meaningless and irrelevant. The usability of academic knowledge must be taught by the teachers and must be experienced by the students if we are to empower learners and raise scores significantly.

What do the findings tell us about the valididty of the OPT as an assessement of academic achievment?
      The findings tell us that OPT performance is in no manner a valid measure of academic achievment: The OPT measures almost exclusively only the quality of life in which the students of the district live.

What do these findings suggest about the validity of the Ohio School Report Card?
      The findings tell us that the Ohio School Report Card, because it is almost entirely based upon OPT performance, is a totally invalid assessment of actual school district performance and should not be used. OSCR is extremely misleading and the general public should be outraged about its use. Likewise, the State Legislature and Governor should be held accountable for misleading the citizens of Ohio and using state monies for such an invalid assessment of school district performance.

What do the findings tell us about accountabilty on the part of districts, administrators, teachers, and Ohio's public school pupils?
      Accountablity is the least understood term in the American political lexicon. For true accountablity to be invoked we must understand that valid accountablity is a function of the decision latitude held by those being held accountable. In other words, it is wrong to hold districts, administrators, teachers, or students accountable for a test that measures variables over which they have absolutely no control whatsoever. This study proves beyond the shadow of any doubt that the OPT is not a measure of anything related to in-school variables; it is a measure of non-school variables, forces, and factors. Therefore to hold those associated with schools accountable for OPT performance is absurd and wrong. It is tantamount to holding the TV weather person accountable for today's weather.

From this study is it possible to assess with some validity the actual levels of Ohio school district performance?
      The answer here is both yes and no. It is "yes" in terms of knowing that the Presage Factor is so very powerful that if we control for its effects, we begin to get a much clearer and certainly much more valid picture of how each district is actually performing. It is "no" in the sense that this performance even controlling for the Presage variables still is primarily based upon the OPT itself. To assess school district performance using the OPT would be foolish and wrong in that it is the public school student who suffers most from the test. In other words, why hurt and mislead the children and parents of Ohio to assess district performance using an invalid test.

When will the results be made public?
      Sometime between late Feburary and late April of 2000 there will be a press release to the Ohio media and the findings will be presented at Ohio's Teaching Learning Conference in Columbus. Copies of the final study will be available following the date of the press release.

Return to the Graphs and Data Table of Contents