Policy issues: Student Performance and Congressional Support

In the first issue of this journal, I wrote about policy issues with which all stakeholders associated with at-risk children and youth should be involved (Carroll, 1996). Continuing in the policy arena, I now speak to student results. The Title I program serves more than 5 million children with a $7 billion appropriation, and school districts need only report to the state the achievement of Title I participants who are tested as part of the annual state assessment program at three grade groupings--Grades 3 to 5, 6 to 8, and 10 to 12. Districts and states are no longer required to conduct pretest and posttest assessments that show the normal curve equivalent growth of children. Instead, adequate yearly progress toward meeting the states' definitions of advanced, proficient, and partially proficient student performance measures is the new yardstick of accountability and program success. These definitions apply no later than the year 2000-2001, when the states must have their student assessments aligned with their content and student performance standards. Even though the new Title I regulations ease up on frequency and coverage of assessment, Title I schools and programs should not. Schools must assess the performance of all their students and show results if we are to garner continued financial and program support from members of Congress and out constituencies at the state and local levels.     

Sampling Variability of Performance Assessments

In this article, performance assessments are cast within a sampling framework. More specifically, a performance assessment is viewed as a sample of student performance drawn from a complex universe defined by a combination of all possible tasks, occasions, raters, and measurement methods. Using generalizability theory, we present evidence bearing on the generalizability and convergent validity of performance assessments sampled from a range of measurement facets and measurement methods. Results at both the individual and school level indicate that task-sampling variability is the major source ofmeasurment error. Large numbers of tasks are needed to get a reliable measure of mathematics and science achievement at the elementary level. With respect to convergent validity, results suggest that methods do not converge. Students' performance scores, then, are dependent on both the task and method sampled.     

The Effect of Ignoring Classroom‐Level Variance in Estimating the Generalizability of School Mean Scores

Contemporary educational accountability systems, including state‐level systems prescribed under No Child Left Behind as well as those envisioned under the “Race to the Top” comprehensive assessment competition, rely on school‐level summaries of student test scores. The precision of these score summaries is almost always evaluated using models that ignore the classroom‐level clustering of students within schools. This paper reports balanced and unbalanced generalizability analyses investigating the consequences of ignoring variation at the level of classrooms within schools when analyzing the reliability of such school‐level accountability measures. Results show that the reliability of school means cannot be determined accurately when classroom‐level effects are ignored. Failure to take between‐classroom variance into account biases generalizability (G) coefficient estimates downward and standard errors (SEs) upward if classroom‐level effects are regarded as fixed, and biases G‐coefficient estimates upward and SEs downward if they are regarded as random. These biases become more severe as the difference between the school‐level intraclass correlation (ICC) and the class‐level ICC increases. School‐accountability systems should be designed so that classroom (or teacher) level variation can be taken into consideration when quantifying the precision of school rankings, and statistical models for school mean score reliability should incorporate this information.     

Alignment and Implications for Test Takers

The use of assessment results to inform school accountability relies on the assumption that the test design appropriately represents the content and cognitive emphasis reflected in the state's standards. Since the passage of the Every Student Succeeds Act and the certification of accountability assessments through federal peer review practices, the content validity arguments supporting accountability have relied almost exclusively on the alignment of statewide assessments to state standards. It is assumed that if alignment does not hold, the scores will not provide valid inferences regarding the degree to which test takers have performed. Although alignment results are commonly used as evidence of test appropriateness, Polikoff (this issue) would argue that given the importance of alignment in policy decisions, research related to alignment is surprisingly limited. Few studies have addressed the adequacy of alignment methodologies and results as support for the inferences to be made (i.e., proficient on state standards). This paper uses an example of test taker performance (and common performance indicators) to investigate to what extent the degree of alignment impacts inferences made about performance (i.e., classification into performance levels, estimates of student ability, and student rank order).     

Developing a Performance Measurement System for University Central Administrative Services

Central administrative services have recently received increasing attention from practitioners and academics due to the challenging need to both manage scarce resources and provide high-quality services. In this context, performance measurement systems (PMSs) may assume a central role, although an unresolved debate remains on the claimed benefits of accountability and the difficulties that have emerged in defining and managing proper measures. This paper contributes to this debate by presenting the results of a study in which a PMS for central administrative services has been developed and tested through an action research approach drawing on actor network theory. The experiment was carried out in 15 Italian universities and five areas of services were dealt with: student support, research support, accounting, human resources, and logistics and procurement. The highly participative method resulted in a comparable system with a complete set of cost and quality indicators across the participating universities. These data proved to be useful at managerial and policy level, providing insights on the presence of scale effects and on the relative importance of quality dimensions for users of services. Participating in the project encouraged the university staff to use indicators in decision making.     

Validity of Inferences from Test-based Educational Accountability Systems

Many states have adopted test-based accountability systems before the No Child Left Behind (NCLB) Act of 2001 became law. NCLB has added a new dimension to test-based educational accountability systems. Features of state and NCLB accountability are discussed with an emphasis on questions of the validity of inferences that are made about school quality. It is concluded that none of the current approaches toe test-based accountability support causal inferences about school quality. It is also shown that tracking progress toward the important goal of closing gaps in achievement requires more than just monitoring changes in the percentages of students who are proficient.     

Performance effects of failure to make Adequate Yearly Progress (AYP): Evidence from a regression discontinuity framework

As the No Child Left Behind (NCLB) law moves through the reauthorization process, it is important to understand the basic performance impacts of its central structure of accountability. In this paper, I examine the effects of failure to make Adequate Yearly Progress (AYP) under NCLB on subsequent student math and reading performance at the school level. Using panel data on Maryland elementary and middle schools from 2003 to 2009, I find that the scope of failure matters: Academic performance suffers in the short run in response to school-wide failure. However, schools that meet achievement targets for the aggregate student group, yet fail to meet at least one demographic subgroup's target see between 3 and 6 percent more students in the failing subgroup score proficiently the following year, compared to if no accountability pressure were in place. I discuss alternative interpretations and policy implications of the main findings.     

The importance of adjusting for pupil background in school value-added models: A study of Progress 8 and school accountability in England

In the UK, USA and elsewhere, school accountability systems increasingly compare schools using value-added measures of school performance derived from pupil scores in high-stakes standardised tests. Rather than naïvely comparing school average scores, which largely reflect school intake differences in prior attainment, these measures attempt to compare the average progress or improvement pupils make during a year or phase of schooling. Schools, however, also differ in terms of their pupil demographic and socioeconomic characteristics and these factors also predict why some schools subsequently score higher than others. Many therefore argue that value-added measures unadjusted for pupil background are biased in favour of schools with more ‘educationally advantaged’ intakes. But others worry that adjusting for pupil background entrenches socioeconomic inequities and excuses low-performing schools. In this article we explore these theoretical arguments and their practical importance in the context of the ‘Progress 8’ secondary school accountability system in England, which has chosen to ignore pupil background. We reveal how the reported low or high performance of many schools changes dramatically once adjustments are made for pupil background, and these changes also affect the reported differential performances of regions and of different school types. We conclude that accountability systems which choose to ignore pupil background are likely to reward and punish the wrong schools and this will likely have detrimental effects on pupil learning. These findings, especially when coupled with more general concerns surrounding high-stakes testing and school value-added models, raise serious doubts about their use in school accountability systems.     

Assessing the Use of Econometric Analysis in Estimating the Costs of Meeting State Education Accountability Standards" Lessons From Texas

In 2004, over 300 school districts in Texas challenged the constitutionality of the Texas system of school finance. In West Orange-Cove et al. v. Neeley et al., the plaintiffs argued that because most school districts were at or near a state-imposed property tax rate ceiling and because the share of state education funding was declining, most school districts had inadequate funds to satisfy the student performance standards mandated by the Texas Educational Accountability system. To address the empirical question of whether school districts have insufficient resources to meet the state's accountability standards, two cost function analyses were conducted. One study, entered into evidence by the state of Texas, reached the conclusion that "in aggregate, the level of education funding in Texas is more than sufficient to meet performance goals consistent with the state's accountability system." The other study, entered into evidence by the plaintiff school districts, concluded that, in aggregate, Texas school districts would need at least $2 billion in additional revenue to satisfy the requirements of the accountability system. In this article we describe the methodological similarities and differences between the two cost function studies and provide an assessment of why the two studies arrive at such different results. Based on the outcome of the case in district court-a victory for the plaintiffs-the article draws some lessons about the use of statistical-based models in a judicial setting.     

At-Risk Students Attending Second Chance Programs: Measuring Performance in Desired Outcome Domains

Meeting the educational needs of at-risk secondary students is a challenge for many states. Some states have blended school reform policies, such as school choice, with specific programs for students at risk of school failure. One blended school choice option that has been in operation for nearly a decade is Minnesota's Second Chance Option. This option allows students at risk of school failure the opportunity to choose alternative programs designed to meet their needs; however, little is known about the effectiveness of these programs. In this article, findings from a study examining a broad array of outcomes for students attending Minnesota's second-chance school choice option for at-risk secondary students are presented. Student performance was assessed in 7 outcome areas, using a model developed by the University of Minnesota's National Center on Educational Outcomes. Students at 3 representative alternative school sites were assessed in the fall and spring of an academic year to determine their level of student performance. This article addresses the performance of persisters (those who remained in the programs) and dropouts, the use of the model for accountability, and the effectiveness of second-chance programs.     

Sample Exclusion and Student Attrition Effects in the Longitudinal Study of Middle School Mathematics Performance

Achievement data from a longitudinally matched student cohort from a large school district in the southwestern United States were analyzed to investigate sample exclusion and student attrition effects on estimates of student, school, and district mathematics performance. Use of 2- and 3-level longitudinal growth models to estimate the growth trajectories of middle school students revealed that mathematics performance differed across 2 sample conditions. Relative to the achievement outcomes associated with a sample that included all students from the longitudinal cohort, district and school achievement were generally higher and student group performance more similar in the smaller, more advantaged student sample used for district accountability reporting. Further investigation of the school performance estimates showed that cross-sample changes in student achievement outcomes were closely related to the proportion of students from special student populations who were excluded from the district accountability sample. The achievement differences and the differential patterns of association demonstrated in this study suggest that conclusions drawn about district and school performance and relationships between student characteristics and student achievement outcomes may depend to some degree on which students are included in an analytic sample. Investigators seeking to take advantage of longitudinal designs in school effectiveness research are cautioned to closely examine their data for nonrandom student attrition and document the impact of sample exclusion and student attrition effects in the research and accountability reports that are produced from longitudinal data sets.     

美国公立高等教育绩效问责制的成效分析

绩效问责制正在成为许多国家分配高校办学和科研经费的一个重要手段。20世纪90年代中期,公立高等教育绩效问责制已经遍布美国的许多州。但是,来自多方面的信息显示,三种形式的绩效问责制都没有对高校的绩效改善产生预期的影响。造成绩效问责制“失效”的原因主要有三个：奖励动力不足、传统的程序性管制的阻碍、难以适用到高校内部基层组织...     

Reflections on “peer acceptance and social play of gifted kindergarten children”

In this article, we review methods of writing assessment, focusing specifically on a method designed for special education: curriculum-based measurement. At the elementary and middle school levels, valid and reliable indicators of performance have been identified. At the high school level, research is sparse and has not yet supported the validity and reliability of the measures. More research is needed on the development of measures at the high school level, on the technical adequacy of the measures as growth indicators, and on the effects of implementation of the measures as they relate to teacher instruction and student performance.     

The ethics of competition: accountability policy enactment in Chilean schools’ everyday life

ABSTRACT

Public policies have a moral order, an ethical horizon. They offer a vocabulary of imagined micro-policies. Using the case of Chile, this paper examines the ways in which accountability policies are reworked within schools and how they affect actors’ subjectivities. It adds new findings to the existing body of research on school accountability policies, offering in-depth evidence based on the case of Chile, which has a high-stakes testing model and a widespread competitive voucher system. The research is based on case studies of ten public and private subsidised schools, framed by a sociological perspective of policy enactment theory. The research findings show the ways that accountability policies are recreated, expanded, and intensified at the local level, permeating an ethic of competition. The analysis focuses on three qualitative trends: school actors’ sense-making of test scores and labels; zones of safety and risk for teachers under an accountability regime; and the emergence of a sticky web of persuasion, surveillance, and coercion among school members in order to improve performance. The practices examined are not understood as ‘secondary effects’ or an ‘implementation problem’, as if they occur unconnected from the policy rationale. The outcomes are consistent with the policy itself in interaction with school life.     

Measurement Error or Meaningful Change? The Consistency of School Achievement in Two School-Based Performance Award Programs

When measures such as student achievement scores are used to hold schools accountable, and in particular to identify some schools as eligible for rewards or sanctions, the reliability or dependability of these measures become an important issues. High levels of random measurement error reduce the dependability with which schools are classified as meeting or not meeting accountability goals, which in turn puts the fairness of school-based performance award programs based on these measures in doubt, and may reduce the motivational value of the rewards or sanctions provided. This paper describes the temporal consistency of school classification observed in the Kentucky, and secondarily, the Charlotte-Mecklenburg, North Carolina, school-based performance award programs. Using information obtained from the Kentucky Department of Education, the degree to which the observed temporal inconsistency of classification in Kentucky could be due to measurement error is evaluated. The paper concludes with a discussion of other potential reasons for the observed temporal inconsistency of school classifications.     

试论美国基础教育学生评价中的高风险测验

在美国的基础教育领域,不管是州级、学区级还是校级的学生评价,都突现出两个特点:测量运动的思想依旧占据很重要的地位;评价与绩效责任紧紧联系在一起.测量运动的余波与绩效责任之风在学生评价领域掀起了新的风浪.这其中,最主要的就是高风险测验的盛行.     

Bureaucratic Control or Professional Autonomy?: Performance management in New Zealand schools

Since 1997 appraisal has been a mandated requirement of New Zealand schools. While the management of teacher performance is not new, schools are increasingly being faced with difficult and complex decisions regarding accountability mechanisms for teacher performance. Moreover, in a climate of school self-management the potential exists for tensions between bureaucratic systems and the professional autonomy of teachers to surface. This article reports on research conducted in 2001 that investigated teachers' perceptions of the bureaucratic and professional approaches to performance management in their schools. In a climate of increasing control of teachers' work and professional activities by the State, results from recent research indicate that school managers have adopted a professional approach to the appraisal of staff. Moreover the involvement of teachers in developing school-level appraisal systems is pinpointed as fundamental to the long-term success of appraisal in New Zealand schools.     

Commentary: Accountability Policy and Scholarly Research

Since 2001, considerations of school reform have been dominated by performance-based accountability. No Child Left Behind (NCLB) has changed the way policymakers and educators talk about education, look at educational performance, and think about educational challenges. Nonetheless, NCLB and the state accountability systems it has spawned have been subjected to little careful scrutiny. This article discusses four recent research contributions and considers how they might inform policymaking on accountability. While scholarly scrutiny will not necessarily settle debates, it can help yield more constructive and informed decisions. In particular, research can clarify the actual consequences of policy decisions; highlight and refine approaches that may be more reliable, stable, and effective than those in use; flag the unanticipated or overlooked effects of design decisions; and ensure that both policymakers and the public are aware of the costs and benefits of accountability.     

A Longitudinal Hierarchical Linear Model for Estimating School Effects and Their Stability

This paper presents a general longitudinal model for estimating school effects and their stability. Previous research on the stability of school performance over successive years has produced inconsistent findings. We argue that the findings have been inconsistent for at least two reasons: researchers have estimated different types of school effects, and they have not distinguished between instability due to true changes in school performance and instability due to measurement and sampling error. We describe two different types of school effects, each relevant to a different policy audience, and we present a longitudinal model that is capable of separating true changes in school effects from sampling and measurement error. The model also provides a means for estimating the effects of school policies and practices while controlling statistically for the effects of factors exogenous to the schooling system. This paper provides an example of the approach based on data describing two cohorts of students from one Education Authority in Scotland. It concludes with a discussion of the limitations of the model and implications for those collecting indicators of school performance for planning and evaluation purposes.