Guidelines for Interpreting and Reporting Subscores

Recent research has proposed a criterion to evaluate the reportability of subscores. This criterion is a value‐added ratio (VAR), where values greater than 1 suggest that the true subscore is better approximated by the observed subscore than by the total score. This research extends the existing literature by quantifying statistical significance and effect size for using VAR to provide practical guidelines for subscore interpretation and reporting. Findings indicate that subscores with VAR ≥ 1.1 are a minimum requirement for a meaningful contribution to a user's score interpretation; subscores with .9 < VAR < 1.1 are redundant with the total score and subscores with VAR ≤ .9 would be misleading to report. Additionally, we discuss what to do when subscores do not add value, yet must be reported, as well as when VAR ≥ 1.1 may be undesirable.     

Equating of Augmented Subscores

Recently, there has been an increasing level of interest in subscores for their potential diagnostic value. Haberman (2008b) suggested reporting an augmented subscore that is a linear combination of a subscore and the total score. Sinharay and Haberman (2008) and Sinharay (2010) showed that augmented subscores often lead to more accurate diagnostic information than subscores. In order to report augmented subscores operationally, they should be comparable across the different forms of a test. One way to achieve comparability is to equate them. We suggest several methods for equating augmented subscores. Results from several operational and simulated data sets show that the error in the equating of augmented subscores appears to be small in most practical situations.     

A Note on Assessing the Added Value of Subscores

Brennan ( 2012 ) noted that users of test scores often want (indeed, demand) that subscores be reported, along with total test scores, for diagnostic purposes. Haberman ( 2008 ) suggested a method based on classical test theory (CTT) to determine if subscores have added value over the total score. According to this method, a subscore has added value if the corresponding true subscore is predicted better by the subscore than by the total score. In this note, parallel‐forms scores are considered. It is proved that another way to interpret the method of Haberman is that a subscore has added value if it is in better agreement than the total score with the corresponding subscore on a parallel form. The suggested interpretation promises to make the method of Haberman more accessible because several practitioners find the concept of parallel forms more acceptable or easier to understand than that of a true score. Results are shown for data from two operational tests.     

When Can We Improve Subscores by Making Them Shorter?: The Case Against Subscores with Overlapping Items

Subscores can be of diagnostic value for tests that cover multiple underlying traits. Some items require knowledge or ability that spans more than a single trait. It is thus natural for such items to be included on more than a single subscore. Subscores only have value if they are reliable enough to justify conclusions drawn from them and if they contain information about the examinee that is distinct from what is in the total test score. In this study we show, for a broad range of conditions of item overlap on subscores, that the value of the subscore is always improved through the removal of such items.     

A New Interpretation of Augmented Subscores and Their Added Value in Terms of Parallel Forms

The value‐added method of Haberman is arguably one of the most popular methods to evaluate the quality of subscores. The method is based on the classical test theory and deems a subscore to be of added value if the subscore predicts the corresponding true subscore better than does the total score. Sinharay provided an interpretation of the added value of subscores in terms of scores and subscores on parallel forms. This article extends the results of Sinharay and considers the prediction of a subscore on a parallel form from both the subscore and the total raw score on the original form. The resulting predictor essentially becomes the augmented subscore suggested by Haberman. The proportional reduction in mean squared error of the resulting predictor is interpreted as a squared multiple correlation coefficient. The practical usefulness of the derived results is demonstrated using an operational data set.     

How Often Do Subscores Have Added Value? Results from Operational and Simulated Data

Recently, there has been an increasing level of interest in subscores for their potential diagnostic value. Haberman suggested a method based on classical test theory to determine whether subscores have added value over total scores. In this article I first provide a rich collection of results regarding when subscores were found to have added value for several operational data sets. Following that I provide results from a detailed simulation study that examines what properties subscores should possess in order to have added value. The results indicate that subscores have to satisfy strict standards of reliability and correlation to have added value. A weighted average of the subscore and the total score was found to have added value more often.     

A Simple Equation to Predict a Subscore's Value

Subscores are often used to indicate test‐takers' relative strengths and weaknesses and so help focus remediation. But a subscore is not worth reporting if it is too unreliable to believe or if it contains no information that is not already contained in the total score. It is possible, through the use of a simple linear equation provided in this note, to determine if a particular subscore adds enough value to be worth reporting.     

Analysis of Added Value of Subscores With Respect to Classification

Brennan noted that users of test scores often want (indeed, demand) that subscores be reported, along with total test scores, for diagnostic purposes. Haberman suggested a method based on classical test theory (CTT) to determine if subscores have added value over the total score. One way to interpret the method is that a subscore has added value only if it has a better agreement than the total score with the corresponding subscore on a parallel form. The focus of this article is on classification of the examinees into “pass” and “fail” (or master and nonmaster) categories based on subscores. A new CTT‐based method is suggested to assess whether classification based on a subscore is in better agreement, than classification based on the total score, with classification based on the corresponding subscore on a parallel form. The method can be considered as an assessment of the added value of subscores with respect to classification. The suggested method is applied to data from several operational tests. The added value of subscores with respect to classification is found to be very similar, except at extreme cutscores, to their added value from a value‐added analysis of Haberman.     

Subscores Based on Classical Test Theory: To Report or Not to Report

There is an increasing interest in reporting subscores, both at examinee level and at aggregate levels. However, it is important to ensure reasonable subscore performance in terms of high reliability and validity to minimize incorrect instructional and remediation decisions. This article employs a statistical measure based on classical test theory that is conceptually similar to the test reliability measure and can be used to determine when subscores have any added value over total scores. The usefulness of subscores is examined both at the level of the examinees and at the level of the institutions that the examinees belong to. The suggested approach is applied to two data sets from a basic skills test. The results provide little support in favor of reporting subscores for either examinees or institutions for the tests studied here.     

Criterion‐Related Validity: Assessing the Value of Subscores

Criterion‐related profile analysis (CPA) can be used to assess whether subscores of a test or test battery account for more criterion variance than does a single total score. Application of CPA to subscore evaluation is described, compared to alternative procedures, and illustrated using SAT data. Considerations other than validity and reliability are discussed, including broad societal goals (e.g., affirmative action), fairness, and ties in expected criterion predictions. In simulation data, CPA results were sensitive to subscore correlations, sample size, and the proportion of criterion‐related variance accounted for by the subscores. CPA can be a useful component in a thorough subscore evaluation encompassing subscore reliability, validity, distinctiveness, fairness, and broader societal goals.     

Exploration of Factors Affecting the Added Value of Test Subscores

Recently, interest in test subscore reporting for diagnosis purposes has been growing rapidly. The two simulation studies here examined factors (sample size, number of subscales, correlation between subscales, and three factors affecting subscore reliability: number of items per subscale, item parameter distribution, and data generating model) that affected the value of reporting subscores within the classical test theory framework. Results showed that a higher proportion of subscores of added value was related to lower correlation between subscales, more items per subscale, no guessing in responses, smaller variability in difficulty parameters, and matched average item difficulty and average examinee ability.     

Comparing Futures or Comparing Pasts?

This article, firstly and briefly, suggests that there is no single or unified 'comparative education' but that there are multiple comparative educations. How may such a variety of comparative educations be distinguished? Rather more importantly and secondly, what might an 'interesting' comparative education constructed in universities look like, and on what criteria would it be interesting? The specific suggestion offered here is that at least one kind of comparative education, for a decade or so, should concentrate on exploring moments of educational metamorphosis, rather than assuming that the equilibrium conditions and the dynamic linearities of development of educational systems can be predicted. Thus for the moment the correct answer to the question, how far can we learn anything of practical value from the study of foreign educational systems is: 'not a lot'. The correct question is, why have we as scholars taken that question so seriously for so long?     

An NCME Instructional Module on Subscores

The purpose of this ITEMS module is to provide an introduction to subscores. First, examples of subscores from an operational test are provided. Then, a review of methods that can be used to examine if subscores have adequate psychometric quality is provided. It is demonstrated, using results from operational and simulated data, that subscores have to be based on a sufficient number of items and have to be sufficiently distinct from each other to have adequate psychometric quality. It is also demonstrated that several operationally reported subscores do not have adequate psychometric quality. Recommendations are made for those interested in reporting subscores for educational tests.     

论新课程标准下数学教师的素质

2l世纪已进入信息社会，新一代公民要有较高的数学素养．作为培养新一代公民的当代数学教师，值此新课程标准实施之际，应如何迎接基础教育大改革新形势的挑战呢?本文试图回答这一问题，提出了对数学教师素质的一些新要求．     

Seeking construct validity grounded in constructivist epistemology: development of the survey of contemporary learning environments

This study focused on the development of a new classroom environment instrument for late-elementary students. The development of the survey of contemporary learning environments (SoCLE) followed a content analysis of three similar instruments on constructivist learning environments and the literature on characteristics of contemporary learning environments. Through a bottom-up development process, grounded in the epistemological foundations of constructivism, we note the difficulty in operationalizing student perceptions of contemporary learning environments and point to challenges in developing well-grounded construct validity. In a field study of 453 fifth–sixth grade students, the SoCLE was defined with three subscales: Reflective Processes (α?=?0.68), Ownership (α?=?0.65), and Multiple Perspectives (α?=?0.53). The Ownership and Multiple Perspectives subscales indicated relationships with overall academic achievement assessed by a nation-wide testing program, the Iowa Tests of Basic Skills. In a multiple linear regression model, ownership predicted the overall composite achievement score, as well as reading comprehension, social studies, and science subscores. Multiple perspectives predicted the language subscore. These findings highlight that characteristics of contemporary learning environments that are epistemologically based may support achievement in late-elementary students.     

Whose Ethics? Toward Clarifying Ethics in Mathematics Education Research

The philosophical branch of ethics is foundationally concerned with the question of right or wrong, benevolent or harmful, and ultimately what is proper conduct. The present inquiry addresses two related questions: (1) How have theories of ethics been applied to mathematics education research? and (2) What alternatives have not been considered? What might the implications be if these alternative formulations were considered? To answer the first question, I offer a review of the philosophy of mathematics education literature, considering those articles which discuss ethics and mathematics education together. The ethical perspectives adopted within the literature span normative and non-normative, modern and postmodern orientations towards ethics. To answer the second question, I explored philosophy literature to identify which philosophical perspectives of ethics have (not) been adopted by philosophers of mathematics education research. The structure of this paper parallels these two questions: the first part considers the philosophy of mathematics education research and how researchers have defined ethics while the second part discusses additional philosophical approaches to ethics and puts those approaches into conversation with those identified in part one. I conclude by intertwining these two strands into my central thesis: ethics per se is construed too narrowly in the philosophy of mathematics education literature and considering additional ethical perspectives from philosophy can be generative of new ideas.     

Detecting Calculator Effects on Item Performance

Calculator effects were examined using methods taken from research on differential item functioning. Use of a calculator was controlled on two experimental forms of a test assembled from operational items used on a standardized university mathematics placement test. Results indicated that calculator effects were not present based on analysis of test scores and in only two of the three subscores composed from homogeneous item types. Analyses of item-level functioning indicated, however, that a number of items, including several not included in the two significant subscore combinations, also contained calculator effects. For those items identified, use of the calculator appeared to have changed the actual objective being tested. The findings were generally consistent with previous research: Items that were easier when a calculator was used required either simple computations or use of a function key on the calculator; items that were more difficult required knowledge of a procedure either with or without additional computation. Analysis at the item level facilitated clearer understanding of the impact of calculator use on measurement of the underlying objective.     

Role of Task-related Factors in Assessing Young Children's Logical Ability in Class Inclusion

One of the ways in which we assess children's logical competence is to ask them questions. This is based on two rather weak assumptions – first, that they understand these questions in the same way adults do and secondly, that logical operations are required to answer these questions. However, since at the start, the child does not know the intentions behind our question, he has to consider the form in which the question is posed and use whatever contextual cues or clues are available to make a correct inference. Relevant information may or may not be contained in the question or the external context. Sometimes, both question and context contain these information, and variation in either can lead to a correct or incorrect answer, independent of the logical abilities theoretically being assessed.     

The Final Ends of Higher Education in Light of an African Moral Theory

From the perspective of an African ethic, analytically interpreted as a philosophical principle of right action, what are the proper final ends of a publicly funded university and how should they be ranked? To answer this question, I first provide a brief but inclusive review of the literature on Africanising higher education from the past 50 years, and contend that the prominent final ends suggested in it can be reduced to five major categories. Then, I spell out an intuitively attractive African moral theory and apply it to these five final ends, arguing that three of them are appropriate but that two of them are not. After that, I maintain that the African moral theory prescribes two additional final ends for a public university that are not salient in the literature. Next, I argue that employing the African moral theory as I do enables one to rebut several criticisms of Africanising higher education that have recently been made from a liberal perspective. I conclude by posing questions suitable for future research.