The assessment of lesson plans in teacher education: a case study in assessment validity and reliability

This paper forms part of an exploration of assessment on one part‐time higher education (HE) course: an in‐service, professional qualification for teachers and trainers in the learning and skills sector which is delivered on a franchise basis across a network of further education colleges in the north of England. This paper proposes that the validity and reliability of portfolio‐based assessment, a key component of many HE programmes in addition to the course being researched here, is contestable. Analysis of the processes of compiling portfolios for assessment, through the conceptual framework of the New Literacy Studies, suggests that the ways in which portfolios are assessed and the ways in which the crucial requisites of validity and reliability are assigned to them, mask complexities and contradictions in their creation by the student. This paper argues for a new, critical analysis of portfolio production and raises a number of questions about the validity, reliability and authenticity of the assessment process that the portfolios reify.     

学校道德氛围问卷的初步编制

通过文献检索、专家访谈、师生访谈等多种途径,编制学校道德氛围调查问卷,为进行学校道德氛围研究提供工具。问卷由“公正”、“关爱”和“宽恕”三个分问卷组成。256名学生参加了问卷调查,结果表明,问卷具有较好的信度和效度。     

Validity and reliability of a Turkish version of the Friendship Activity Scale

The purpose of this study was to develop a valid and reliable Turkish version of the Friendship Activity Scale (FAS). Both the English and Turkish versions of the FAS were administered to 36 students to check for language equivalence. The Turkish version of the FAS was then administered to 226 students to ensure internal consistency, and to 61 students to test re-test reliability. The FAS and Adjective Check List (ACL) were administered to 49 students to check concurrent validity. There was a significant relationship between the English and Turkish versions of the FAS (r?=?.78), and there were no significant differences between the two measurements (p?<?.05). Cronbach Alpha Reliability of the FAS for the Turkish sample was .86. The Intraclass Correlation Coefficient of the repeated measurements of the FAS appeared to be sufficiently high (.89), and there were no significant differences between the mean values of two interval measurements (p?<?.05). The FAS significantly correlated with the ACL (r?=?0.58). The total item correlations of the FAS were acceptable, ranging between 0.26 and 0.65. In conclusion, the FAS appears to be a valid and reliable instrument for measuring the behavioral intentions of Turkish elementary school children, and their commitment to initiating a friendship with a peer with a disability.     

Student evaluation of instruction in higher education: exploring issues of validity and reliability

Many personnel committees at colleges and universities in the USA use student evaluation of faculty instruction to make decisions regarding tenure, promotion, merit pay or faculty professional development. This study examines the construct validity and internal consistency reliability of the student evaluation of instruction (SEI) used at a large mid‐western university in the USA for both administrative and instructional purposes. The sample consisted of 73,500 completed SEIs for undergraduate students who self‐reported as freshman, sophomore, junior or senior. Confirmatory factor analysis via structural equation modelling was used to explore the construct validity of the SEI instrument. The internal consistency of students' ratings was reported to provide reliability evidence. The results of this study showed that the model fits the data for the sample. The significance of this study as well as areas for further research are discussed.     

Predicting child maltreatment: A meta-analysis of the predictive validity of risk assessment instruments

Risk assessment is crucial in preventing child maltreatment since it can identify high-risk cases in need of child protection intervention. Despite widespread use of risk assessment instruments in child welfare, it is unknown how well these instruments predict maltreatment and what instrument characteristics are associated with higher levels of predictive validity. Therefore, a multilevel meta-analysis was conducted to examine the predictive accuracy of (characteristics of) risk assessment instruments. A literature search yielded 30 independent studies (N = 87,329) examining the predictive validity of 27 different risk assessment instruments. From these studies, 67 effect sizes could be extracted. Overall, a medium significant effect was found (AUC = 0.681), indicating a moderate predictive accuracy. Moderator analyses revealed that onset of maltreatment can be better predicted than recurrence of maltreatment, which is a promising finding for early detection and prevention of child maltreatment. In addition, actuarial instruments were found to outperform clinical instruments. To bring risk and needs assessment in child welfare to a higher level, actuarial instruments should be further developed and strengthened by distinguishing risk assessment from needs assessment and by integrating risk assessment with case management.     

Establishing reliability and construct validity for an instrument to measure environmental connectedness

The purpose of this preliminary study is to establish a reliable and valid measure of environmental connectedness (EC) to allow for further exploration of the Swedish Outdoor Recreation in Change national survey data. The Nordic concept of friluftsliv (nature-based outdoor recreation) and the environmental psychology concept of EC are explored to provide a foundation for the research. Reliability and construct validity testing on items from the Outdoor Recreation in Change survey have been tested and have demonstrated both reliability and construct validity. A reliable and construct valid measure of EC will facilitate research into the possible relationships between EC, nature-based outdoor recreation, and environmental behavior. A better understanding of these relationships may serve to further understanding of the human relationship with nature.     

博士学位论文评审指标体系可靠吗——基于全数据的信度和效度研究

论文评审是学位论文质量评价的重要环节。为了使评审结果能真实地反映博士学位论文的质量和水平,必须首先保证论文评审指标体系的准确性、可靠性和有效性。信度和效度分析是验证指标体系准确性、可靠性和有效性的重要方法。文章利用北京师范大学五年博士学位论文匿名评审全数据量化结果,对评审指标体系的信度和效度进行实证研究。结果表明,目前广泛采用的评审指标体系具有良好的信度和效度,能够真实地反映博士学位论文的质量和水平。     

Portfolio assessment: direct from the classroom

Portfolios have been regarded as a means of personal self‐expression. This study reports on student real‐life experiences with portfolio assessment. The focus group comprised 150 freshmen (100 females) from a small campus of a tertiary educational institution. For two semesters (approximately 30 weeks), students engaged in numerous activities selected to encourage deep learning and understanding of mathematical concepts. Because students were not involved in the experiment, ecological validity was maximised, and observations may be regarded as fairly authentic and worthy of analysis. Generally, students reported learning much from portfolio assessment and felt an integral part of the assessment process. Portfolio assessment appeared to empower students and provide them with the self‐respect they desired. Future research could compare results from everyday observations with those from experiments.     

The development of a questionnaire on metacognition for students in higher education

Interest in the role of metacognition has been steadily rising in most forms of education. This study focuses on the construction of a questionnaire for measuring metacognitive knowledge, metacognitive regulation and metacognitive responsiveness among students in higher education and the subsequent process of testing to determine its validity.

Purpose

The aim of the study was to construct an original instrument for measuring features of metacognition, henceforth referred to as the Awareness of Independent Learning Inventory (AILI), and further to establish the similarities and differences between this model and existing instruments for measuring metacognition.

Sample

The AILI questionnaire was distributed to 1058 students in various types of Teacher Training Institutes in the Netherlands and Belgium. The abridged English version of the questionnaire was administered to another sample of 729 students reading Economics and Business Administration at the University of Maastricht in the south of the Netherlands.

Design and methods

The AILI instrument was constructed on the basis of a facet design along two dimensions: components of metacognition and topics of concern to students in higher education. The data gathered with the instrument was analyzed by means of a generalisability study and a decision study, respectively. The validity of the instrument was investigated by using confirmatory factor analysis.

Results

The generalisability study showed that the reliability of the instrument was satisfactory. The decision study revealed that the number of items included in the questionnaire could be reduced substantially by leaving out two components of one of the dimensions in the facet design, without losing too much generalisability. The validity study showed that there was a considerable level of congruity between parts of the AILI questionnaire and the relevant parts of the Motivated Strategies for Learning Questionnaire (MSLQ).

Conclusions

The AILI questionnaire is a reliable and valid instrument for measuring metacognitive knowledge, regulation and responsiveness. It is suitable for use in the evaluation of the effects of interventions that purport to increase metacognitive knowledge, regulation and responsiveness of students in higher education.     

Rethinking assessment: managing behaviour and reducing disaffection

Disaffection and behaviour problems continue to be a cause of concern in many schools. Although many schemes and initiatives have attempted to address these issues, few have proved completely successful. It is suggested that in the light of the current phase of educational reforms, with its emphasis on raising standards and target setting, the role of increased assessment in the generation of these problems may have been overlooked. The experiences of a number of schools in exploring a different approach to pupil assessment emphasising the development of a number of generic ‘life skills’ are explored and the effects on these two issues discussed.     

Promoting prosocial pupil behaviour and the roles of counselling

Antisocial behaviour of pupils isgetting more and more attention in many countries.Information is given about relevant definitions andthe prevalence of antisocial behaviour of pupils inthe Netherlands. Policy strategies and actions toreduce antisocial behaviour at the national level,regional or municipal level, school and class level,and individual or pupil level are focussed. Researchto explain antisocial behaviour and to promoteprosocial behaviour provides clues to a moresystematic prosocial approach, which is formulated infive guidelines towards optimum education. Practiceexperiences in developing optimum education since thebeginning in kindergarten also suggest roles ofcounselling.     

中学生生活技能量表的编制和信效度检验

编制中学生生活技能量表,并检测其信度和效度。方法:用量表评估初高中四个班学生211人,分析量表的信效度。结果:①新编量表有较好的信效度。②我国中学生生活技能水平总体较低。结论:①该量表有较满意的心理测量指标。②需要在我国中学生中开展系统的学校生活技能教育。     

英语口语测试的信度与效度分析

英语口语考试的信度和效度受口试形式、评分标准和考官素质等多方面因素的影响。提高英语口试的效度和信度,需坚持英语口试形式与内容的统一,设计出科学、客观并具有可操作性的评分标准。高信度与效度的英语口语测试对教学具有积极的反拨作用。     

小学校长工作压力源问卷的编制研究

小学校长在新课程改革中起到了举足轻重的作用,他们带领老师们走在这场改革的最前列,承担着种种风险和压力,进行着改革的实践,成为了不起的人物。把问卷调查作为小学校长工作压力源测量工具,对探讨小学校长如何应对工作压力,维护心理健康,具有重要的现实意义。     

The effect of adaptivity on the reliability coefficient in adaptive comparative judgement

Comparative Judgement (CJ) is an increasingly widely investigated method in assessment for creating a scale, for example of the quality of essays. One area that has attracted attention in CJ studies is the optimisation of the selection of pairs of objects for judgement. One approach is known as adaptive comparative judgement (ACJ). It has been claimed in the literature that ACJ produces very high reliability, often higher than can be obtained by conventional marking. Bramley showed by simulation that adaptivity can substantially inflate the apparent reliability in ACJ. The empirical study described here compared an adaptive with a non-adaptive CJ study using GCSE English essays. An all-play-all set of comparisons of a subset of the essays allowed the extent of scale inflation to be quantified: the reported adaptive reliability was 0.97 whereas the deflated value was 0.84. The value from the non-adaptive study was 0.72. However, the scale from the non-adaptive study correlated slightly higher with external variables, suggesting the non-adaptive study was no less valid than the adaptive one.     

Analysis and assessment of the Jordan National Test for Controlling the Quality of Science Instruction (NTCQSI): a Rasch measurement perspective

The purpose of this study was to analyze and assess the Jordan National Test for Controlling the Quality of Science Instruction (NTCQSI) from the perspective provided by Rasch measurement. The test was administered on a stratified random sample that consisted of 41,556 tenth graders from all over Jordan. The test results were saved in a data bank. A random sample of 150 participants' records was selected from this data bank. To address the purpose of this study, a series of analyses were conducted. WINSTEPS and RUMM programs were used for the analysis. The procedures that were used in this paper might be used by worldwide testing agencies to clarify or outline how Rasch measurement may be used to obtain evidence for the validity of inferences of tests data.     

The Conceptions about Teamwork Questionnaire: design,reliability and validity with secondary students

In this study, we present the conceptions about teamwork questionnaire designed to evaluate the conceptions that secondary students have about teamwork. Participants were 309 students aged 15–16 from eight secondary schools, seven from Barcelona and one from Girona (Spain). The original 27-item questionnaire was reduced according to expert assessment and exploratory factor analysis to 20 items related to three conceptions about teamwork: individualistic, complementary and cooperative. By scores of factor analysis, the results show that empirically there are these three conceptions about teamwork with an appropriate level of reliability and of construct validity according to the theoretical hypotheses. There is no gender difference, but there are differences regarding types of school. The results are discussed with an emphasis on the relevance of this questionnaire as an instrument to identify and train secondary students in the conceptions about teamwork.