Online student evaluations of teaching: what are we sacrificing for the affordances of technology?

In the context of increased emphasis on quality assurance of teaching, it is crucial that student evaluations of teaching (SET) methods be both reliable and workable in practice. Online SETs particularly tend to raise criticisms with those most reactive to mechanisms of teaching accountability. However, most studies of SET processes have been conducted with convenience, small and cross-sectional samples. Longitudinal studies are rare, as comparison studies on SET methodological approaches are generally pilot studies followed shortly after by implementation. The investigation presented here significantly contributes to the debate by examining the impact of the online administration method of SET on a very large longitudinal sample at the course level rather than attending to the student unit, thus compensating for the inter-dependency of students’ responses according to the instructor variable. It explores the impact of the administration method of SET (paper based in-class vs. out-of-class online collection) on scores, with a longitudinal sample of over 63,000 student responses collected over a total period of 10 years. Having adjusted for the confounding effect of class size, faculty, year of evaluation, years of teaching experience and student performance, it is observed that the actual effect of the administration method exists, but is insignificant.     

How can we enhance enjoyment of secondary school? The student view

This paper considers enjoyment of formal education for young people aged 14 to 16, largely from their own perspective, based on the view of around 3000 students in England. The data include documentary analysis, official statistics, interviews and surveys with staff and students. Enjoyment of school tends to be promoted by factors such as successful social relationships, small classes, variation in learning and students having some control of their learning. Enjoyment tends to be inhibited by perceived lack of respect or concern by teaching staff and passive pedagogy. For some disengaged students, a work or college environment with more adult relationships appears to restore enjoyment and enthusiasm. Enjoyment, unlike attainment, for example, is not particularly stratified by the standard student background variables. Nor is there evidence of a clear school effect. This means that enjoyment should be easy to enhance more widely, positively affecting the learner identities of all young people, including the more reluctant learners.     

Do gooders or do betters? An analysis of the motivation of student tutors

Across the UK hundreds of undergraduate students are volunteering their time and skills to assist teachers in local schools. This research investigates the factors that motivate already pressured undergraduate students to give up their precious free time for community work. Specifically, it addresses the question: are they primarily seeking to better their community or themselves? This short report argues that the primary motivation for such apparently community‐focused activity among undergraduates, in the south‐east, is actually centred on a concept of self‐improvement. The volunteer student, by undertaking unpaid community work, is engaged in a conscious attempt to enhance his/her chances of success in seeking post‐education employment. The paper goes on to argue that this finding has significant implications for the administration of student tutoring schemes, as recruitment and funding are the two major obstacles to establishing and maintaining an extracurricular activity in higher education (Stadium, 1997). Therefore, when the factors known to motivate undergraduates to undertake additional responsibilities are incorporated into recruitment and fund‐raising strategies, the activities can be tailored to maximize potential.     

Teachers’ perceptions of student evaluations of teaching

Evaluation of teaching in higher education has drawn much attention due to the need for greater accountability and improvement in student learning. Our review of literature on Student Evaluation of Teaching (SET) surveys suggests that considerable controversy and criticism have surrounded its use, fairness, and validity. Yet, many universities in Hong Kong are depending primarily on student feedback in SET surveys to provide evidence for their improvement of teaching and learning, and for their decisions made on teachers' pay-rise and tenure. If universities are to continue to use SET surveys as the main source of mechanism for measuring teaching effectiveness, teachers will have to understand and be aware of its purposes. But how much do we know about teachers' perceptions of teaching and learning and, consequently, their opinions of using SET surveys as the primary standardized evaluation mechanism of teaching for promotion and tenure and for teaching effectiveness?     

How and why do students of higher education participate in online seminars?

Online education is continuing to gain popularity in educational institutions and organizations. Hitherto, most research has occurred at aggregated levels, while few researchers have studied how and why individuals participate in online education. It is essential to examine individual perceptions and relationships in order to understand how students behave in relation to others. This paper investigates how students of higher education participate in online seminars and why they participate in certain ways. An online class that attended asynchronous and synchronous online seminars was studied. Electronic logs were used to examine how students participated and interviews were used to illustrate why they participated. It was revealed that the participation of students varied between aspects such as exchanging information, managing tasks and providing social support and the emphasis of these aspects were related to the tool they communicated through. A number of participation inhibitors were identified and it was also suggested how these inhibitors can be addressed.     

Understanding academics’ resistance towards (online) student evaluation

Many higher educational institutions and academic staff are still sceptical about the validity and reliability of student evaluation questionnaires, in particular when these evaluations are completed online. One month after a university-wide implementation from paper to online evaluation across 629 modules, (perceived) resistance and ambivalence amongst academic staff were unpacked. A mixed-method study was conducted amongst 104 academics using survey methods and follow-up semi-structured interviews. Despite a successful ‘technical’ transition (i.e. response rate of 60%, similar scores to previous evaluations), more than half of respondents reported a negative experience with this transition. The results indicate that the multidimensional nature of ambivalence towards change and the dual nature of student evaluations can influence the effectiveness of organisational transition processes.     

Whose feedback? A multilevel analysis of student completion of end-of-term teaching evaluations

Student evaluation of teaching (SET) is now common practice across higher education, with the results used for both course improvement and quality assurance purposes. While much research has examined the validity of SETs for measuring teaching quality, few studies have investigated the factors that influence student participation in the SET process. This study aimed to address this deficit through the analysis of an SET respondent pool at a large Canadian research-intensive university. The findings were largely consistent with available research (showing influence of student gender, age, specialisation area and final grade on SET completion). However, the study also identified additional influential course-specific factors such as term of study, course year level and course type as statistically significant. Collectively, such findings point to substantively significant patterns of bias in the characteristics of the respondent pool. Further research is needed to specify and quantify the impact (if any) on SET scores. We conclude, however, by recommending that such bias does not invalidate SET implementation, but instead should be embraced and reported within standard institutional practice, allowing better understanding of feedback received, and driving future efforts at recruiting student respondents.     

How do instructional designers evaluate? A qualitative study of evaluation in practice

This study employed a qualitative research design to investigate how instructional designers use evaluation in everyday design practice. While past research has examined how designers spend their time, how they generally make decisions, and expert-novice differences, little attention has been paid to use of context, input, process, or product evaluation, from the perspective of practicing designers. Based on interviews of practitioners, our findings included ten themes regarding how designers use evaluation to improve their products. While these results substantiate to some degree the claim that practitioners believe clients will not pay for formal evaluations, they also suggest that practitioners use evaluation in important but less formal ways. Other conclusions regarding the role of evaluation in design are provided and future directions for training and research are discussed.     

How do students integrate multiple texts? An investigation of top-down processing

European Journal of Psychology of Education - This study describes an in-depth investigation of students’ integration or connection formation across multiple texts. Students were asked to...     

Why bother? Student motivation to participate in student evaluations of teaching

When response rates on student evaluations of teaching (SETs) are low, inability to properly interpret and use responses from the students who do participate is a big problem. Where does the motivation to participate break down, and where and how does it make sense to invest efforts in rectifying that? In this study, we examined 641 university students’ reported behaviours and motivation related to their SET participation. In terms of behaviour, students who seldom or never participate in online SET tools reported a willingness to invest, at most, five minutes in the process, though the majority never even open the online evaluation links when they receive them. In terms of motivation, they significantly differed from students who always participate with distinctly lower levels of: (1) willingness to participate at all, (2) perception of autonomy and competence, (3) meaningfulness, (4) personal value, (5) engagement in others’ participation, and (6) understanding of the value of their own participation for others’ benefit. Based on these findings, we propose a strategy for increasing future response rates, particularly among recalcitrant students, in order to be able to gather sufficient and reliable results for the work of improving teaching.     

What do we know and where do we go? Formative assessment in developing student teachers’ professional learning of teaching science

In the context of teacher education, it could well be suggested that assessment activities that build on formative interactions between student teachers and teacher educators might offer new windows into better understanding teaching and learning. This paper presents findings from a study into a primary science teacher education initiative that seeks to build the foundations on which 24 primary science student teachers, through the use of formative assessment of their science teaching and learning, can begin developing their pedagogical content knowledge (PCK). In the project, formative assessment consists of activities used by teacher educators to stimulate interactions, self- and peer-assessment in order to provide insights into how student teachers develop their PCK during a semester. Content Representations (CoRes), were used as a tool to unpack the student teachers’ approach to teaching a science topic and the reasons for that approach. The results indicate that the use of CoRes, together with subsequent self-assessment and formative interactions with teacher educators and peers, do have the potential for PCK development for student teachers. The results further highlight the need for developing reliable and valid tools for capturing and assessing student teachers’ PCK in pre-service teacher education.     

How do substitute teachers substitute? An empirical study of substitute-teacher labor supply

This paper examines the daily labor supply of a potentially important, but often overlooked, source of instruction in U.S. public schools: substitute teachers. I estimate a sequential binary-choice model of substitute teachers’ job-offer acceptance decisions using data on job offers made by a randomized automated calling system. Importantly, this randomization generates exogenous variation in offer quality. I find that determinants of substitutes’ offer-acceptance decisions include the offer's arrival time, commute time, day of week, classroom type, school type, and school quality. Interestingly, conditional on school quality student demographics do not significantly influence substitutes’ decisions. Longer and higher paying full-day jobs are preferred to half-day jobs, although conditional on daily pay, job length does not significantly impact daily labor-supply decisions. Preferences for several job characteristics are found to vary with substitutes’ regular-teacher certification status. Policy implications of these findings are discussed.     

How do we create ASD‐friendly schools? A dilemma of placement

Scope exists within the Northern Ireland (NI) education system to transform mainstream schools into autistic spectrum disorder (ASD)‐friendly environments. The efficacy of mainstream inclusion is discussed prior to discussing the creation of ASD‐friendly schools. The transformation of mainstream school environments is underpinned by concepts such as inclusive pedagogical approach, universal design for learning and learning without limits. These are discussed alongside strategies to enact core inclusive principles of equity, participation and belonging. However, the need for ASD‐specific approaches and schools is also recognised. Our perspectives as educators influence pedagogy, attitudes and approaches to educating autistic children. A social constructivist consciousness is fundamental to moving from deficit SEN rhetoric to creating enabling education for autistic young people. The question of how to achieve this is answered within this article. The use of identity‐first language is preferred by a large proportion of the autism community; therefore the term ‘autistic child’, rather than ‘child with autism’, is adopted throughout.     

Convenience or credibility? A study of college student online research behaviors

The purpose of this study was to investigate where students turn for course-related assignments, whether an ordered pattern could be described in terms of which sources students turn to and how students evaluated the information they chose to use. Data were drawn from open-ended questionnaires (n = 282). Semantic network analysis was conducted using CATPAC, artificial neural network software. Results verify previous findings that students turn to the Internet before the library, but a deeper investigation revealed different preferences for study versus project-related research. Specifically, using search engines or Wikipedia was a pre-stage, rather than a final destination, for project work. Interestingly, students were relatively confident in their abilities to discern courses using the Internet. Recommendations for promoting information literacy, as well as recommendations for improving library resource use, are included.     

How do teachers learn in the workplace? An examination of teacher learning activities

In this study, two data collection instruments were used to examine how Dutch secondary school teachers learn in the workplace. Firstly, they completed a questionnaire on their preferences for learning activities on two occasions. Secondly, during the intermediate period, they reported learning experiences in digital logs. Results of both instruments indicate that teachers often learn by critical individual reflection and by involving colleagues in particular challenging or problematic situations. An additional finding concerns the reporting of sequences of learning activities in the digital logs which is clearly different from the focus on single learning activities as found in most literature and as used in the questionnaire. Furthermore, the digital logs provided a fine‐tuning of the concepts of ‘involvement of colleagues’ and ‘experimentation’ in relation to teacher learning. The study concludes with a critical reflection on both data collection instruments. Finally, implications for future research on how teachers learn are discussed.     

Teacher and student responses to interdisciplinary aspects of sustainability education: what do we really know?

That the value of interdisciplinarity in sustainability education is largely taken for granted reflects widely held views about the inherent complexity and uncertainty of sustainability issues and the need for holistic responses to them. Although there is an extensive literature on interdisciplinarity, there has been relatively little investigation of the ways in which teachers and learners deal with and respond to this central aspect of sustainability education. This paper presents data from two case studies of masters course in sustainability education, one in an English university and one in a Chinese university, and examines some of the contextual factors that support or limit their attempts to go beyond the bounded ways of thinking associated with more traditional academic disciplines. While some learners respond enthusiastically to interdisciplinarity, others experience forms of ‘cognitive dissonance’. Interdisciplinarity requires individual teachers to operate as members of teams and to find ways of promoting dialogue between different perspectives, discourses and methods of approaching problems. The key to interdisciplinarity is an open attitude, a willingness to learn and an ability to engage with different ways of thinking about issues we face in common.