Supervised machine learning in multimodal learning analytics for estimating success in project‐based learning

Multimodal learning analytics provides researchers new tools and techniques to capture different types of data from complex learning activities in dynamic learning environments. This paper investigates the use of diverse sensors, including computer vision, user‐generated content, and data from the learning objects (physical computing components), to record high‐fidelity synchronised multimodal recordings of small groups of learners interacting. We processed and extracted different aspects of the students' interactions to answer the following question: Which features of student group work are good predictors of team success in open‐ended tasks with physical computing? To answer this question, we have explored different supervised machine learning approaches (traditional and deep learning techniques) to analyse the data coming from multiple sources. The results illustrate that state‐of‐the‐art computational techniques can be used to generate insights into the "black box" of learning in students' project‐based activities. The features identified from the analysis show that distance between learners' hands and faces is a strong predictor of students' artefact quality, which can indicate the value of student collaboration. Our research shows that new and promising approaches such as neural networks, and more traditional regression approaches can both be used to classify multimodal learning analytics data, and both have advantages and disadvantages depending on the research questions and contexts being investigated. The work presented here is a significant contribution towards developing techniques to automatically identify the key aspects of students success in project‐based learning environments, and to ultimately help teachers provide appropriate and timely support to students in these fundamental aspects.     

From signals to knowledge: A conceptual model for multimodal learning analytics

Multimodality in learning analytics and learning science is under the spotlight. The landscape of sensors and wearable trackers that can be used for learning support is evolving rapidly, as well as data collection and analysis methods. Multimodal data can now be collected and processed in real time at an unprecedented scale. With sensors, it is possible to capture observable events of the learning process such as learner's behaviour and the learning context. The learning process, however, consists also of latent attributes, such as the learner's cognitions or emotions. These attributes are unobservable to sensors and need to be elicited by human‐driven interpretations. We conducted a literature survey of experiments using multimodal data to frame the young research field of multimodal learning analytics. The survey explored the multimodal data used in related studies (the input space) and the learning theories selected (the hypothesis space). The survey led to the formulation of the Multimodal Learning Analytics Model whose main objectives are of (O1) mapping the use of multimodal data to enhance the feedback in a learning context; (O2) showing how to combine machine learning with multimodal data; and (O3) aligning the terminology used in the field of machine learning and learning science.     

Learning linkages: Integrating data streams of multiple modalities and timescales

Increasingly, student work is being conducted on computers and online, producing vast amounts of learning‐related data. The educational analytics fields have produced many insights about learning based solely on tutoring systems' automatically logged data, or “log data.” But log data leave out important contextual information about the learning experience. For example, a student working at a computer might be working independently with few outside influences. Alternatively, he or she might be in a lively classroom, with other students around, talking and offering suggestions. Tools that capture these other experiences have potential to augment and complement log data. However, the collection of rich, multimodal data streams and the increased complexity and heterogeneity in the resulting data pose many challenges to researchers. Here, we present two empirical studies that take advantage of multimodal data sources to enrich our understanding of student learning. We leverage and extend quantitative models of student learning to incorporate insights derived jointly from data collected in multiple modalities (log data, video, and high‐fidelity audio) and contexts (individual vs. collaborative classroom learning). We discuss the unique benefits of multimodal data and present methods that take advantage of such benefits while easing the burden on researchers' time and effort.     

Exploring Online Learners' Interactive Dynamics by Visually Analyzing Their Time‐anchored Comments

MOOCs (Massive Open Online Courses) are increasingly prevalent as an online educational resource open to everyone and have attracted hundreds of thousands learners enrolling these online courses. At such scale, there is potentially rich information of learners' behaviors embedded in the interactions between learners and videos that may help instructors and content producers adjust the instructions and refine the online courses. However, the lack of tools to visualize information from interactive data, including messages left to the videos at particular timestamps as well as the temporal variations of learners' online participation and perceived experience, has prevented people from gaining more insights from video‐watching logs. In this paper, we focus on extracting and visualizing useful information from time‐anchored comments that learners left to specific time points of the videos when watching them. Timestamps as a kind of metadata of messages can be useful to recover the interactive dynamics of learners occurring around the videos. Therefore, we present a visualization system to analyze and categorize time‐anchored comments based on topics and content types. Our system integrates visualization methods of temporal text data, namely ToPIN and ThemeRiver, which can help people understand the quality and quantity of online learners' feedback and their states of learning. To evaluate the proposed system, we visualized time‐anchored commenting data from two online course videos, and conducted two user studies participated by course instructors and third‐party educational evaluators. The results validate the usefulness of the approach and show how the quantitative and qualitative visualizations can be used to gain interesting insights around learners' online learning behaviors.     

The impact of on-demand metacognitive help on effortful behaviour: A longitudinal study using task-related visual analytics

This longitudinal study investigates the differences in learners' effortful behaviour over time due to receiving metacognitive help—in the form of on-demand task-related visual analytics. Specifically, learners' interactions (N = 67) with the tasks were tracked during four self-assessment activities, conducted at four discrete points in time, over a period of 8 weeks. The considered and coded time points were: (a) prior to providing the metacognitive help; (b) while the task-related visual analytics were available (treatment); (c) after the removal of the treatment; and (d) while the option to receive metacognitive help was available again. To measure learners' effortful behaviour across the self-assessment activities, this study utilized learners' response-times to correctly/wrongly complete the tasks and on-task effort expenditure. The panel data analysis shown that the usage of metacognitive help caused statistically significant changes in learners' effortful behaviour, mostly in the third and fourth phase. Statistically significant changes were detected also in the usage of metacognitive help. These results provide empirical evidence on the benefits of task-related visual analytics to support learners' on-task engagement, and suggest relevant cues on how metacognitive help could be designed and prompted by focusing on the “task”, instead of the “self”.     

Interplay of prior knowledge,self‐regulation and motivation in complex multimedia learning environments

This study examined the direct and indirect effects of medical clerkship students' prior knowledge, self‐regulation and motivation on learning performance in complex multimedia learning environments. The data from 386 medical clerkship students from six medical schools were analysed using structural equation modeling. The structural model revealed that medical students' prior knowledge directly positively affected their learning outcome, self‐efficacy and performance approach goal orientation. The learners' self‐regulation had a significant positive direct effect on learning outcome. The learners' mastery goal orientation directly affected their learning outcome. Interestingly, inconsistent with our hypothesis, the learners' performance approach goal orientation showed a significant negative direct effect on learning outcome, and performance avoidance goal orientation had a significant positive effect on learning outcome. These findings help develop a more comprehensive understanding of the role of individual characteristics on learning performance of complex tasks in multimedia learning environments.     

A web-based tutoring system with styles-matching strategy for spatial geometric transformation

It has been a major objective for researchers to develop computer systems that can effectively deliver instruction to learners. Therefore, how to incorporate instructional strategies in computer-assisted learning systems in a systematic manner deserves further investigation. In this paper, a style-matching strategy that attempts to match learning materials' styles to learners' latent traits is proposed and realized in a web-based tutoring system, called CooTutor. The mechanism of adaptive material selection takes learners' different spatial ability and learning styles as an integral learning profile into account, and performs traits-based personalization of learning experience. This system is specifically designed to conquer the difficulty of tutoring the topic on fundamental spatial geometry in conventional curriculums. By conducting empirical evaluation with a small group of students, it is found that CooTutor is generally beneficial to learning the domain, but the effect of the styles-matching mechanism remains inconclusive. The work aims to contribute to the community of adaptive hypermedia in providing an explorative example adopting the concern of individual difference for personalization. The system design, a usage scenario, and an exploratory evaluation are presented in this paper as implications for further studies.     

Enhancing multimodal learning through personalized gesture recognition

Gestural recognition systems are important tools for leveraging movement‐based interactions in multimodal learning environments but personalizing these interactions has proven difficult. We offer an adaptable model that uses multimodal analytics, enabling students to define their physical interactions with computer‐assisted learning environments. We argue that these interactions are foundational to developing stronger connections between students' physical actions and digital representations within a multimodal space. Our model uses real time learning analytics for gesture recognition, training a hierarchical hidden‐Markov model with a “one‐shot” construct, learning from user‐defined gestures, and accessing 3 different modes of data: skeleton positions, kinematics features, and internal model parameters. Through an empirical comparison with a “pretrained” model, we show that our model can achieve a higher recognition accuracy in repeatability and recall tasks. This suggests that our approach is a promising way to create productive experiences with gesture‐based educational simulations, promoting personalized interfaces, and analytics of multimodal learning scenarios.     

Language learners' digital literacies: Focus on students' information literacy and reading practices online

The widespread use of technology and the Internet have changed many of language learners' everyday practices, including literacies. While there have been many studies with the focus on language learners' digital literacies, few, however, have explored language learners' digital information literacy and online reading practices with the use of social bookmarking tools, especially in a community college setting. We address this gap by investigating community-college language learners' digital literacies when social bookmarking with the focus on digital information and online reading practices from an ecological perspective. In this qualitative multiple-case study, the focal participants were five English learners, students in an English as a Second Language writing course in a community college in the northeastern United States. Data collection included interviews, observations, and researchers' e-journals. Thorough within- and cross-case analysis of data shows that language learners searched for digital texts and evaluated them based on relevance, reliability, interest, language, and importance for them and their learning community in the social bookmarking tool. The participants struggled with the number of results in search engines, keywords, and evaluation of digital texts for relevance and reliability. We show the need for more instruction, support, and guidance of language learners' digital information literacy practices as well as the benefits of providing students with opportunities to read digital texts. Our suggestions for future research include investigating the role of multimodality and other factors that influence language learners' evaluation practices when they look for and read information online.     

Designing learner-controlled educational interactions based on learning/cognitive style and learner behaviour

Recently, research in individual differences and in particular, learning and cognitive style, has been used as a basis to consider learner preferences in a web-based educational context. Modelling style in a web-based learning environment demands that developers build a specific framework describing how to design a variety of options for learners with different approaches to learning. In this paper two representative examples of educational systems, Flexi-OLM and INSPIRE, that provide learners a variety of options designed according to specific style categorisations, are presented. Experimental results from two empirical studies performed on the systems to investigate learners' learning and cognitive style, and preferences during interaction, are described. It was found that learners do have a preference regarding their interaction, but no obvious link between style and approaches offered, was detected. Derived from an examination of this experimental data, we suggest that while style information can be used to inform the design of learning environments that accommodate learners' individual differences, it would be wise to recommend interactions based on learners' behaviour. Learning environments should allow learners or learners' interaction behaviour to select or trigger the appropriate approach for the particular learner in the specific context. Alternative approaches towards these directions are also discussed.     

Mobile learning: Two case studies of supporting inquiry learning in informal and semiformal settings

Mobile technologies can support learning across different contexts as their portability enables them to be used by the learner in whichever context she or he is in. They can be particularly beneficial in informal and semiformal contexts where learners have more control over their learning goals and where motivation is often high. Inquiries in informal contexts are likely to be personally relevant in terms of topics of interest and capitalise on learners' location as learners decide what, where, when and whether to learn. There is considerable interest in how such benefits can be harnessed for more formal learning and one challenge is how to make inquiries personally relevant in such contexts. However, there is little literature that considers the structure needed to support informal and semiformal inquiry learning. This paper contributes to that literature by examining dimensions for researchers and designers to consider investigating or developing support for inquiries in informal or semiformal settings.The paper examines two case studies of inquiry learning in contrasting settings in order to understand more about learner control and how technology can support learners' inquiries. Case study one considers the use of web based software to support science inquiry learning by 14–15 year olds in a semiformal context, whilst the second case study reports on informal adult learners using their own mobile technologies to learn about landscape. These case studies are compared and contrasted in terms of the dimensions of learner control, location of learning, and the different support mechanisms for inquiry learning and a framework is proposed for considering these dimensions.     

Role of GSS on collaborative problem-based learning: a study on knowledge externalisation

Based on the success of using Group Support Systems (GSS) to support collaborative work, a growing community of GSS researchers investigates the potential of GSS in business education with special attention to collaborative learning. This paper extends their works and proposes a model of collaborative problem-based learning (CPBL). Based on the proposed model, we explain the effects of GSS on CPBL in general, and externalisation of learners' contributions (initiated ideas, questions and feedback) in particular. A controlled experiment was carried out to investigate the effects of GSS on externalization of the learners' contributions in a CPBL environment. The results indicate that learners in an anonymous GSS-supported CPBL environment externalise more initiated ideas, fewer questions, and fewer but better feedback than those in a non-GSS supported one.     

支持深度学习的视觉数据库管理系统研究进展

计算机视觉因其强大的学习能力,在各种真实场景中得到了广泛应用.随着数据库的发展,利用数据库中成熟的数据管理技术来处理视觉分析应用,已成为一种日益增长的研究趋势.图像、视频和文本等多模态数据的相互融合处理,也促进了视觉分析应用的多样性和准确性.近年来,因深度学习的兴起,支持深度学习的视觉分析应用开始受到广泛关注.然而,传统的数据库管理技术在深度学习场景下面临着复杂视觉分析语义难以表达、应用执行效率低等问题.因此,支持深度学习的视觉数据库管理系统得到了广泛关注.综述了目前视觉数据库管理系统的研究进展:首先,总结了视觉数据库管理系统在不同层面上面临的挑战,包括编程接口、查询优化、执行调度和数据存储;其次,分别探讨了上述4个层面上的相关技术;最后,对视觉数据库管理系统未来的研究方向进行了展望.     

Assessing the effects of different multimedia materials on emotions and learning performance for visual and verbal style learners

Multimedia materials are now increasingly used in curricula. However, individual preferences for multimedia materials based on visual and verbal cognitive styles may affect learners' emotions and performance. Therefore, in-depth studies that investigate how different multimedia materials affect learning performance and the emotions of learners with visual and verbal cognitive styles are needed. Additionally, many education scholars have argued that emotions directly affect learning performance. Therefore, a further study that confirms the relationships between learners' emotions and performance for learners with visual and verbal cognitive styles will provide useful knowledge in terms of designing an emotion-based adaptive multimedia learning system for supporting personalized learning. To investigate these issues, the study applies the Style of Processing (SOP) scale to identify verbalizers and visualizers. Moreover, the emotion assessment instrument emWave, which was developed by HeartMath, is applied to assess variations in emotional states for verbalizers and visualizers during learning processes. Three different multimedia materials, static text and image-based multimedia material, video-based multimedia material, and animated interactive multimedia material, were presented to verbalizers and visualizers to investigate how different multimedia materials affect individual learning performance and emotion, and to identify relationships between learning performance and emotion. Experimental results show that video-based multimedia material generates the best learning performance and most positive emotion for verbalizers. Moreover, dynamic multimedia materials containing video and animation are more appropriate for visualizers than static multimedia materials containing text and image. Finally, a partial correlation exists between negative emotion and learning performance; that is, negative emotion and pretest scores considered together and negative emotion alone can predict learning performance of visualizers who use video-based multimedia material for learning.     

Assessing the effects of different multimedia materials on emotions and learning performance for visual and verbal style learners

Multimedia materials are now increasingly used in curricula. However, individual preferences for multimedia materials based on visual and verbal cognitive styles may affect learners' emotions and performance. Therefore, in-depth studies that investigate how different multimedia materials affect learning performance and the emotions of learners with visual and verbal cognitive styles are needed. Additionally, many education scholars have argued that emotions directly affect learning performance. Therefore, a further study that confirms the relationships between learners' emotions and performance for learners with visual and verbal cognitive styles will provide useful knowledge in terms of designing an emotion-based adaptive multimedia learning system for supporting personalized learning. To investigate these issues, the study applies the Style of Processing (SOP) scale to identify verbalizers and visualizers. Moreover, the emotion assessment instrument emWave, which was developed by HeartMath, is applied to assess variations in emotional states for verbalizers and visualizers during learning processes. Three different multimedia materials, static text and image-based multimedia material, video-based multimedia material, and animated interactive multimedia material, were presented to verbalizers and visualizers to investigate how different multimedia materials affect individual learning performance and emotion, and to identify relationships between learning performance and emotion. Experimental results show that video-based multimedia material generates the best learning performance and most positive emotion for verbalizers. Moreover, dynamic multimedia materials containing video and animation are more appropriate for visualizers than static multimedia materials containing text and image. Finally, a partial correlation exists between negative emotion and learning performance; that is, negative emotion and pretest scores considered together and negative emotion alone can predict learning performance of visualizers who use video-based multimedia material for learning.     

Assessing implicit science learning in digital games

Building on the promise shown in game-based learning research, this paper explores methods for Game-Based Learning Assessments (GBLA) using a variety of educational data mining techniques (EDM). GBLA research examines patterns of behaviors evident in game data logs for the measurement of implicit learning—the development of unarticulated knowledge that is not yet expressible on a test or formal assessment. This paper reports on the study of two digital games showing how the combination of human coding with EDM has enabled researchers to measure implicit learning of Physics. In the game Impulse, researchers combined human coding of video with educational data mining to create a set of automated detectors of students' implicit understanding of Newtonian mechanics. For Quantum Spectre, an optics puzzle game, human coding of Interaction Networks was used to identify common student errors. Findings show that several of our measures of student implicit learning within these games were significantly correlated with improvements in external postassessments. Methods and detailed findings were different for each type of game. These results suggest GBLA shows promise for future work such as adaptive games and in-class, data-driven formative assessments, but design of the assessment mechanics must be carefully crafted for each game.     

Experimenting with electromagnetism using augmented reality: Impact on flow student experience and educational effectiveness

Educational researchers have recognized Augmented Reality (AR) as a technology with great potential to impact affective and cognitive learning outcomes. However, very little work has been carried out to substantiate these claims. The purpose of this study was to assess to which extent an AR learning application affects learners' level of enjoyment and learning effectiveness. The study followed an experimental/control group design using the type of the application (AR-based, web-based) as independent variable. 64 high school students were randomly assigned to the experimental or control group to learn the basic principles of electromagnetism. The participants' knowledge acquisition was evaluated by comparing pre- and post-tests. The participants' level overall-state perception on flow was measured with the Flow State Scale and their flow states were monitored throughout the learning activity. Finally, participants' perceptions of benefits and difficulties of using the augmented reality application in this study were qualitatively identified. The results showed that the augmented reality approach was more effective in promoting students' knowledge of electromagnetic concepts and phenomena. The analysis also indicated that the augmented reality application led participants to reach higher flow experience levels than those achieved by users of the web-based application. However, not all the factors seem to have influence on learners' flow state, this study found that they were limited to: concentration, distorted sense of time, sense of control, clearer direct feedback, and autotelic experience. A deeper analysis of the flow process showed that neither of the groups reported being in flow in those tasks that were very easy or too difficult. However, for those tasks that were not perceived as difficult and included visualization clues, the experimental group showed higher levels of flow that the control group. The study suggests that augmented reality can be exploited as an effective learning environment for learning the basic principles of electromagnetism at high school provided that learning designers strike a careful balance between AR support and task difficulty.     

Using augmented reality to support a software editing course for college students

This study aimed to explore whether integrating augmented reality (AR) techniques could support a software editing course and to examine the different learning effects for students using online‐based and AR‐based blended learning strategies. The researcher adopted a comparative research approach with a total of 103 college students participating in the study. The experimental group (E.G.) learned with the AR‐based contents, while the control group (C.G.) learned with the online‐based support. The findings demonstrated the potential of AR techniques for supporting students' learning motivation and peer learning interaction, and the AR‐based contents could be used as scaffolding to better support blended learning strategies. The AR‐based learning interaction could also be a trigger arousing learners' interest in becoming active learners and the students presented great learning involvement after the AR‐based supports were removed, while the learners in the C.G. were passive once the supports had been removed. Moreover, it was found that (1) their lack of experience with AR interaction and applications, (2) the slow speed of the Internet in the school, (3) the affordances of each learner's mobile learning devices, (4) the screen size of the learning interface and (5) the overloading of the learning information from the AR contents and teacher lectures might be the reasons why the learners were still more used to the online‐based support. It was therefore concluded that when integrating AR applications into a course, technology educational researchers should take into careful consideration the target learning content design, the amount of information displayed on the mobile screen and the affordances of the learning equipment and classroom environment so as to achieve a suitable learning scenario.     

Designing the User Interface for Multimodal Speech and Pen-Based Gesture Applications: State-of-the-Art Systems and Future Research Directions

The growing interest in multimodal interface design is inspired in large part by the goals of supporting more transparent, flexible, efficient, and powerfully expressive means of human-computer interaction than in the past. Multimodal interfaces are expected to support a wider range of diverse applications, be usable by a broader spectrum of the average population, and function more reliably under realistic and challenging usage conditions. In this article, we summarize the emerging architectural approaches for interpreting speech and pen-based gestural input in a robust manner-including early and late fusion approaches, and the new hybrid symbolic-statistical approach. We also describe a diverse collection of state-of-the-art multimodal systems that process users' spoken and gestural input. These applications range from map-based and virtual reality systems for engaging in simulations and training, to field medic systems for mobile use in noisy environments, to web-based transactions and standard text-editing applications that will reshape daily computing and have a significant commercial impact. To realize successful multimodal systems of the future, many key research challenges remain to be addressed. Among these challenges are the development of cognitive theories to guide multimodal system design, and the development of effective natural language processing, dialogue processing, and error-handling techniques. In addition, new multimodal systems will be needed that can function more robustly and adaptively, and with support for collaborative multiperson use. Before this new class of systems can proliferate, toolkits also will be needed to promote software development for both simulated and functioning systems.     

CC–LR: providing interactive,challenging and attractive Collaborative Complex Learning Resources

Many researchers argue that students must be meaningfully engaged in the learning resources for effective learning to occur. However, current online learners still report a problematic lack of attractive and challenging learning resources that engage them in the learning process. This endemic problem is even more evident in online collaborative learning approaches whose resources lack of authentic interactivity, user empowerment, social identity and challenge, thus having a negative effect on learners' self‐motivation and engagement. To overcome these and other limitations and deficiencies, in this paper, a new type of learning resource named Collaborative Complex Learning Resources (CC–LR) is presented based on the virtualization of collaborative learning with the aim of leveraging knowledge elicited during live sessions. During the CC–LR execution, the collaborative sessions are animated so learners can observe how avatars discuss and collaborate, how discussion threads grow and how knowledge is been constructed, refined and consolidated. In addition, complex aspects of the learning process can be incorporated in the CC–LRs during their creation, such as cognitive assessment and emotional awareness. The system produced from this research is tested to evaluate the CC‐LR enriched with complex information and analyze its effects in the discussion process. The research reported in this paper was undertaken within the Seventh Framework Programme (FP7) European project called ‘Adaptive Learning via Intuitive/Interactive, Collaborative and Emotional systems’.