Electronic Task Management System: A Pediatric Institution's Experience

Background  Electronic medical task management systems (ETMs) have been adopted in health care institutions to improve health care provider communication. ETMs allow for the requesting and resolution of nonurgent tasks between clinicians of all craft groups. Visibility, ability to provide close-loop feedback, and a digital trail of all decisions and responsible clinicians are key features of ETMs. An embedded ETM within an integrated electronic health record (EHR) was introduced to the Royal Children''s Hospital Melbourne on April 30, 2016. The ETM is used hospital-wide for nonurgent tasks 24 hours a day. It facilitates communication of nonurgent tasks between clinical staff, with an associated designated timeframe in which the task needs to be completed (2, 4, and 8 hours). Objective  This study aims to examine the usage of the ETM at our institution since its inception. Methods  ETM usage data from the first 3 years of use (April 2016 to April 2019) were extracted from the EHR. Data collected included age of patient, date and time of task request, ward, unit, type of task, urgency of task, requestor role, and time to completion. Results  A total of 136,481 tasks were placed via the ETM in the study period. There were approximately 125 tasks placed each day (24-hour period). The most common time of task placement was around 6:00 p.m. Task placement peaked at approximately 8 a.m., 2 p.m., and 9 p.m.—consistent with nursing shift change times. In total, 63.16% of tasks were placed outside business hours, indicating predominant usage for after-hours task communication. The ETM was most highly utilized by surgical units. The majority of tasks were ordered by nurses for medical staff to complete (97.01%). A significant proportion (98.79%) of tasks was marked as complete on the ETM, indicating closed-loop feedback after tasks were requested. Conclusion  An ETM function embedded in our EHR has been highly utilized in our institution since its introduction. It has multiple benefits for the clinician in the form of efficiencies in workflow and improvement in communication and also workflow management. By allowing collection, tracking, audit, and prioritization of tasks, it also provides a stream of actionable data for quality-improvement activities.     

Provider Preferences for Patient-Generated Health Data Displays in Pediatric Asthma: A Participatory Design Approach

Objective  There is a lack of evidence on how to best integrate patient-generated health data (PGHD) into electronic health record (EHR) systems in a way that supports provider needs, preferences, and workflows. The purpose of this study was to investigate provider preferences for the graphical display of pediatric asthma PGHD to support decisions and information needs in the outpatient setting. Methods  In December 2019, we conducted a formative evaluation of information display prototypes using an iterative, participatory design process. Using multiple types of PGHD, we created two case-based vignettes for pediatric asthma and designed accompanying displays to support treatment decisions. Semi-structured interviews and questionnaires with six participants were used to evaluate the display usability and determine provider preferences. Results  We identified provider preferences for display features, such as the use of color to indicate different levels of abnormality, the use of patterns to trend PGHD over time, and the display of environmental data. Preferences for display content included the amount of information and the relationship between data elements. Conclusion  Overall, provider preferences for PGHD include a desire for greater detail, additional sources, and visual integration with relevant EHR data. In the design of PGHD displays, it appears that the visual synthesis of multiple PGHD elements facilitates the interpretation of the PGHD. Clinicians likely need more information to make treatment decisions when PGHD displays are introduced into practice. Future work should include the development of interactive interface displays with full integration of PGHD into EHR systems.     

Workflow Automation for a Virtual Hypertension Management Program

Objectives  Hypertension is a modifiable risk factor for numerous comorbidities and treating hypertension can greatly improve health outcomes. We sought to increase the efficiency of a virtual hypertension management program through workflow automation processes. Methods  We developed a customer relationship management (CRM) solution at our institution for the purpose of improving processes and workflow for a virtual hypertension management program and describe here the development, implementation, and initial experience of this CRM system. Results  Notable system features include task automation, patient data capture, multi-channel communication, integration with our electronic health record (EHR), and device integration (for blood pressure cuffs). In the five stages of our program (intake and eligibility screening, enrollment, device configuration/setup, medication titration, and maintenance), we describe some of the key process improvements and workflow automations that are enabled using our CRM platform, like automatic reminders to capture blood pressure data and present these data to our clinical team when ready for clinical decision making. We also describe key limitations of CRM, like balancing out-of-the-box functionality with development flexibility. Among our first group of referred patients, 76% (39/51) preferred email as their communication method, 26/51 (51%) were able to enroll electronically, and 63% of those enrolled (32/51) were able to transmit blood pressure data without phone support. Conclusion  A CRM platform could improve clinical processes through multiple pathways, including workflow automation, multi-channel communication, and device integration. Future work will examine the operational improvements of this health information technology solution as well as assess clinical outcomes.     

Leveraging American College of Obstetricians and Gynecologists Guidelines for Point-of-Care Decision Support in Obstetrics

Background  The American College of Obstetricians and Gynecologists (ACOG) provides numerous narrative documents containing formal recommendations and additional narrative guidance within the text. These guidelines are not intended to provide a complete “care pathway” for patient management, but these elements of guidance can be useful for clinical decision support (CDS) in obstetrical and gynecologic care and could be exposed within electronic health records (EHRs). Unfortunately, narrative guidelines do not easily translate into computable CDS guidance. Objective  This study aimed to describe a method of translating ACOG clinical guidance into clear, implementable items associated with specific obstetrical problems for integration into the EHR. Methods  To translate ACOG clinical guidance in Obstetrics into implementable CDS, we followed a set of steps including selection of documents, establishing a problem list, extraction and classification of recommendations, and assigning tasks to those recommendations. Results  Our search through ACOG clinical guidelines produced over 500 unique documents. After exclusions, and counting only sources relevant to obstetrics, we used 245 documents: 38 practice bulletins, 113 committee opinions, 16 endorsed publications, 1 practice advisory, 2 task force and work group reports, 2 patient education, 2 obstetric care consensus, 60 frequently asked questions (FAQ), 1 women''s health care guidelines, 1 Prolog series, and 9 others (non-ACOG). Recommendations were classified as actionable ( n  = 576), informational ( n  = 493), for in-house summary ( n  = 124), education/counseling ( n  = 170), policy/advocacy ( n  = 33), perioperative care ( n  = 4), delivery recommendations ( n  = 50), peripartum care ( n  = 13), and non-ACOG ( n  = 25). Conclusion  We described a methodology of translating ACOG narrative into a semi-structured format that can be more easily applied as CDS in the EHR. We believe this work can contribute to developing a library of information within ACOG that can be continually updated and disseminated to EHR systems for the most optimal decision support. We will continue documenting our process in developing executable code for decision support.     

Searching of Clinical Trials Made Easier in cBioPortal Using Patients' Genetic and Clinical Profiles

Background  Molecular tumor boards (MTBs) cope with the complexity of an increased usage of genome sequencing data in cancer treatment. As for most of these patients, guideline-based therapy options are exhausted, finding matching clinical trials is crucial. This search process is often performed manually and therefore time consuming and complex due to the heterogeneous and challenging dataset. Objectives  In this study, a prototype for a search tool was developed to demonstrate how cBioPortal as a clinical and genomic patient data source can be integrated with ClinicalTrials.gov, a database of clinical studies to simplify the search for trials based on genetic and clinical data of a patient. The design of this tool should rest on the specific needs of MTB participants and the architecture of the integration should be as lightweight as possible and should not require manual curation of trial data in advance with the goal of quickly and easily finding a matching study. Methods  Based on a requirements analysis, interviewing MTB experts, a prototype was developed. It was further refined using a user-centered development process with multiple feedback loops. Finally, the usability of the application was evaluated with user interviews including the thinking-aloud protocol and the system usability scale (SUS) questionnaire. Results  The integration of ClinicalTrials.gov in cBioPortal is achieved by a new tab in the patient view where the genomic profile for the search is prefilled and additional parameters can be adjusted. These parameters are then used to query the application programming interface (API) of ClinicalTrials.gov. The returned search results subsequently are ranked and presented to the user. The evaluation of the application resulted in an SUS score of 83.5. Conclusion  This work demonstrates the integration of cBioPortal with ClinicalTrials.gov to use clinical and genomic patient data to search for appropriate trials within an MTB.     

Identifying Opportunities for Workflow Automation in Health Care: Lessons Learned from Other Industries

Background  Workflow automation, which involves identifying sequences of tasks that can be streamlined by using technology and modern computing, offers opportunities to address the United States health care system''s challenges with quality, safety, and efficiency. Other industries have successfully implemented workflow automation to address these concerns, and lessons learned from those experiences may inform its application in health care. Objective  Our aim was to identify and synthesize (1) current approaches in workflow automation across industries, (2) opportunities for applying workflow automation in health care, and (3) considerations for designing and implementing workflow automation that may be relevant to health care. Methods  We conducted a targeted review of peer-reviewed and gray literature on automation approaches. We identified relevant databases and terms to conduct the searches across sources and reviewed abstracts to identify 123 relevant articles across 11 disciplines. Results  Workflow automation is used across industries such as finance, manufacturing, and travel to increase efficiency, productivity, and quality. We found automation ranged from low to full automation, and this variation was associated with task and technology characteristics. The level of automation is linked to how well a task is defined, whether a task is repetitive, the degree of human intervention and decision-making required, and the sophistication of available technology. We found that identifying automation goals and assessing whether those goals were reached was critical, and ongoing monitoring and improvement would help to ensure successful automation. Conclusion  Use of workflow automation in other industries can inform automating health care workflows by considering the critical role of people, process, and technology in design, testing, implementation, use, and ongoing monitoring of automated workflows. Insights gained from other industries will inform an interdisciplinary effort by the Office of the National Coordinator for Health Information Technology to outline priorities for advancing health care workflow automation.     

Impact of a Prototype Visualization Tool for New Information in EHR Clinical Documents

Background

EHR clinical document synthesis by clinicians may be time-consuming and error-prone due to the complex organization of narratives, excessive redundancy within documents, and, at times, inadvertent proliferation of data inconsistencies. Development of EHR systems that are easily adaptable to the user’s work processes requires research into visualization techniques that can optimize information synthesis at the point of care.

Objective

To evaluate the effect of a prototype visualization tool for clinically relevant new information on clinicians’ synthesis of EHR clinical documents and to understand how the tool may support future designs of clinical document user interfaces.

Methods

A mixed methods approach to analyze the impact of the visualization tool was used with a sample of eight medical interns as they synthesized EHR clinical documents to accomplish a set of four pre-formed clinical scenarios using a think-aloud protocol.

Results

Differences in the missing (unretrieved) patient information (2.3±1.2 [with the visualization tool] vs. 6.8±1.2 [without the visualization tool], p = 0.08) and accurate inferences (1.3±0.3 vs 2.3±0.3, p = 0.09) were not statistically significant but suggest some improvement with the new information visualization tool. Despite the non-significant difference in total times to task completion (43±4 mins vs 36±4 mins, p = 0.35) we observed shorter times for two scenarios with the visualization tool, suggesting that the time-saving benefits may be more evident with certain clinical processes. Other observed effects of the tool include more intuitive navigation between patient details and increased efforts towards methodical synthesis of clinical documents.

Conclusion

Our study provides some evidence that new information visualization in clinical notes may positively influence synthesis of patient information from EHR clinical documents. Our findings provide groundwork towards a more effective display of EHR clinical documents using advanced visualization applications.     

U.S. COVID-19 State Government Public Dashboards: An Expert Review

Background  In the United States, all 50 state governments deployed publicly viewable dashboards regarding the novel coronavirus disease 2019 (COVID-19) to track and respond to the pandemic. States dashboards, however, reflect idiosyncratic design practices based on their content, function, and visual design and platform. There has been little guidance for what state dashboards should look like or contain, leading to significant variation. Objectives  The primary objective of our study was to catalog how information, system function, and user interface were deployed across the COVID-19 state dashboards. Our secondary objective was to group and characterize the dashboards based on the information we collected using clustering analysis. Methods  For preliminary data collection, we developed a framework to first analyze two dashboards as a group and reach agreement on coding. We subsequently doubled coded the remaining 48 dashboards using the framework and reviewed the coding to reach total consensus. Results  All state dashboards included maps and graphs, most frequently line charts, bar charts, and histograms. The most represented metrics were total deaths, total cases, new cases, laboratory tests, and hospitalization. Decisions on how metrics were aggregated and stratified greatly varied across dashboards. Overall, the dashboards were very interactive with 96% having at least some functionality including tooltips, zooming, or exporting capabilities. For visual design and platform, we noted that the software was dominated by a few major organizations. Our cluster analysis yielded a six-cluster solution, and each cluster provided additional insights about how groups of states engaged in specific practices in dashboard design. Conclusion  Our study indicates that states engaged in dashboard practices that generally aligned with many of the goals set forth by the Centers for Disease Control and Prevention, Essential Public Health Services. We highlight areas where states fall short of these expectations and provide specific design recommendations to address these gaps.     

Optimizing Clinical Monitoring Tools to Enhance Patient Review by Pharmacists

Background  The Clinical Monitoring List (CML) is a real-time scoring system and intervention tool used by Mayo Clinic pharmacists caring for hospitalized patients. Objective  The study aimed to describe the iterative development and implementation of pharmacist clinical monitoring tools within the electronic health record at a multicampus health system enterprise. Methods  Between October 2018 and January 2019, pharmacists across the enterprise were surveyed to determine opportunities and gaps in CML functionality. Responses were received from 39% ( n  = 162) of actively staffing inpatient pharmacists. Survey responses identified three main gaps in CML functionality: (1) the desire for automated checklists of tasks, (2) additional rule logic closely aligning with clinical practice guidelines, and (3) the ability to dismiss and defer rules. The failure mode and effect analysis were used to assess risk areas within the CML. To address identified gaps, two A/B testing pilots were undertaken. The first pilot analyzed the effect of updated CML rule logic on pharmacist satisfaction in the domains of automated checklists and guideline alignment. The second pilot assessed the utility of a Clinical Monitoring Navigator (CMN) functioning in conjunction with the CML to display rules with selections to dismiss or defer rules until a user-specified date. The CMN is a workspace to guide clinical end user workflows; permitting the review and actions to be completed within one screen using EHR functionality. Results  A total of 27 pharmacists across a broad range of practice specialties were selected for two separate two-week pilot tests. Upon pilot completion, participants were surveyed to assess the effect of updates on performance gaps. Conclusion  Findings from the enterprise-wide survey and A/B pilot tests were used to inform final build decisions and planned enterprise-wide updated CML and CMN launch. This project serves as an example of the utility of end-user feedback and pilot testing to inform project decisions, optimize usability, and streamline build activities.     

Building a Learning Health System: Creating an Analytical Workflow for Evidence Generation to Inform Institutional Clinical Care Guidelines

Background  One key aspect of a learning health system (LHS) is utilizing data generated during care delivery to inform clinical care. However, institutional guidelines that utilize observational data are rare and require months to create, making current processes impractical for more urgent scenarios such as those posed by the COVID-19 pandemic. There exists a need to rapidly analyze institutional data to drive guideline creation where evidence from randomized control trials are unavailable. Objectives  This article provides a background on the current state of observational data generation in institutional guideline creation and details our institution''s experience in creating a novel workflow to (1) demonstrate the value of such a workflow, (2) demonstrate a real-world example, and (3) discuss difficulties encountered and future directions. Methods  Utilizing a multidisciplinary team of database specialists, clinicians, and informaticists, we created a workflow for identifying and translating a clinical need into a queryable format in our clinical data warehouse, creating data summaries and feeding this information back into clinical guideline creation. Results  Clinical questions posed by the hospital medicine division were answered in a rapid time frame and informed creation of institutional guidelines for the care of patients with COVID-19. The cost of setting up a workflow, answering the questions, and producing data summaries required around 300 hours of effort and $300,000 USD. Conclusion  A key component of an LHS is the ability to learn from data generated during care delivery. There are rare examples in the literature and we demonstrate one such example along with proposed thoughts of ideal multidisciplinary team formation and deployment.     

Implementing an Application Programming Interface for PROMIS Measures at Three Medical Centers

Background  There is an increasing body of literature advocating for the collection of patient-reported outcomes (PROs) in clinical care. Unfortunately, there are many barriers to integrating PRO measures, particularly computer adaptive tests (CATs), within electronic health records (EHRs), thereby limiting access to advances in PRO measures in clinical care settings. Objective  To address this obstacle, we created and evaluated a software integration of an Application Programming Interface (API) service for administering and scoring Patient-Reported Outcomes Measurement Information System (PROMIS) measures with the EHR system. Methods  We created a RESTful API and evaluated the technical feasibility and impact on clinical workflow at three academic medical centers. Results  Collaborative teams (i.e., clinical, information technology [IT] and administrative staff) performed these integration efforts addressing issues such as software integration as well as impact on clinical workflow. All centers considered their implementation successful based on the high rate of completed PROMIS assessments (between January 2016 and January 2021) and minimal workflow disruptions. Conclusion  These case studies demonstrate not only the feasibility but also the pathway for the integration of PROMIS CATs into the EHR and routine clinical care. All sites utilized diverse teams with support and commitment from institutional leadership, initial implementation in a single clinic, a process for monitoring and optimization, and use of custom software to minimize staff burden and error.     

Use,Impact, Weaknesses,and Advanced Features of Search Functions for Clinical Use in Electronic Health Records: A Scoping Review

Objective  Although vast amounts of patient information are captured in electronic health records (EHRs), effective clinical use of this information is challenging due to inadequate and inefficient access to it at the point of care. The purpose of this study was to conduct a scoping review of the literature on the use of EHR search functions within a single patient''s record in clinical settings to characterize the current state of research on the topic and identify areas for future study. Methods  We conducted a literature search of four databases to identify articles on within-EHR search functions or the use of EHR search function in the context of clinical tasks. After reviewing titles and abstracts and performing a full-text review of selected articles, we included 17 articles in the analysis. We qualitatively identified themes in those articles and synthesized the literature for each theme. Results  Based on the 17 articles analyzed, we delineated four themes: (1) how clinicians use search functions, (2) impact of search functions on clinical workflow, (3) weaknesses of current search functions, and (4) advanced search features. Our review found that search functions generally facilitate patient information retrieval by clinicians and are positively received by users. However, existing search functions have weaknesses, such as yielding false negatives and false positives, which can decrease trust in the results, and requiring a high cognitive load to perform an inclusive search of a patient''s record. Conclusion  Despite the widespread adoption of EHRs, only a limited number of articles describe the use of EHR search functions in a clinical setting, despite evidence that they benefit clinician workflow and productivity. Some of the weaknesses of current search functions may be addressed by enhancing EHR search functions with collaborative filtering.     

A Mobile,Electronic Health Record-Connected Application for Managing Team Workflows in Inpatient Care

Background  Clinical workflows require the ability to synthesize and act on existing and emerging patient information. While offering multiple benefits, in many circumstances electronic health records (EHRs) do not adequately support these needs. Objectives  We sought to design, build, and implement an EHR-connected rounding and handoff tool with real-time data that supports care plan organization and team-based care. This article first describes our process, from ideation and development through implementation; and second, the research findings of objective use, efficacy, and efficiency, along with qualitative assessments of user experience. Methods  Guided by user-centered design and Agile development methodologies, our interdisciplinary team designed and built Carelign as a responsive web application, accessible from any mobile or desktop device, that gathers and integrates data from a health care institution''s information systems. Implementation and iterative improvements spanned January to July 2016. We assessed acceptance via usage metrics, user observations, time–motion studies, and user surveys. Results  By July 2016, Carelign was implemented on 152 of 169 total inpatient services across three hospitals staffing 1,616 hospital beds. Acceptance was near-immediate: in July 2016, 3,275 average unique weekly users generated 26,981 average weekly access sessions; these metrics remained steady over the following 4 years. In 2016 and 2018 surveys, users positively rated Carelign''s workflow integration, support of clinical activities, and overall impact on work life. Conclusion  User-focused design, multidisciplinary development teams, and rapid iteration enabled creation, adoption, and sustained use of a patient-centered digital workflow tool that supports diverse users'' and teams'' evolving care plan organization needs.     

An Analysis of Electronic Health Record Work to Manage Asynchronous Clinical Messages among Breast Cancer Care Teams

Objective  Asynchronous messaging is an integral aspect of communication in clinical settings, but imposes additional work and potentially leads to inefficiency. The goal of this study was to describe the time spent using the electronic health record (EHR) to manage asynchronous communication to support breast cancer care coordination. Methods  We analyzed 3 years of audit logs and secure messaging logs from the EHR for care team members involved in breast cancer care at Vanderbilt University Medical Center. To evaluate trends in EHR use, we combined log data into sequences of events that occurred within 15 minutes of any other event by the same employee about the same patient. Results  Our cohort of 9,761 patients were the subject of 430,857 message threads by 7,194 employees over a 3-year period. Breast cancer care team members performed messaging actions in 37.5% of all EHR sessions, averaging 29.8 (standard deviation [SD] = 23.5) messaging sessions per day. Messaging sessions lasted an average of 1.1 (95% confidence interval: 0.99–1.24) minutes longer than nonmessaging sessions. On days when the cancer providers did not otherwise have clinical responsibilities, they still performed messaging actions in an average of 15 (SD = 11.9) sessions per day. Conclusion  At our institution, clinical messaging occurred in 35% of all EHR sessions. Clinical messaging, sometimes viewed as a supporting task of clinical work, is important to delivering and coordinating care across roles. Measuring the electronic work of asynchronous communication among care team members affords the opportunity to systematically identify opportunities to improve employee workload.     

Lessons Learned for Identifying and Annotating Permissions in Clinical Consent Forms

Background  The lack of machine-interpretable representations of consent permissions precludes development of tools that act upon permissions across information ecosystems, at scale. Objectives  To report the process, results, and lessons learned while annotating permissions in clinical consent forms. Methods  We conducted a retrospective analysis of clinical consent forms. We developed an annotation scheme following the MAMA (Model-Annotate-Model-Annotate) cycle and evaluated interannotator agreement (IAA) using observed agreement ( A _o ), weighted kappa ( κ _w ), and Krippendorff''s α . Results  The final dataset included 6,399 sentences from 134 clinical consent forms. Complete agreement was achieved for 5,871 sentences, including 211 positively identified and 5,660 negatively identified as permission-sentences across all three annotators ( A _o  = 0.944, Krippendorff''s α  = 0.599). These values reflect moderate to substantial IAA. Although permission-sentences contain a set of common words and structure, disagreements between annotators are largely explained by lexical variability and ambiguity in sentence meaning. Conclusion  Our findings point to the complexity of identifying permission-sentences within the clinical consent forms. We present our results in light of lessons learned, which may serve as a launching point for developing tools for automated permission extraction.     

Designing and Implementing an IT Architecture for a Digital Multicenter Dementia Registry: digiDEM Bayern

Background  Registries are an essential research tool to investigate the long-term course of diseases and their impact on the affected. The project digiDEM Bayern will set up a prospective dementia registry to collect long-term data of people with dementia and their caregivers in Bavaria (Germany) supported by more than 300 research partners. Objective  The objective of this article is to outline an information technology (IT) architecture for the integration of a registry and comprehensive participant management in a dementia study. Measures to ensure high data quality, study governance, along with data privacy, and security are to be included in the architecture. Methods  The architecture was developed based on an iterative, stakeholder-oriented process. The development was inspired by the Twin Peaks Model that focuses on the codevelopment of requirements and architecture. We gradually moved from a general to a detailed understanding of both the requirements and design through a series of iterations. The experience learned from the pilot phase was integrated into a further iterative process of continuous improvement of the architecture. Results  The infrastructure provides a standardized workflow to support the electronic data collection and trace each participant''s study process. Therefore, the implementation consists of three systems: (1) electronic data capture system for Web-based or offline app-based data collection; (2) participant management system for the administration of the identity data of participants and research partners as well as of the overall study governance process; and (3) videoconferencing software for conducting interviews online. First experiences in the pilot phase have proven the feasibility of the framework. Conclusion  This article outlines an IT architecture to integrate a registry and participant management in a dementia research project. The framework was discussed and developed with the involvement of numerous stakeholders. Due to its adaptability of used software systems, a transfer to other projects should be easily possible.     

Systematic Review of Approaches to Preserve Machine Learning Performance in the Presence of Temporal Dataset Shift in Clinical Medicine

Objective  The change in performance of machine learning models over time as a result of temporal dataset shift is a barrier to machine learning-derived models facilitating decision-making in clinical practice. Our aim was to describe technical procedures used to preserve the performance of machine learning models in the presence of temporal dataset shifts. Methods  Studies were included if they were fully published articles that used machine learning and implemented a procedure to mitigate the effects of temporal dataset shift in a clinical setting. We described how dataset shift was measured, the procedures used to preserve model performance, and their effects. Results  Of 4,457 potentially relevant publications identified, 15 were included. The impact of temporal dataset shift was primarily quantified using changes, usually deterioration, in calibration or discrimination. Calibration deterioration was more common ( n  = 11) than discrimination deterioration ( n  = 3). Mitigation strategies were categorized as model level or feature level. Model-level approaches ( n  = 15) were more common than feature-level approaches ( n  = 2), with the most common approaches being model refitting ( n  = 12), probability calibration ( n  = 7), model updating ( n  = 6), and model selection ( n  = 6). In general, all mitigation strategies were successful at preserving calibration but not uniformly successful in preserving discrimination. Conclusion  There was limited research in preserving the performance of machine learning models in the presence of temporal dataset shift in clinical medicine. Future research could focus on the impact of dataset shift on clinical decision making, benchmark the mitigation strategies on a wider range of datasets and tasks, and identify optimal strategies for specific settings.