Use of a support vector machine for categorizing free-text notes: assessment of accuracy across two institutions

Background

Electronic health record (EHR) users must regularly review large amounts of data in order to make informed clinical decisions, and such review is time-consuming and often overwhelming. Technologies like automated summarization tools, EHR search engines and natural language processing have been shown to help clinicians manage this information.

Objective

To develop a support vector machine (SVM)-based system for identifying EHR progress notes pertaining to diabetes, and to validate it at two institutions.

Materials and methods

We retrieved 2000 EHR progress notes from patients with diabetes at the Brigham and Women''s Hospital (1000 for training and 1000 for testing) and another 1000 notes from the University of Texas Physicians (for validation). We manually annotated all notes and trained a SVM using a bag of words approach. We then used the SVM on the testing and validation sets and evaluated its performance with the area under the curve (AUC) and F statistics.

Results

The model accurately identified diabetes-related notes in both the Brigham and Women''s Hospital testing set (AUC=0.956, F=0.934) and the external University of Texas Faculty Physicians validation set (AUC=0.947, F=0.935).

Discussion

Overall, the model we developed was quite accurate. Furthermore, it generalized, without loss of accuracy, to another institution with a different EHR and a distinct patient and provider population.

Conclusions

It is possible to use a SVM-based classifier to identify EHR progress notes pertaining to diabetes, and the model generalizes well.     

Understanding differences in electronic health record (EHR) use: linking individual physicians’ perceptions of uncertainty and EHR use patterns in ambulatory care

Objective

Electronic health records (EHR) hold great promise for managing patient information in ways that improve healthcare delivery. Physicians differ, however, in their use of this health information technology (IT), and these differences are not well understood. The authors study the differences in individual physicians'' EHR use patterns and identify perceptions of uncertainty as an important new variable in understanding EHR use.

Design

Qualitative study using semi-structured interviews and direct observation of physicians (n=28) working in a multispecialty outpatient care organization.

Measurements

We identified physicians'' perceptions of uncertainty as an important variable in understanding differences in EHR use patterns. Drawing on theories from the medical and organizational literatures, we identified three categories of perceptions of uncertainty: reduction, absorption, and hybrid. We used an existing model of EHR use to categorize physician EHR use patterns as high, medium, and low based on degree of feature use, level of EHR-enabled communication, and frequency that EHR use patterns change.

Results

Physicians'' perceptions of uncertainty were distinctly associated with their EHR use patterns. Uncertainty reductionists tended to exhibit high levels of EHR use, uncertainty absorbers tended to exhibit low levels of EHR use, and physicians demonstrating both perspectives of uncertainty (hybrids) tended to exhibit medium levels of EHR use.

Conclusions

We find evidence linking physicians'' perceptions of uncertainty with EHR use patterns. Study findings have implications for health IT research, practice, and policy, particularly in terms of impacting health IT design and implementation efforts in ways that consider differences in physicians'' perceptions of uncertainty.     

Harmonized patient-reported data elements in the electronic health record: supporting meaningful use by primary care action on health behaviors and key psychosocial factors

Background

Electronic health records (EHR) have the potential to improve patient care through efficient access to complete patient health information. This potential may not be reached because many of the most important determinants of health outcome are rarely included. Successful health promotion and disease prevention requires patient-reported data reflecting health behaviors and psychosocial issues. Furthermore, there is a need to harmonize this information across different EHR systems.

Methods

To fill this gap a three-phased process was used to conceptualize, identify and recommend patient-reported data elements on health behaviors and psychosocial factors for the EHR. Expert panels (n=13) identified candidate measures (phase 1) that were reviewed and rated by a wide range of health professionals (n=93) using the grid-enabled measures wiki social media platform (phase 2). Recommendations were finalized through a town hall meeting with key stakeholders including patients, providers, researchers, policy makers, and representatives from healthcare settings (phase 3).

Results

Nine key elements from three areas emerged as the initial critical patient-reported elements to incorporate systematically into EHR—health behaviors (eg, exercise), psychosocial issues (eg, distress), and patient-centered factors (eg, demographics). Recommendations were also made regarding the frequency of collection ranging from a single assessment (eg, demographic characteristics), to annual assessment (eg, health behaviors), or more frequent (eg, patient goals).

Conclusions

There was strong stakeholder support for this initiative reflecting the perceived value of incorporating patient-reported elements into EHR. The next steps will include testing the feasibility of incorporating these elements into the EHR across diverse primary care settings.     

Openness of patients' reporting with use of electronic records: psychiatric clinicians' views

Objectives

Improvements in electronic health record (EHR) system development will require an understanding of psychiatric clinicians'' views on EHR system acceptability, including effects on psychotherapy communications, data-recording behaviors, data accessibility versus security and privacy, data quality and clarity, communications with medical colleagues, and stigma.

Design

Multidisciplinary development of a survey instrument targeting psychiatric clinicians who recently switched to EHR system use, focus group testing, data analysis, and data reliability testing.

Measurements

Survey of 120 university-based, outpatient mental health clinicians, with 56 (47%) responding, conducted 18 months after transition from a paper to an EHR system.

Results

Factor analysis gave nine item groupings that overlapped strongly with five a priori domains. Respondents both praised and criticized the EHR system. A strong majority (81%) felt that open therapeutic communications were preserved. Regarding data quality, content, and privacy, clinicians (63%) were less willing to record highly confidential information and disagreed (83%) with including their own psychiatric records among routinely accessed EHR systems.

Limitations

single time point; single academic medical center clinic setting; modest sample size; lack of prior instrument validation; survey conducted in 2005.

Conclusions

In an academic medical center clinic, the presence of electronic records was not seen as a dramatic impediment to therapeutic communications. Concerns regarding privacy and data security were significant, and may contribute to reluctances to adopt electronic records in other settings. Further study of clinicians'' views and use patterns may be helpful in guiding development and deployment of electronic records systems.     

Validity of electronic health record-derived quality measurement for performance monitoring

Background

Since 2007, New York City''s primary care information project has assisted over 3000 providers to adopt and use a prevention-oriented electronic health record (EHR). Participating practices were taught to re-adjust their workflows to use the EHR built-in population health monitoring tools, including automated quality measures, patient registries and a clinical decision support system. Practices received a comprehensive suite of technical assistance, which included quality improvement, EHR customization and configuration, privacy and security training, and revenue cycle optimization. These services were aimed at helping providers understand how to use their EHR to track and improve the quality of care delivered to patients.

Materials and Methods

Retrospective electronic chart reviews of 4081 patient records across 57 practices were analyzed to determine the validity of EHR-derived quality measures and documented preventive services.

Results

Results from this study show that workflow and documentation habits have a profound impact on EHR-derived quality measures. Compared with the manual review of electronic charts, EHR-derived measures can undercount practice performance, with a disproportionately negative impact on the number of patients captured as receiving a clinical preventive service or meeting a recommended treatment goal.

Conclusion

This study provides a cautionary note in using EHR-derived measurement for public reporting of provider performance or use for payment.     

The impact of the electronic medical record on structure,process, and outcomes within primary care: a systematic review of the evidence

Background

The electronic medical record (EMR)/electronic health record (EHR) is becoming an integral component of many primary-care outpatient practices. Before implementing an EMR/EHR system, primary-care practices should have an understanding of the potential benefits and limitations.

Objective

The objective of this study was to systematically review the recent literature around the impact of the EMR/EHR within primary-care outpatient practices.

Materials and methods

Searches of Medline, EMBASE, CINAHL, ABI Inform, and Cochrane Library were conducted to identify articles published between January 1998 and January 2010. The gray literature and reference lists of included articles were also searched. 30 studies met inclusion criteria.

Results and discussion

The EMR/EHR appears to have structural and process benefits, but the impact on clinical outcomes is less clear. Using Donabedian''s framework, five articles focused on the impact on healthcare structure, 21 explored healthcare process issues, and four focused on health-related outcomes.     

Are Meaningful Use Stage 2 certified EHRs ready for interoperability? Findings from the SMART C-CDA Collaborative

Background and objective

Upgrades to electronic health record (EHR) systems scheduled to be introduced in the USA in 2014 will advance document interoperability between care providers. Specifically, the second stage of the federal incentive program for EHR adoption, known as Meaningful Use, requires use of the Consolidated Clinical Document Architecture (C-CDA) for document exchange. In an effort to examine and improve C-CDA based exchange, the SMART (Substitutable Medical Applications and Reusable Technology) C-CDA Collaborative brought together a group of certified EHR and other health information technology vendors.

Materials and methods

We examined the machine-readable content of collected samples for semantic correctness and consistency. This included parsing with the open-source BlueButton.js tool, testing with a validator used in EHR certification, scoring with an automated open-source tool, and manual inspection. We also conducted group and individual review sessions with participating vendors to understand their interpretation of C-CDA specifications and requirements.

Results

We contacted 107 health information technology organizations and collected 91 C-CDA sample documents from 21 distinct technologies. Manual and automated document inspection led to 615 observations of errors and data expression variation across represented technologies. Based upon our analysis and vendor discussions, we identified 11 specific areas that represent relevant barriers to the interoperability of C-CDA documents.

Conclusions

We identified errors and permissible heterogeneity in C-CDA documents that will limit semantic interoperability. Our findings also point to several practical opportunities to improve C-CDA document quality and exchange in the coming years.     

Are physicians' perceptions of healthcare quality and practice satisfaction affected by errors associated with electronic health record use?

Background

Electronic health record (EHR) adoption is a national priority in the USA, and well-designed EHRs have the potential to improve quality and safety. However, physicians are reluctant to implement EHRs due to financial constraints, usability concerns, and apprehension about unintended consequences, including the introduction of medical errors related to EHR use. The goal of this study was to characterize and describe physicians'' attitudes towards three consequences of EHR implementation: (1) the potential for EHRs to introduce new errors; (2) improvements in healthcare quality; and (3) changes in overall physician satisfaction.

Methods

Using data from a 2007 statewide survey of Massachusetts physicians, we conducted multivariate regression analysis to examine relationships between practice characteristics, perceptions of EHR-related errors, perceptions of healthcare quality, and overall physician satisfaction.

Results

30% of physicians agreed that EHRs create new opportunities for error, but only 2% believed their EHR has created more errors than it prevented. With respect to perceptions of quality, there was no significant association between perceptions of EHR-associated errors and perceptions of EHR-associated changes in healthcare quality. Finally, physicians who believed that EHRs created new opportunities for error were less likely be satisfied with their practice situation (adjusted OR 0.49, p=0.001).

Conclusions

Almost one third of physicians perceived that EHRs create new opportunities for error. This perception was associated with lower levels of physician satisfaction.     

Research and applications: Primary care practitioners’ views on test result management in EHR-enabled health systems: a national survey

Context

Failure to notify patients of test results is common even when electronic health records (EHRs) are used to report results to practitioners. We sought to understand the broad range of social and technical factors that affect test result management in an integrated EHR-based health system.

Methods

Between June and November 2010, we conducted a cross-sectional, web-based survey of all primary care practitioners (PCPs) within the Department of Veterans Affairs nationwide. Survey development was guided by a socio-technical model describing multiple inter-related dimensions of EHR use.

Findings

Of 5001 PCPs invited, 2590 (51.8%) responded. 55.5% believed that the EHRs did not have convenient features for notifying patients of test results. Over a third (37.9%) reported having staff support needed for notifying patients of test results. Many relied on the patient''s next visit to notify them for normal (46.1%) and abnormal results (20.1%). Only 45.7% reported receiving adequate training on using the EHR notification system and 35.1% reported having an assigned contact for technical assistance with the EHR; most received help from colleagues (60.4%). A majority (85.6%) stayed after hours or came in on weekends to address notifications; less than a third reported receiving protected time (30.1%). PCPs strongly endorsed several new features to improve test result management, including better tracking and visualization of result notifications.

Conclusions

Despite an advanced EHR, both social and technical challenges exist in ensuring notification of test results to practitioners and patients. Current EHR technology requires significant improvement in order to avoid similar challenges elsewhere.     

The marginal value of pre-visit paper reminders when added to a multifaceted electronic health record based quality improvement system

Objective

We have reported that implementation of an electronic health record (EHR) based quality improvement system that included point-of-care electronic reminders accelerated improvement in performance for multiple measures of chronic disease care and preventive care during a 1-year period. This study examined whether providing pre-visit paper quality reminders could further improve performance, especially for physicians whose performance had not improved much during the first year.

Design

Time-series analysis at a large internal medicine practice using a commercial EHR. All patients eligible for each measure were included (range approximately 100–7500).

Measurements

The proportion of eligible patients in the practice who satisfied each of 15 quality measures after removing those with exceptions from the denominator. To analyze changes in performance for individual physicians, two composite measures were used: prescribing seven essential medications and completion of five preventive services.

Results

During the year after implementing pre-encounter reminders, performance continued to improve for eight measures, remained stable for four, and declined for three. Physicians with the worst performance at the start of the pre-encounter reminders showed little absolute improvement over the next year, and most remained below the median performance for physicians in the practice.

Conclusions

Paper pre-encounter reminders did not appear to improve performance beyond electronic point-of-care reminders in the EHR alone. Lagging performance is likely not due to providers'' EHR workflow alone, and trying to step backwards and use paper reminders in addition to point-of-care reminders in the EHR may not be an effective strategy for engaging slow adopters.     

Area-level factors associated with electronic health record adoption and meaningful use in the Regional Extension Center Program

Objective

To identify area-level correlates of electronic health record (EHR) adoption and meaningful use (MU) among primary care providers (PCPs) enrolled in the Regional Extension Center (REC) Program.

Materials and methods

County-level data on 2013 EHR adoption and MU among REC-enrolled PCPs were obtained from the Office of the National Coordinator for Health Information Technology and linked with other county-level data sources including the Area Resource File, American Community Survey, and Federal Communications Commission''s broadband availability database. Hierarchical models with random intercepts for RECs were employed to assess associations between a broad set of area-level factors and county-level rates of EHR adoption and MU.

Results

Among the 2715 counties examined, the average county-level EHR adoption and MU rates for REC-enrolled PCPs were 87.5% and 54.2%, respectively. Community health center presence and Medicaid enrollment concentration were positively associated with EHR adoption, while metropolitan status and Medicare Advantage enrollment concentration were positively associated with MU. Health professional shortage area status and minority concentration were negatively associated with EHR adoption and MU.

Discussion

Increased financial incentives in areas with greater concentrations of Medicaid and Medicare enrollees may be encouraging EHR adoption and MU among REC-enrolled PCPs. Disparities in EHR adoption and MU in some low-resource and underserved areas remain a concern.

Conclusions

Federal efforts to spur EHR adoption and MU have demonstrated some early success; however, some geographic variations in EHR diffusion indicate that greater attention needs to be paid to ensuring equitable uptake and use of EHRs throughout the US.     

Implementation and Use of an Electronic Health Record within the Indian Health Service

Objectives

There are limited data regarding implementing electronic health records (EHR) in underserved settings. We evaluated the implementation of an EHR within the Indian Health Service (IHS), a federally funded health system for Native Americans.

Design

We surveyed 223 primary care clinicians practicing at 26 IHS health centers that implemented an EHR between 2003 and 2005.

Methods

The survey instrument assessed clinician attitudes regarding EHR implementation, current utilization of individual EHR functions, and attitudes regarding the use of information technology to improve quality of care in underserved settings. We fit a multivariable logistic regression model to identify correlates of increased utilization of the EHR.

Results

The overall response rate was 56%. Of responding clinicians, 66% felt that the EHR implementation process was positive. One-third (35%) believed that the EHR improved overall quality of care, with many (39%) feeling that it decreased the quality of the patient–doctor interaction. One-third of clinicians (34%) reported consistent use of electronic reminders, and self-report that EHRs improve quality was strongly associated with increased utilization of the EHR (odds ratio 3.03, 95% confidence interval 1.05–8.8). The majority (87%) of clinicians felt that information technology could potentially improve quality of care in rural and underserved settings through the use of tools such as online information sources, telemedicine programs, and electronic health records.

Conclusions

Clinicians support the use of information technology to improve quality in underserved settings, but many felt that it was not currently fulfilling its potential in the IHS, potentially due to limited use of key functions within the EHR.     

The impact of an electronic health record on nurse sensitive patient outcomes: an interrupted time series analysis

Objectives

To evaluate the impact of electronic health record (EHR) implementation on nursing care processes and outcomes.

Design

Interrupted time series analysis, 2003–2009.

Setting

A large US not-for-profit integrated health care organization.

Participants

29 hospitals in Northern and Southern California.

Intervention

An integrated EHR including computerized physician order entry, nursing documentation, risk assessment tools, and documentation tools.

Main outcome measures

Percentage of patients with completed risk assessments for hospital acquired pressure ulcers (HAPUs) and falls (process measures) and rates of HAPU and falls (outcome measures).

Results

EHR implementation was significantly associated with an increase in documentation rates for HAPU risk (coefficient 2.21, 95% CI 0.67 to 3.75); the increase for fall risk was not statistically significant (0.36; −3.58 to 4.30). EHR implementation was associated with a 13% decrease in HAPU rates (coefficient −0.76, 95% CI −1.37 to −0.16) but no decrease in fall rates (−0.091; −0.29 to 0.11). Irrespective of EHR implementation, HAPU rates decreased significantly over time (−0.16; −0.20 to −0.13), while fall rates did not (0.0052; −0.01 to 0.02). Hospital region was a significant predictor of variation for both HAPU (0.72; 0.30 to 1.14) and fall rates (0.57; 0.41 to 0.72).

Conclusions

The introduction of an integrated EHR was associated with a reduction in the number of HAPUs but not in patient fall rates. Other factors, such as changes over time and hospital region, were also associated with variation in outcomes. The findings suggest that EHR impact on nursing care processes and outcomes is dependent on a number of factors that should be further explored.     

How is the electronic health record being used? Use of EHR data to assess physician-level variability in technology use

Background

Studies of the effects of electronic health records (EHRs) have had mixed findings, which may be attributable to unmeasured confounders such as individual variability in use of EHR features.

Objective

To capture physician-level variations in use of EHR features, associations with other predictors, and usage intensity over time.

Methods

Retrospective cohort study of primary care providers eligible for meaningful use at a network of federally qualified health centers, using commercial EHR data from January 2010 through June 2013, a period during which the organization was preparing for and in the early stages of meaningful use.

Results

Data were analyzed for 112 physicians and nurse practitioners, consisting of 430 803 encounters with 99 649 patients. EHR usage metrics were developed to capture how providers accessed and added to patient data (eg, problem list updates), used clinical decision support (eg, responses to alerts), communicated (eg, printing after-visit summaries), and used panel management options (eg, viewed panel reports). Provider-level variability was high: for example, the annual average proportion of encounters with problem lists updated ranged from 5% to 60% per provider. Some metrics were associated with provider, patient, or encounter characteristics. For example, problem list updates were more likely for new patients than established ones, and alert acceptance was negatively correlated with alert frequency.

Conclusions

Providers using the same EHR developed personalized patterns of use of EHR features. We conclude that physician-level usage of EHR features may be a valuable additional predictor in research on the effects of EHRs on healthcare quality and costs.     

Impact of a web-based personally controlled health management system on influenza vaccination and health services utilization rates: a randomized controlled trial

Objective

To assess the impact of a web-based personally controlled health management system (PCHMS) on the uptake of seasonal influenza vaccine and primary care service utilization among university students and staff.

Materials and methods

A PCHMS called Healthy.me was developed and evaluated in a 2010 CONSORT-compliant two-group (6-month waitlist vs PCHMS) parallel randomized controlled trial (RCT) (allocation ratio 1:1). The PCHMS integrated an untethered personal health record with consumer care pathways, social forums, and messaging links with a health service provider.

Results

742 university students and staff met inclusion criteria and were randomized to a 6-month waitlist (n=372) or the PCHMS (n=370). Amongst the 470 participants eligible for primary analysis, PCHMS users were 6.7% (95% CI: 1.46 to 12.30) more likely than the waitlist to receive an influenza vaccine (waitlist: 4.9% (12/246, 95% CI 2.8 to 8.3) vs PCHMS: 11.6% (26/224, 95% CI 8.0 to 16.5); χ²=7.1, p=0.008). PCHMS participants were also 11.6% (95% CI 3.6 to 19.5) more likely to visit the health service provider (waitlist: 17.9% (44/246, 95% CI 13.6 to 23.2) vs PCHMS: 29.5% (66/224, 95% CI: 23.9 to 35.7); χ²=8.8, p=0.003). A dose–response effect was detected, where greater use of the PCHMS was associated with higher rates of vaccination (p=0.001) and health service provider visits (p=0.003).

Discussion

PCHMS can significantly increase consumer participation in preventive health activities, such as influenza vaccination.

Conclusions

Integrating a PCHMS into routine health service delivery systems appears to be an effective mechanism for enhancing consumer engagement in preventive health measures.

Trial registration

Australian New Zealand Clinical Trials Registry ACTRN12610000386033. http://www.anzctr.org.au/trial_view.aspx?id=335463.     

Electronic health records in small physician practices: availability,use, and perceived benefits

Objective

To examine variation in the adoption of electronic health record (EHR) functionalities and their use patterns, barriers to adoption, and perceived benefits by physician practice size.

Design

Mailed survey of a nationally representative random sample of practicing physicians identified from the Physician Masterfile of the American Medical Association.

Measurements

We measured, stratified by practice size: (1) availability of EHR functionalities, (2) functionality use, (3) barriers to the adoption and use of EHR, and (4) impact of the EHR on the practice and quality of patient care.

Results

With a response rate of 62%, we found that <2% of physicians in solo or two-physician (small) practices reported a fully functional EHR and 5% reported a basic EHR compared with 13% of physicians from 11+ group (largest group) practices with a fully functional system and 26% with a basic system. Between groups, a 21–46% difference in specific functionalities available was reported. Among adopters there were moderate to large differences in the use of the EHR systems. Financial barriers were more likely to be reported by smaller practices, along with concerns about future obsolescence. These differences were sizable (13–16%) and statistically significant (p<0.001). All adopters reported similar benefits.

Limitations

Although we have adjusted for response bias, influences may still exist.

Conclusion

Our study found that physicians in small practices have lower levels of EHR adoption and that these providers were less likely to use these systems. Ensuring that unique barriers are addressed will be critical to the widespread meaningful use of EHR systems among small practices.     

Behavioral health providers' beliefs about health information exchange: a statewide survey

Objective

To assess behavioral health providers'' beliefs about the benefits and barriers of health information exchange (HIE).

Methods

Survey of a total of 2010 behavioral health providers in a Midwestern state (33% response rate), with questions based on previously reported open-ended beliefs elicitation interviews.

Results

Factor analysis resulted in four groupings: beliefs that HIE would improve care and communication, add cost and time burdens, present access and vulnerability concerns, and impact workflow and control (positively and negatively). A regression model including all four factors parsimoniously predicted attitudes toward HIE. Providers clustered into two groups based on their beliefs: a majority (67%) were positive about the impact of HIE, and the remainder (33%) were negative. There were some professional/demographic differences between the two clusters of providers.

Discussion

Most behavioral health providers are supportive of HIE; however, their adoption and use of it may continue to lag behind that of medical providers due to perceived cost and time burdens and concerns about access to and vulnerability of information.     

Sociotechnical challenges to developing technologies for patient access to health information exchange data

Background

Providing patients with access to their medical data is widely expected to help educate and empower them to manage their own health. Health information exchange (HIE) infrastructures could potentially help patients access records across multiple healthcare providers. We studied three HIE organizations as they developed portals to give consumers access to HIE data previously exchanged only among healthcare organizations.

Objective

To follow the development of new consumer portal technologies, and to identify barriers and facilitators to patient access to HIE data.

Methods

Semistructured interviews of 15 key informants over a 2-year period spanning the development and early implementation of three new projects, coded according to a sociotechnical framework.

Results

As the organizations tried to develop functionality that fully served the needs of both providers and patients, plans were altered by technical barriers (primarily related to data standardization) and cultural and legal issues surrounding data access. Organizational changes also played an important role in altering project plans. In all three cases, patient access to data was significantly scaled back from initial plans.

Conclusions

This prospective study revealed how sociotechnical factors previously identified as important in health information technology success and failure helped to shape the evolution of three novel consumer informatics projects. Barriers to providing patients with seamless access to their HIE data were multifactorial. Remedies will have to address technical, organizational, cultural, and other factors.