Assistant Professor, Department of Medical Informatics & Clinical Epidemiology
This month, a paper I co-wrote, Metrics for assessing physician activity using electronic health record log data, was published in the Journal of the American Medical Informatics Association (JAMIA). The paper was written by a workgroup of the National Research Network for Audit Log Data, a group of researchers who use audit logs as a data source in their work. For those of you interested in how this paper came about and how I came to lead the workgroup and be last author on this paper, this blog post provides the background and impetus for this work.
As electronic health record (EHR) adoption and use has become widespread over the past decade, so have complaints about its inefficiency and lack of usability. Concerned about how this may be contributing to burnout, researchers, informaticians, and healthcare administrators have attempted to measure providers’ EHR use. The adage, “If you cannot measure it, you cannot improve it,” is particularly true here; as informaticians we must be aware of how much time providers are using the EHR before and after attempting any improvements.
How do we measure EHR use? Ideally, we would like an objective measure of the time that users spend completing their work in the EHR. Traditional observational studies are time and resource intensive, and are limited to measuring EHR use at work (and often only during business hours). Self-reporting is an option, but humans are known to be inaccurate at estimating time, as well as potentially biased at reporting their EHR use. In addition, self-reporting adds administrative burden to already busy provider schedules.
An increasingly popular method for measuring EHR use is audit-log analysis. Since all EHR vendors must record data about EHR accesses in audit logs, this data provides a rough sketch of what users do and when in the EHR without the need for observation or self-reporting. While there are specific minimum data requirements for audit logs, EHR vendors implement them differently and commonly provide multiple granularity levels of access recordings—from low level mouse clicks and key stroke data to higher level accesses of screens or parts of the EHR system. These audit logs can be used to estimate all sorts of metrics related to clinical care: how long users are spending in the EHR system, which tasks they are performing and in what order, how long corresponding clinical workflows took, and how much teamwork was involved in patients’ care.
Researchers, including myself, have been using audit logs in their research. My research group has used them to build simulation models of outpatient clinics and evaluate the effectiveness of new scheduling templates based on these models, for determining trends of EHR use over a decade, for studying chart review patterns, and for determining the impact of scribes on provider workflows (paper in progress). In addition, EHR vendors provide metrics based on these audit log entries; for example, OHSU EHR vendor Epic has productivity metrics such as Signal reports (previously PEP reports). Because the use of audit logs has significantly grown over the past several years, Julia Adler-Milstein, PhD started the National Research Network for EHR Audit Log Data in 2018 to provide a forum for researchers, vendors, and analysts to share projects and methodologies during monthly webinars. In 2019, we formed workgroups to focus on particular aspects of this work in an attempt to shape and further the field. The three workgroup areas are task identification, measure development, and standards development. I currently lead the measures workgroup, which has had a very active year.
One of the first things we did was to survey the existing literature to determine how EHR audit logs have been used in research. Adam Rule, PhD, who is a postdoctoral fellow in our department, led this effort and published the resulting systematic review in JAMIA this past November. He identified the different ways EHR audit logs had been used in 85 articles as of September, 2019. Details about the methods and their validation were not given in the majority of the articles, however, making comparison or meta-analysis impossible. This paper recommends the development of standardized and validated measures as a collaborative effort by vendors, healthcare institutions, and the audit log research community to improve the transparency, reproducibility, and comparability of studies using audit logs.
Not surprisingly, the next focus for the workgroup was the definition of standardized measures. Given the current interest in measuring physician EHR use and burnout, the workgroup developed a recommended set of seven standard measures of outpatient physician work using audit logs. The definition of this measure set was led by workgroup member and well-known thought leader on physician burnout and the role of the EHR, Christine Sinsky, MD of the American Medical Association (AMA). Guided by work she had been doing with the AMA on measuring burnout, the workgroup discussed which measures to include along with their potential use cases and limitations. We hope the resulting paper will provide guidance for vendors, administrators, and researchers as they move forward with implementing audit log measures for physician work.
Interest in audit logs as a source of information about clinic work and timings is growing. Over 100 people participate in the NRN group, and there are over 35 people in the audit log measures workgroup. The NRN group had a panel presentation at the American Medical Informatics Association (AMIA) Annual Symposium in November, 2019. I also presented an overview of our work at the Office of the National Coordinator for Health IT (ONC) Annual Meeting in January, 2020. We still have much work to the accomplish; the other workgroups are continuing development of task definitions and standards, and the measures workgroup is pushing ahead with defining more standardized measures for inpatient settings and more clinician roles, implementing and validating the standardized measures, and documenting and mapping existing EHR vendor metrics to the standardized measures. We invite you to be part of this exciting new data and research area by participating in the NRN monthly webinars and/or one of the workgroups. Contact firstname.lastname@example.org for more information!