PRP139: Precision implementation: Developing and validating predictive models of information technology tool adoption
Nathalie Huguet, PhD; Miguel Marino, PhD; Katie Fankhauser, MPH; Jean O'Malley, MPH, PStat; Jennifer DeVoe, MD, DPhil; Heather Angier, PhD, MPH; Erin Jamieson
Abstract
Context: There is strong evidence that different implementation support strategies (e.g., facilitation, audit and feedback, performance benchmarking) can help clinical practices with adoption and maintenance of evidence-based guidelines. There are, however, costs to both providing and receiving implementation support. While evidence demonstrates the effectiveness of various implementation strategies, relatively little is known about which practices will benefit most from a particular implementation strategy, how much assistance a practice might need, or if practices could improve on their own. New methods are needed to predict which practices may implement targeted changes with less support and which will need more.
Objective: To develop and validate predictive models that estimate the likelihood of adoption of electronic health record (EHR)-related tool.
Study Design: Retrospective, observational study.
Setting or Dataset: EHR data from 351 CHCs from the OCHIN Network.
Population Studied: 351 practices eligible to receive insurance support information technology (IT) tool with an active EHR from 05/01/2017 to 06/30/2019.
Outcome Measures: Binary indicator of any tool uptake, rates of monthly tool use, and number of patients the tool was used on.
Results: We will use an ensemble of machine learning algorithms (i.e., Super Learner) to develop and validate a set of predictive models based on practice-level characteristics to predict our adoption outcome measures. We will split data into development and test data sets (80%/20% split) and use area under the receiver operating characteristics curve (AUC) and mean squared error to evaluate predictive performance. We will report a cross-validated estimate of performance of each algorithm in the ensemble as well as performance of the ensemble on external testing data. We hypothesize that EHR-based data will have moderate prediction performance (as evidence by an AUC>0.60) on tool adoption for CHC practices.
Conclusion: This work is the next step toward advancing the science of ‘precision implementation’ and how to efficiently tailor and deploy implementation support strategies for IT innovations.
Objective: To develop and validate predictive models that estimate the likelihood of adoption of electronic health record (EHR)-related tool.
Study Design: Retrospective, observational study.
Setting or Dataset: EHR data from 351 CHCs from the OCHIN Network.
Population Studied: 351 practices eligible to receive insurance support information technology (IT) tool with an active EHR from 05/01/2017 to 06/30/2019.
Outcome Measures: Binary indicator of any tool uptake, rates of monthly tool use, and number of patients the tool was used on.
Results: We will use an ensemble of machine learning algorithms (i.e., Super Learner) to develop and validate a set of predictive models based on practice-level characteristics to predict our adoption outcome measures. We will split data into development and test data sets (80%/20% split) and use area under the receiver operating characteristics curve (AUC) and mean squared error to evaluate predictive performance. We will report a cross-validated estimate of performance of each algorithm in the ensemble as well as performance of the ensemble on external testing data. We hypothesize that EHR-based data will have moderate prediction performance (as evidence by an AUC>0.60) on tool adoption for CHC practices.
Conclusion: This work is the next step toward advancing the science of ‘precision implementation’ and how to efficiently tailor and deploy implementation support strategies for IT innovations.