IVR in the IHR: a Tale of Two Acronyms
There is a lot of discussion these days about how to establish a virtuous cycle of research and implementation in health care (1). Even advocates of the "learning healthcare system" model do not always appreciate that the tools developed to conduct research studies can be as valuable for clinical and operational purposes as the findings of those studies. The history of interactive voice response (IVR) technology in the Institute for Health Research (IHR) and in Kaiser Permanente (KP) Colorado illustrates this point.

IVR applications allow computers to interact with people through voice and telephone keypad data entry. All IVR activities for KP Colorado are housed in the IHR, where our team of project managers and programmers develops new applications in collaboration with operational and clinical leaders and IHR investigators. We send IVR messages to our members to remind them of appointments, preventive services, medication refills, or to collect health information. We can provide messages in English, Spanish, or a culturally congruent voice (2). In turn, our members can identify care needs that we automatically refer to advice lines, appointment schedulers, or pharmacists. On average, we send over 45,000 calls and a comparable number of text messages each month.

IHR investigators began to use IVR in 2005 to collect data about physical activity and healthy eating for large-scale prevention studies (3;4). More recently, our researchers have used IVR to improve immunization rates, blood pressure control and medication adherence (5), to educate members about specific health issues, and to collect data about symptoms or treatment side effects. Overall, the IHR has used IVR for at least 20 operational projects and 14 research projects in the last nine years.

Shortly after I joined KP Colorado in 2008, I received an IVR call asking for information about my prior screening for colorectal cancer and offering education about screening options as part of the first joint IVR project between IHR and our Population Care and Preventive Services department. The developers of the program used a "stepped-wedge" design to demonstrate a four-fold increase in screening among individuals receiving the IVR intervention (6). The program won a KP Colorado CARE Award in 2009. During the "flu season" in 2012, a randomized trial demonstrated that IVR contacts were as effective as mailed reminders for influenza vaccination, and less expensive when applied across the large population of KP. As a result, KP Colorado has relied exclusively on IVR for "flu shot" reminders in 2013 and 2014. We are currently analyzing data from a randomized, internal evaluation of a single IVR reminder call for primary care appointments at our Westminster clinic. This evaluation is showing that, even though the rate of missed appointments was already low, reminders further reduced the rate, leading to an extra 30 "kept" appointments per week at this site.

Over this time, we've learned several lessons that are relevant to the national conversation about learning healthcare systems:

• Proximity is important. Strong communication between researchers and operational leaders led to rapid conversion of IVR from a purely research tool to an operational asset. It helps that IHR and our Population Care and Preventive Services Department are on adjacent floors of the same building. Operational adoption of IVR likely would have been much more difficult in an environment in which researchers were not "embedded" in their delivery system.
• Rigorous evaluation improves decisions. Operational leaders, clinicians, and researchers are all persuaded by the findings of carefully designed evaluations. We are learning that randomization is feasible for some of these evaluations. In fact, our 2012 study of influenza reminders was initiated by operational leaders who demanded a randomized trial! The IHR was more than willing to oblige.
• Research can be conducted within operational time frames. The influenza reminder study was designed in the summer of 2012 in anticipation of "flu" season, conducted that autumn, and analyzed over the next few months so that IVR could be universally deployed by the fall of 2013. For our recent intervention to reduce missed appointments, only 22 weeks elapsed from the first discussion of the project to provision of data to operational sponsors – and about half that time was spent actually conducting the intervention.
• Even small effects can be operationally meaningful. The effects of IVR on the outcomes of interest are often modest by research standards. But small effects multiplied by large populations can produce meaningful improvements in care or efficiency for operational leaders and managers.
• If you're ready to learn, everyone is a teacher. Both operational and research requests have helped our IVR programmers develop new capabilities of the IVR system. They can now incorporate data from multiple sources within KP Colorado and export information back to the electronic health record. They can recycle content from one project to the next. In other words, they have learned how to accelerate the virtuous cycle of the learning health care system.

The nine-year experience with IVR in the IHR and KP Colorado has shown us all that continuous learning in health care is a reality, not just an ideal. No master plan has guided the evolving uses of IVR. Instead, an informal group of researchers, technical experts, operational leaders, and clinicians has tried a succession of interventions to meet their individual needs. The process has been messy, but I think this is how we really learn, in life and in health care. We build things. We try them. We fix them if they don't work right. We evaluate. We fail. We adapt. We try again. We get better. And sometimes we even publish!

I would like to end by acknowledging our superb IVR team: programmers Jonah Langer and David Steffen, project managers Mike Shainline and Lisa Pieper, and IVR team manager Mike Bodily.

Warm Regards,
John F. Steiner, MD, MPH
Senior Director

References
(1) Greene SM, Reid RJ, Larson EB. Implementing the learning health system: from concept to action. Ann Intern Med 2012 Aug 7;157(3):207-10.
(2) Albright K, Richardson T, Kempe KL, Wallace K. Toward a trustworthy voice: increasing the effectiveness of automated outreach calls to promote colorectal cancer screening among African Americans. Perm J 2014;18(2):33-7.
(3) Estabrooks PA, Smith-Ray RL. Piloting a behavioral intervention delivered through interactive voice response telephone messages to promote weight loss in a pre-diabetic population. Patient Educ Couns 2008 Jul;72(1):34-41.
(4) Estabrooks PA, Shoup JA, Gattshall M, Dandamudi P, Shetterly S, Xu S. Automated telephone counseling for parents of overweight children: a randomized controlled trial. Am J Prev Med 2009 Jan;36(1):35-42.
(5) Magid DJ, Ho PM, Olson KL, Brand DW, Welch LK, Snow KE, Lambert-Kerzner AC, Plomondon ME, Havranek EP. A multimodal blood pressure control intervention in 3 healthcare systems. Am J Manag Care 2011 Apr;17(4):e96-103.
(6) Kempe KL, Shetterly SM, France EK, Levin TR. Automated phone and mail population outreach to promote colorectal cancer screening. Am J Manag Care 2012 Jul;18(7):370-8.

---

“Evaluation can help create and inform new directions for our work whether it is toward research, program and intervention development, or community benefit work.”

---

Cristy Geno Rasmussen, PhD, MPH, is a new Staff Researcher at the IHR. Dr. Geno has been with the department for over 12 years, serving as a Senior Project Manager and Health Services Research Manager. Her new role as a Staff Researcher is an opportunity to partner with evaluators and researchers across Kaiser Permanente regions to increase evaluation collaborations and explore additional community health initiative (CHI) evaluation opportunities. Dr. Geno’s research interests include obesity and chronic disease prevention, health disparities, evaluation methods, and dissemination and implementation research, with a current focus on increasing evaluation capacity and expertise within CHI across Kaiser Permanente. As a Staff Researcher, Dr. Geno has been working with research and evaluation mentors to increase her understanding of evaluation methods and metrics. She is hoping to expand current evaluation efforts between Kaiser and its partners to help prioritize and create important initiatives and programs across Kaiser communities. Congratulations, Dr. Geno!

Cheryl Kelly Buening, PhD, MPH, MA, is a new Evaluation Investigator at the IHR. Dr. Kelly joins us from the University of Colorado, Colorado Springs. She has extensive experience in program and policy evaluation, as well as evaluating the relationship of the built environment and related policies with healthy behaviors, including access to grocery stores, sidewalks, parks, and safe neighborhoods. Dr. Kelly's time will be supported by KP Colorado’s Community Benefit (CB) department to develop a comprehensive evaluation plan of CB's community health needs assessment implementation strategy addressing access to quality care and healthy eating and active living. Dr. Kelly’s passion for this work contributes to closing the gap between research and practice in public health. Welcome, Dr. Kelly!

Elizabeth Bayliss, MD, MSPH, Liza Reifler, MPH, Chan Zeng, PhD, Deanna McQuillan, MA, Jennifer Ellis, MBA, MSPH, and John Steiner, MD, MPH, co-authored a paper in the Journal of Comorbidity evaluating the risks of cardiovascular and other comorbid events in patients with cancer. Among the group of patients with a predicted 5-year survival rate of 26-50%, the novel modeling approach found that cancer diagnosis carried the greatest risk of adverse outcomes, particularly in the first year of disease. However, it is important for cancer patients with multiple illnesses to be provided with information about the combined risk of all of their comorbidities.

Matthew Daley, MD, and David Magid, MD, MPH, co-authored a study published in the Journal of Pediatrics examining the effects of combined oral contraceptive (COC) use on blood pressure (BP) and body mass index (BMI) in adolescents. The study found no significant differences in BP or BMI in the COC group versus controls. These findings provide further resassuring evidence to both patients and their doctors that a change in BMI or BP following start of COCs is not necessarily correlated to the medication.

Several IHR staff attended the American Evaluation Association's 28th Annual Conference held in Denver, CO, October 15-18. Lisa Harner, MA, presented a poster titled, "Enhancing the population dose evaluation method with geographic information systems." Cristy Geno Rasmussen, PhD, MPH, Erica F. Morse, MA, Heather Nuanes, Carmen Luna, MPH, and Michelle Henton, MA, presented a poster titled, "Lessons learned from a multilevel evaluation structure: Kaiser Permanente's Community Health Initiative (CHI) cross-site evaluation."

Jennifer Barrow, MSPH, and David Tabano, MA, presented at the Colorado Public Health Association's conference, Public Health in the Rockies, in Fort Collins, CO, on September 18. Their presentation was titled, "Body Mass Index (BMI) Monitoring System: A Pilot for Mapping Obesity Data."

David Tabano, MA, presented at the American College of Epidemiology Annual Meeting on September 8 in Silver Springs, MD as well as at the New York City Department of Health and Mental Hygiene's conference, Setting the Agenda: Population Health Monitoring and EHRs on October 17. His presentation was titled, "Colorado BMI Surveillance System: A Pilot for a Public Health Surveillance Network." 

Kaiser Permanente Colorado (KPCO) has experienced a substantial increase in new members following the Affordable Care Act mandate for all Americans to obtain health insurance. Many new members purchased coverage through the state’s marketplace, Connect for Health Colorado, while others purchased directly from KPCO. Several grants have been awarded to the IHR aimed at understanding the characteristics of the new members and to assist with operational decisions and provision of quality care.

As part of this work, two surveys have been launched with new members. The first is a four-page mail and telephone survey for randomly-selected new individuals and the second is a nine-question onboarding survey done with all new members who call or come in to KPCO medical offices. We have already learned a great deal from these surveys, including:

• Most new members had previous insurance, particularly those purchasing directly from KPCO;
• The majority of new members self-report they are in excellent or very good health;
• The majority of new members were up-to-date on preventive screenings;
• Members purchasing through the marketplace tend to be older, members of ethnic minorities, and have lower income;
• ​Members purchasing through the marketplace often delayed care because of cost, and Medicaid patients reported their health was impacted because of other expenses;
• Patients receiving Medicaid had the highest rate of emergency department (ED) use and hospitalizations in the previous year; and
• ​​Depressive disorder was the most reported chronic condition followed by high blood pressure.

​In the coming months, project teams will focus on analyzing member utilization data to answer a variety of operational and policy questions. A follow-up survey with the same randomly-selected new members from the first survey will be conducted with hope of measuring satisfaction after one-year of enrollment, as well as reasons for re-enrolling or dis-enrolling. Stay tuned for more information and publication of our findings.

The IHR would like to welcome several new staff members to the team. Kathy Gleason, PhD, is a new Research Specialist. Dr. Gleason has an extensive background in immunology and will be working with Jason Glanz, PhD, on various vaccine-related studies. Letoynia Coombs, EdD, MS, joins us as a Data Specialist. Dr. Coombs is coming from the University of Colorado and has a diverse background, bringing not only statisticial skills, but evaluation and mathematics instruction as well. Allen Malone has been with KP for many years, but is now joining the IHR as a Data Specialist. Allen has extensive background in statistical application development and testing and will be focusing his efforts on virtual data warehouse (VDW) projects. Welcome!

After over 5 years in the IHR, Director of Research Administration James Beaudry, MBA, is leaving the department. Jim will be joining the Government, External Relations and Research (GERR) area of KPCO serving as Senior Director of Business Operations and Strategy. During his time here, Jim has become a respected leader and partner, guiding the IHR to the forefront of excellence in research administration as well as science. We wish you all the best in your new leadership role, Jim!

The 2014 IHR publications can be found here.