The fundamental premise of a “learning health system” is that researchers can collaborate effectively with delivery system leaders to answer pressing clinical and operational questions. As we work with leaders in Kaiser Permanente Colorado (KPCO), researchers from the Institute for Health Research (IHR) are focusing on two areas: application of advanced analytical tools to make better use of our rich clinical and administrative data, and rapid evaluation of service delivery interventions. These collaborations require us all to address six critical issues. We must:

▪ Ask the right question
▪ Convene the right team to answer the question
▪ Use the right project design
▪ Assemble the right data
▪ Apply the right analytical tools
▪ Provide the right interpretation of the findings

In this issue I’ll focus on the “third right”, choosing the right design to evaluate operational interventions.

Leaders are interventionists by nature. They make decisions about staffing, clinic hours, referral protocols, professional education, decision support tools, communication with members, and a host of other issues. Researchers recognize that the standard of evidence to inform operational decisions can be different from the standards of conventional research. Leaders need to make rapid decisions under conditions of substantial uncertainty, while researchers slowly and methodically conduct studies that leave little to chance. Leaders typically need to learn many things quickly, while researchers typically learn one big thing slowly. Leaders prize relevance, researchers prize rigor.

Experiments in fields outside of health care are identifying a meeting place between these perspectives. The Harvard Business Review publishes articles about the design and analysis of experiments to inform business decisions. A 2010 article in The New Yorker described field experiments of strategies for economic development in poor countries that were designed to assess “different ways to save the world”. The Federal Office of Management and Budget and the White House Office of Science and Technology have called for rigorous studies to evaluate the effectiveness of social programs.

The importance of randomization is a common theme across these fields. A randomized study may be termed an “A/B experiment” by a businessman, a “micro-experiment” by an economist, or a rigorous “rapid test of change” by the quality improvement community. All recognize that a well-designed and executed randomized trial is the best way to reduce the threat of bias and the play of chance. A trial based on a well-formed question (the first “right” in the IHR litany), should be informative to a decision-maker whatever its outcome. If it achieves its intended outcome, a successful intervention can be disseminated. If not, the intervention can be stopped, or modified and re-tested. If the same experiment is conducted in different settings (be they classrooms, villages, or on-line communities) the findings of a single trial can be confirmed and extended. Most importantly, such studies can be conducted within practical constraints of time, resources, and data availability.

Evidence from randomized trials has long been the “gold standard” to support clinical decisions. Randomization has entered the arena of health care delivery more slowly, however. Organizational leaders often lack the time, training, or data to design field trials. They may be concerned about the consequences of delaying an important decision or the cost of conducting a randomized evaluation. They may be uncertain about the need for human subjects review by an Institutional Review Board. They may worry that their patients or members will not want to be “guinea pigs”. These concerns are shared by ethicists and regulators, who struggle with the distinction between operations, quality improvement, and research. The public is also ambivalent. Some individuals resist becoming the subjects of experiments, while others invite experimentation.

A growing body of work suggests that these issues can be resolved. A recent publication from the Agency for Healthcare Research and Quality provides important guidance for the conduct of rapid-cycle research in health care settings. Many investigators in the IHR have conducted randomized trials within KPCO, on topics ranging from medication adherence, to vaccine hesitancy, blood pressure monitoring, immunization delivery, and appointment-keeping. In all these studies, operational engagement and prospective planning have been critical to success. My old adage is that “an ounce of planning is worth a pound of data analysis”. As we continue to build a learning health system within KPCO, the IHR is confident that we can conduct randomized trials rapidly enough and inexpensively enough to provide information that leaders need, all while maintaining the scientific rigor that the research community rightly demands.

Warmest Regards,
John F. Steiner, MD, MPH
Senior Director

Kaiser Permanente (KP) continues to enhance its investment in Genetic Research with the inter-regional Kaiser Permanente Research Bank, a long-term research program funded by Kaiser Foundation Health Plan Community benefit, designed to help better understand how people’s health is affected by their genes, behaviors, and the environment. Currently the KP Research Bank holds 220,000 individual DNA samples, with a goal to add an additional 280,000 from across all seven KP regions.

Starting this month, the Colorado region will begin recruitment efforts. Invitations are being sent to a sample of racially and ethnically diverse KP members across the state, but all adult KP members 18 years and older are invited and encouraged to participate. In order to join, members are asked to complete a brief health survey and contribute a small DNA blood sample. Plans are to assemble a general group of participants as well as specialty groups of people who are pregnant or have cancer.

Development of the cancer group is being led by IHR Investigator Heather Spencer Feigelson, PhD, MPH. The goal of this group is to recruit 30,000 newly diagnosed cancer patients to complete a cancer-specific health survey as well as give a blood sample. KP is uniquely positioned to identify new cancer cases soon after diagnosis to obtain a pre-treatment sample as well as to collect samples from patients with advanced cancers that other studies often miss. This specialty group will enhance the availability of pathology specimens as well as medical record data on co-morbidities, medication use, and other patient characteristics that may influence cancer treatment and survival. The information gathered will become a critical resource in the future of cancer research by offering a more complete picture of treatment, adherence, recurrence, morbidity, and mortality.

The success of this work is greatly due to the efforts of the local Colorado team which includes:

• Andrea Burnett-Hartman, PhD, MPH, Site Co-Investigator, Colorado Pregnancy Cohort Lead, and Survey Lead;
• Christina Clarke, MPH, Lead Biostatistician for the Cancer Cohort Rapid Case Ascertainment and Lead Site Biostatistician;
• Bill Harding, Site Programmer;
• Michelle Wrenn, MA, Colorado Operations Lead and Cancer Cohort Project Manager;
• Erica Blum-Barnett, MSPH, Sr. Research Assistant, Coordinating the Pregnancy Cohort and Liaison with the Lab Team; 
• Kate Burniece, Sr. Research Specialist, Liaison with the Lab Team.

In 2014 the IHR implemented the ARISE (Accountability, Research, Innovation, Service, Excellence) Award to recognize staff for outstanding service and contributions to the department. Peer nominations are reviewed by a committee comprised of IHR staff to determine six finalists. Congratulations to all of the nominees and to finalists Nikki Carroll, Jennifer Ellis, Mai Khang, Dr. Matt Daley, and Allen Malone--and to our 2015 winner Ann Blakeslee!

IHR Staff Researcher Cristy Geno Rasmussen, PhD, MPH, has accepted a joint position as a Behavioral Scientist at the Gretchen Swanson Center for Nutrition and an Adjunct Assistant Professor at the University of Nebraska Omaha (UNO) Medical Center College of Public Health. Dr. Geno has been with the IHR since 2002 and worked on a number of behavioral and health services research projects. We wish you all the best in your new endeavors!

Senior Project Manager Jo Ann Shoup, MSW has successfully defended her PhD dissertation titled, “Concerns, Attitudes, Beliefs, and Intentions of Parents about Vaccines for their Children: Development and Evaluation of a Survey Instrument in an Integrated Health Care System in Colorado”. Congratulations, Dr. Shoup!

The IHR was recently awarded a new contract from the Colorado Department of Public Health and Environment (CDPHE) to evaluate CDPHE’s Cancer, Cardiovascular, and Pulmonary Disease Initiative and Health Disparities Grant Program. The Initiative includes 45 grantees across Colorado implementing 17 different strategies (e.g. physical activity, healthy eating, cancer screening, HPV vaccinations, patient navigators, disease management, medication adherence, and diabetes and hypertension management). Grantees have received funding through June 30, 2018. The evaluation includes measuring reach, effectiveness and implementation across all grantees and assessing the overall impact of the grant portfolio on health behaviors, health outcomes, systems and policies. Led by Dr. Cheryl Kelly and the Kaiser Permanente Evaluation Team, the team also includes content and clinical advisors from the IHR (Drs. John Steiner and Matt Daley) and the University of Colorado Denver. This is a large community-facing evaluation that will help the IHR build relationships with many local public health agencies, other health providers, and clinics and local non-profit agencies.

At the annual Center for Effectiveness & Safety Research (CESR) meeting, teams from across the Kaiser regions competed in a "Shark Tank" event where each of the teams proposed an intervention that was innovative and in need of additional evaluation to assess its effectiveness. IHR researchers participated on 3 different teams addressing Behavioral Health/Primary Care, Diabetes Management, and Food Insecurity. Each team had 3 minutes to propose the innovation and 3 minutes to present the evaluation, with one round of questions from the moderators. Conference attendees then voted for the innovation-evaluation they would like to see happen. The prize is a commitment from CESR to help move the work forward. The Food Insecurity team comprised of Dr. John Steiner, Dr. Cheryl Kelly, and Dr. Marissa Allen from the IHR and Dr. Sandra Stenmark of Kaiser Colorado's Community Health Initiatives Preventive Medicine department, was the winner! The team is now in the process of developing a more fully developed proposal. Their intervention will encourage providers and clinics to screen for food insecurity and assess if screening leads to increased enrollment in food assistance programs, increased access to food and, ultimately, improved health outcomes. There will be more to share on this work in the coming months.

Check out IHR Publications here.