Forward icon

Message from the Director

Asking Good Questions

In my last column, I began to discuss the guiding principles for collaborations between IHR researchers and clinical and operational leaders in Kaiser Permanente Colorado and the Colorado Permanente Medical Group. These principles are:

▪ Ask the right question
▪ Convene the right team to answer the question
▪ Use the right project design
▪ Assemble the right data
▪ Apply the right analytical tools
▪ Provide the right interpretation of the findings

How do we ask the right question? None of us in the IHR ever took a formal course on the topic. Yet we know that asking good questions is a prerequisite for good research. We all know researchers who are particularly good at asking questions. And we can often recognize a good question, even outside our own field.

Is asking good questions a gift or a skill?  In the hope that someone could answer this question about questions, I spent some time searching the literature. I first found a string of published clinical commentaries with titles such as: “Postpartum urinary incontinence: asking the right questions”, “Mandibular fractures – are we asking the right questions?”, “What can we learn about postnatal care in Ghana if we ask the right questions?”

Clearly, some commentators think that researchers in their own field need to ask better questions. Although disparaging conventional research questions may just be a rhetorical device for authors who want to claim superior insight, I think these papers point to a real problem. We researchers can become so enamored of our research questions that we no longer ask whether those questions are really the most important ones. This pitfall has important implications for operational collaborations. If experienced researchers still struggle to ask good questions, we need to approach our operational and clinical collaborations with a healthy dose of humility.

A few papers did provide insight into the process of asking questions. The PICO (or PICOT) criteria can be used to both formulate literature searches and report research findings. These criteria propose that a “well-formed” question specifies the Patient or Problem, the Intervention, the Comparison intervention, and the clinical Outcome. Some commentators added the Timing or Temporal association between the intervention and the outcome to that list. A text on clinical research design proposed a set of criteria for a good research question using the acronym FINER: Feasible, Interesting, Novel, Ethical, and Relevant. Others in the business world proposed that by asking a series of five questions that begin with “why” we can move from the statement of a problem to the underlying, fundamental question.

These approaches are quite different. PICOT defines the variables needed to specify a clinical question. FINER identifies judgments that need to be made in assessing a question. The “5 whys” proposes a method of inquiry. All are important components of asking good questions, but their very diversity emphasizes the complexity of the process.

My own thoughts on asking questions are anecdotal:

  1. Identifying good questions is a team sport that is enhanced by multiple perspectives. Collaborations among researchers, patients, clinicians and operational leaders are likely to improve the quality and relevance of the question.
  2. Most questions have been asked before. Searching for existing answers is worth the effort. That said, we are more likely to believe local answers to our questions even when the work of others could guide us.
  3. We should propose our questions before we dive into the data. While new questions can emerge as we analyze data, more often we wind up chasing phantoms if we don’t have a plan in advance.
  4. Researchers have developed strategies for reducing bias in answering questions, but we are all susceptible to asking biased questions. We need to commit to taking action on the answer, whatever it may be. Any answer to a good question will be informative, and the answer we don’t expect may be more informative than the answer we want.
  5. Most big questions can only be answered through a succession of small questions, yet each small question may be difficult to answer in its own right. The patience and persistence necessary to build understanding can be one of the most challenging aspects of collaboration with clinical and operational colleagues. But we do need to avoid the trap of becoming so immersed in small questions that we lose sight of the big ones.
  6. We need to be wary of asking only questions for which we already have data. Many good questions require the time and expense necessary to collect the right data, another guiding principle in the six “rights”.
  7. Even good questions generally have provisional answers. A partial answer is still better than no answer at all, particularly for our colleagues who need to make decisions and take action. We researchers can be too willing to conclude simply that more research is necessary.

I’m convinced that asking good questions is much more an acquired skill than an innate gift. We gain the skill of asking good questions by asking progressively better ones. In that process (as Lennon and McCartney noted), we get by with a little help from our friends. A learning health system brings together researchers who love asking questions with colleagues who are accountable for the answers. We have much to teach each other.

Warmest Regards,
John F. Steiner, MD, MPH
Senior Director


2015 Financial Results

2015 continued to be a challenging year for research funding nationwide. For the IHR, 2015 funding levels remained consistent with the prior year. Our researchers maintained a diverse portfolio of extramural grants and we added two new funders to our portfolio with awards from the Department of Defense and the Bureau of Justice. Additionally, the implementation of the Integrated Regional Analytics (IRA) initiative as part of the overall Data, Analytics, and Reporting Excellence (DARE) program strengthened our collaboration with Kaiser Permanente Colorado (KPCO) leaders. The IRA formalizes our partnership with the region in answering questions of organizational importance and evaluating key operational initiatives. For example, IRA team members are collaborating with regional leaders to answer questions such as:

  • How can we predict needs for Medical Financial Assistance funds?
  • What are predictions for membership utilization in 2016?
  • How can we optimize care to Medicaid members given constraints?
  • Can a predictive rule for missed opportunities be applied to the scheduling process?

By responding to questions such as these, the IHR helps KPCO fulfill its mission to provide high-quality, affordable health care services to improve the health of our members and the communities we serve.

IHR Researchers Leading Vaccine Safety Study Agenda

Vaccine Safety Datalink (VSD) researchers and IHR Investigators Matthew Daley, MD, and Jason Glanz, PhD, recently published a White Paper commissioned by the Centers for Disease Control and Prevention (CDC) on the feasibility of studying the safety of the childhood immunization schedule within the VSD. Despite the large majority of parents in the U.S. that support and follow the recommended vaccine schedule, a growing number of parents are delaying or refusing vaccines for their children. In light of this, the Institute of Medicine decided it would be of value to study differences in health outcomes between fully vaccinated children and those on delayed or alternative vaccine schedules. Collaborating with a number of IHR staff members, including Sophia Newcomer, MPH, and Jo Ann Shoup, PhD, the paper outlines a valuable research plan aimed at maintaining overall confidence in a vital public health intervention.

IHR Investigator and Father, Dr. Ted Palen Skis for Prader-Willi Research

On Thursday, March 10, 2016, IHR investigator Ted Palen, PhD, MD, MSPH, and two friends skied non-stop for 9 hours to raise funds for the Foundation for Prader-Willi Research. Taking part in a Vertical Feet Ski Marathon, Ted made 61 runs and descended 100,990 vertical feet. As the parent of a child with Prader-Willi Syndrome and an avid skier, Ted combined two great loves to contribute over $2900 towards on-going research. You can make a post-event donation here. Ted thanks all of the staff for their support and encouragement!

IHR Evaluation Team Tackles Food Insecurity

IHR Evaluation Project Manager Marisa Allen, PhD, is leading an evaluation of 13 new Kaiser Colorado Community Benefit grants aimed at decreasing food insecurity. These grantee partners include community-based organizations and safety net clinics around the state. They will be working over the next two years to increase enrollment in the Supplemental Nutrition Assistance Program (SNAP) and Summer Food Service Programs (SFSP). Evaluation Investigator Cheryl Kelly, PhD, MPH, Research Specialist Lisa Harner, MA, and Senior Professional Research Assistant Mariana Villarreal, are also working on this evaluation. The project will be tracking information about barriers and facilitators to SNAP and SFSP enrollment as well as changes in grantees' organizational capacity. This work adds to the evaluation team's growing portfolio and strengthens our content expertise and relationships with new community partners.


Check out the most recent IHR Publications here.