Strategies for Measuring Outcomes and ROI for Managed Care Programs

MANAGED CARE November 2011. MediMedia USA

Determining return on investment and improvement of outcomes is essential when new programs are being tried, but the research method must be chosen carefully

Andrew Kolbasovsky, PsyD, MBA

Director of provider group clinical management at EmblemHealth, New York, N.Y.

Introduction

As wellness, case, and disease management programs proliferate, health plans must measure their outcomes and potential for return on investment. Here are a few strategies to help with that evaluation.

Recent years have witnessed a proliferation of wellness, case, and disease management programs in managed care. Despite their prevalence, few are subjected to the rigorous research designs necessary to determine whether financial goals are achieved. As a result, the effectiveness of these programs in reducing costs remains to be determined (Linden 2006; Mattke 2007).

The lack of stringent evaluation of these programs is often attributed to difficulty in identifying a suitable control group (Linden 2006; Linden 2003). Program evaluation in managed care has therefore tended to rely on observational pre-post studies that are subject to selection bias and regression to the mean (Linden 2006).

Program evaluators often resort to using a pre-post design with a relatively small percentage of enrolled members, comparing utilization for a specified pre-enrollment period with a similar post-enrollment period, or they compare the results of members enrolled in the program with eligible members who were unable to be enrolled. Each of these designs has critical weaknesses.

The pre-post design is highly subject to regression toward the mean. Members are identified for intervention as a function of being a utilization outlier. Regression toward the mean suggests that outliers at one point in time are likely to be closer to the mean at re-measurement, even in the absence of intervention. As a result, reductions in cost and utilization cannot be fully attributed to the intervention.

Comparing enrolled members to unenrolled members is subject to selection bias. Members who cannot be enrolled in phone- or mail-based interventions may be fundamentally different from enrolled members. Unenrolled members are unreachable or refuse to accept services. Being unreachable may suggest greater instability in housing or telephone service, or greater financial and psychosocial stressors. Or refusal to accept a free intervention may reflect a low level of motivation to improve health. If enrolled members have fewer financial and psychosocial issues, or are inherently more motivated to improve their health, these differences might explain group outcomes, rather than the intervention.

Given the importance of managing health care costs, it is critical that administrators appropriately evaluate outcomes of managed care programs aimed at reducing costs.

Measurement strategies are available that avoid regression toward the mean and selection bias, yet they are frequently overlooked. The purpose of this paper is to provide a framework for program evaluators to select the strongest, most feasible design for program evaluation and to provide general strategies to improve program evaluation.

Outcome Designs

Several designs can be implemented to objectively measure outcomes. The strongest is the randomized controlled design. When randomized control is not possible, evaluators may choose to employ a historical, intent-to-treat design or, when that is not possible, a matched comparison design. Each design has limitations; selecting the appropriate one depends on the context of the evaluation. The figure below illustrates a flow chart for selecting a design. The table following provides an example of each design.