Health Services and Outcomes Research Methodology

Patient relevant outcomes, such as cognitive functioning and functional status, measured using questionnaires, have become important endpoints in medical studies. Traditionally, responses to individual items are simply summed to obtain a score for each patient. Recently, there has been interest in another paradigm, item response theory (IRT), proposed as an alternative to summed scores. The benefits of the use of IRT are greatest, when it is used in conjunction with a calibrated item bank. This is a collection of items, which have been presented to large groups of patients, whose responses are used to estimate the measurement properties of the individual items. This article examines the methodology surrounding the use of IRT to construct and calibrate an item bank and uses the AMC Linear Disability Score project, which aims to develop an item bank to measure functional status as expressed by the ability to perform activities of daily life, as an illustration.

The Medicare program, private insurers, and managed care organizations reimburse hospitals for inpatient admissions using the Diagnosis Related Group (DRG). The DRG is determined from a complicated algorithm based on patient medical records. Previous studies generated concerns about ‘DRG upcoding’, in which incorrect DRG codes may be selected by the hospital to obtain higher reimbursement. Insurers rely on expensive manual audits of claims to verify the appropriateness of the DRG coding. A statistical system that can adaptively detect claims with incorrect DRG codes would provide a powerful improvement to current practice. This paper describes two aspects of the statistical system that provides proof that the concept is viable. The first aspect of the paper is the design of a hierarchical Bayesian model to be applied to claims data (without audit) to estimate the probability that a claim is coded incorrectly. The second aspect of the paper is the use of the Bayesian model to aid in the selection process of claims to audit by proposing that a claim should be investigated if the predicted recovery is more than the cost of auditing that claim. This approach improves upon that used currently by auditing 88% of the claims and recovering 98% of the overpayments. While these results improve upon the current approach for determining which claims to investigate, they are based on data that have been systematically selected for audit based on one insurer's past experience. Future work will create an adaptive system to determine the selection of claims to audit from the entire paid claims database, and that can be generalized for use by other insurers.

To slow the spread of COVID-19, most countries implemented stay-at-home orders, social distancing, and other nonpharmaceutical mitigation strategies. To understand individual preferences for mitigation strategies, we piloted a web-based Respondent Driven Sampling (RDS) approach to recruit participants from four universities in three countries to complete a computer-based Discrete Choice Experiment (DCE). Use of these methods, in combination, can serve to increase the external validity of a study by enabling recruitment of populations underrepresented in sampling frames, thus allowing preference results to be more generalizable to targeted subpopulations. A total of 99 students or staff members were invited to complete the survey, of which 72% started the survey (n = 71). Sixty-three participants (89% of starters) completed all tasks in the DCE. A rank-ordered mixed logit model was used to estimate preferences for COVID-19 nonpharmaceutical mitigation strategies. The model estimates indicated that participants preferred mitigation strategies that resulted in lower COVID-19 risk (i.e. sheltering-in-place more days a week), financial compensation from the government, fewer health (mental and physical) problems, and fewer financial problems. The high response rate and survey engagement provide proof of concept that RDS and DCE can be implemented as web-based applications, with the potential for scale up to produce nationally-representative preference estimates.

With health care reform passing in the United States, much effort is directed toward developing and disseminating comparative information on standardized processes of care for health care providers. We propose the use of Bayesian multilevel item response theory models to estimate hospital quality from multiple process measures and to assess geographical variation in hospital quality. Our approach fully incorporates the nesting structure of measures, patients, hospitals, and various levels of geographical units to provide a summary of hospital quality. A national dataset of patients treated for a heart attack, heart failure, or pneumonia illustrates our methods. We find considerable geographical differences in hospital quality for these conditions with variations across census regions and states accounting for slightly more than 10% of the total variation. Some states performed well for all three conditions (e.g., the respective posterior probabilities of having better than the national average performance was close to 1 in Iowa, New Jersey, South Dakota, and Wisconsin). In contrast, quality of other states varied across conditions (e.g., the corresponding posterior probability was close to 1 in Massachusetts for heart attack and heart failure quality, but less than .5 for pneumonia care). Our framework provides a comprehensive approach to assessment of hospital performance at both regional and national levels, and might be informative for policy development.

Intervention trials with nested designs seek to balance sites randomized regarding key site characteristics. Among the goals of such site-level balancing is to accrue patient-level equivalence among treatment arms. We investigated patient-level equivalence in a cluster randomized controlled trial, which balanced study waves on site-level characteristics. The Behavioral Health Interdisciplinary Program—Collaborative Chronic Care Model project utilized a stepped wedge design to stagger implementation of an evidence-based team-oriented mental health patient management system at 9 Veteran Affairs Medical Centers. Study sites were balanced on eight site-level characteristics over time (3 balanced waves [consecutive time periods] with 3 sites per wave) to minimize trend. Sites were balanced on selected site-level characteristics but not on patient-level variables. We explored internal differences in patient demographics across the three study waves. Eligible patients had at least two visits to a participating mental health clinic in the prior year and did not have a diagnosis of dementia (n = 5,596). We found modest but statistically significant inter-site differences in age, marital status, ethnicity, service-related disability, mental health hospitalizations, and selected diagnoses by study wave. Although many of the differences in patient demographics by study wave were statistically significant, only a few results were practically meaningful as measured by effect size. A bipolar diagnosis (49.0%, 21.0%, 17.0% in waves 1–3, respectively; Cramer’s V = 0.3124) and Hispanic ethnicity (2.9%, 29.6%, 2.0% in waves 1–3, respectively; Cramer’s V = 0.3949) resulted in differences that were considered a ‘moderate’ effect size. The number of patient characteristics that were both statistically and meaningfully different by study wave among all possible site assignments was comparable to the 34 most balanced site assignments identified in our balancing algorithm. Using a balancing algorithm to reduce imbalance among site characteristics across time periods did not appear to negatively affect the balance of patient characteristics across sites over time. A site-level balancing algorithm that includes characteristics with a direct relationship to relevant patient-level factors may improve the overall balance across key elements of the study, and aide in the interpretation of results.

Propensity score methodology can be used to help design observational studies in a way analogous to the way randomized experiments are designed: without seeing any answers involving outcome variables. The typical models used to analyze observational data (e.g., least squares regressions, difference of difference methods) involve outcomes, and so cannot be used for design in this sense. Because the propensity score is a function only of covariates, not outcomes, repeated analyses attempting to balance covariate distributions across treatment groups do not bias estimates of the treatment effect on outcome variables. This theme will the primary focus of this article: how to use the techniques of matching, subclassification and/or weighting to help design observational studies. The article also proposes a new diagnostic table to aid in this endeavor, which is especially useful when there are many covariates under consideration. The conclusion of the initial design phase may be that the treatment and control groups are too far apart to produce reliable effect estimates without heroic modeling assumptions. In such cases, it may be wisest to abandon the intended observational study, and search for a more acceptable data set where such heroic modeling assumptions are not necessary. The ideas and techniques will be illustrated using the initial design of an observational study for use in the tobacco litigation based on the NMES data set.

First introduced in early 2000s, the accountable care organization (ACO) is designed to lower health care costs while improving quality of care and has become one of the most important coordinated care technologies in the United States. In this research, we use the Medicare fee-for-service claims data from 2009–2014 to estimate the heterogeneous effects of Medicare ACO programs on hospital admissions across hospital referral regions and provider groups. To conduct our analysis, a model for a difference-in-difference study is embellished in multiple ways to account for intricacies and complexity with the data not able to be accounted for using existing models. Of particular note, we propose a Gaussian mixture model to account for the inability to observe the practice group affiliation of physicians if the organization they worked for did not become an ACO, which is needed to ensure appropriate partitioning of variation across the different units. The results suggest that the ACO programs reduced the rate of readmission to hospital, that the ACO program may have reduced heterogeneity in readmission rates, and that the effect of joining an ACO varied considerably across medical groups.

Data derived from electronic health records (EHR) are heterogeneous with availability of specific measures dependent on the type and timing of patients’ healthcare interactions. This creates a challenge for research using EHR-derived exposures because gold-standard exposure data, determined by a definitive assessment, may only be available for a subset of the population. Alternative approaches to exposure ascertainment in this case include restricting the analytic sample to only those patients with gold-standard exposure data available (exclusion); using gold-standard data, when available, and using a proxy exposure measure when the gold standard is unavailable (best available); or using a proxy exposure measure for everyone (common data). Exclusion may induce selection bias in outcome/exposure association estimates, while incorporating information from a proxy exposure via either the best available or common data approaches may result in information bias due to measurement error. The objective of this paper was to explore the bias and efficiency of these three analytic approaches across a broad range of scenarios motivated by a study of the association between chronic hyperglycemia and 5-year mortality in an EHR-derived cohort of colon cancer survivors. We found that the best available approach tended to mitigate inefficiency and selection bias resulting from exclusion while suffering from less information bias than the common data approach. However, bias in all three approaches can be severe, particularly when both selection bias and information bias are present. When risk of either of these biases is judged to be more than moderate, EHR-based analyses may lead to erroneous conclusions.