Health Services and Outcomes Research Methodology

Weighted estimators are commonly used for estimating exposure effects in observational settings to establish causal relations. These estimators have a long history of development when the exposure of interest is binary and where the weights are typically functions of an estimated propensity score. Recent developments in optimization-based estimators for constructing weights in binary exposure settings, such as those based on entropy balancing, have shown more promise in estimating treatment effects than those methods that focus on the direct estimation of the propensity score using likelihood-based methods. This paper explores recent developments of entropy balancing methods to continuous exposure settings and the estimation of population dose-response curves using nonparametric estimation combined with entropy balancing weights, focusing on factors that would be important to applied researchers in medical or health services research. The methods developed here are applied to data from a study assessing the effect of non-randomized components of an evidence-based substance use treatment program on emotional and substance use clinical outcomes.

Understanding how best to estimate state-level policy effects is important, and several unanswered questions remain, particularly about the ability of statistical models to disentangle the effects of concurrently enacted policies. In practice, many policy evaluation studies do not attempt to control for effects of co-occurring policies, and this issue has not received extensive attention in the methodological literature to date. In this study, we utilized Monte Carlo simulations to assess the impact of co-occurring policies on the performance of commonly-used statistical models in state policy evaluations. Simulation conditions varied effect sizes of the co-occurring policies and length of time between policy enactment dates, among other factors. Outcome data (annual state-specific opioid mortality rate per 100,000) were obtained from 1999 to 2016 National Vital Statistics System (NVSS) Multiple Cause of Death mortality files, thus yielding longitudinal annual state-level data over 18 years from 50 states. When co-occurring policies are ignored (i.e., omitted from the analytic model), our results demonstrated that high relative bias (> 82%) arises, particularly when policies are enacted in rapid succession. Moreover, as expected, controlling for all co-occurring policies will effectively mitigate the threat of confounding bias; however, effect estimates may be relatively imprecise (i.e., larger variance) when policies are enacted in near succession. Our findings highlight several key methodological issues regarding co-occurring policies in the context of opioid-policy research yet also generalize more broadly to evaluation of other state-level policies, such as policies related to firearms or COVID-19, showcasing the need to think critically about co-occurring policies that are likely to influence the outcome when specifying analytic models.

Difference-in-differences (DID) is a popular approach in observational and quasi-experimental studies to estimate the effects of a treatment with discrete statuses. In many studies, however, the treatment can have a range of dosages or exposure levels. In our paper, “medical homeness” is a semi-continuous score ranging from 0 to 100 to indicate the extent to which a patient-centered medical home model is achieved. We developed a causal DID approach to estimating the effects of a treatment with semi-continuous dosages. The proposed approach allows for mixed-type designs as well as different propensity models. We applied the proposed approach to evaluate the dosage effect of medical homeness scores on the utilization and quality of children’s health care. We found that there was a roughly linear effect of medical homeness scores on the annual number of visits to doctor offices when medical homeness scores were below 60 points. The number of office visits did not further increase when medical homeness scores were above 60. A similar relationship was found between medical homeness scores and ratings for health care quality.

Medicaid claims are an important, but underutilized source of data for neonatal health services research in the United States. However, identifying live births in Medicaid claims data is challenging due to variation in coding practices by state and year. Methods of identifying live births in Medicaid claims data have not been validated, and it is not known which methods are most appropriate for different research questions. The objective of this study is to describe and validate five approaches to identifying births using Medicaid Analytic eXtract (MAX) from 45 states (2006–2014). We calculated total number of MAX births by state-year using five definitions: (1) any claim within 30 days of birth date listed in personal summary (PS) file, (2) any claim within 7 days of PS birth date, (3) live birth ICD-9 in inpatient or other therapies file, (4) live birth ICD-9 code in inpatient file, (5) live birth ICD-9 in inpatient file with matching PS birth date. We then compared the number of MAX births by state and year to expected counts using outside data sources. Definition 1 identified the most births (14,189,870) and was closest to total expected count (98.3%). Each definition produced over- and underestimates compared to expected counts for given state-years. Findings suggest that the broadest definition of live births (Definition 1) was closest to expected counts, but that the most appropriate definition depends on research question and state-years of interest.

Commercial health plans need member racial/ethnic information to address disparities, but often lack it. We incorporate the U.S. Census Bureau’s latest surname list into a previous Bayesian method that integrates surname and geocoded information to better impute self-reported race/ethnicity. We validate this approach with data from 1,921,133 enrollees of a national health plan. Overall, the new approach correlated highly with self-reported race-ethnicity (0.76), which is 19% more efficient than its predecessor (and 41% and 108% more efficient than single-source surname and address methods, respectively, P < 0.05 for all). The new approach has an overall concordance statistic (area under the Receiver Operating Curve or ROC) of 0.93. The largest improvements were in areas where prior performance was weakest (for Blacks and Asians). The new Census surname list accounts for about three-fourths of the variance explained in the new estimates. Imputing Native American and multiracial identities from surname and residence remains challenging.

The Medicare Consumer Assessment of Healthcare Providers and Systems (CAHPS®) surveys collect standardized information about patient experiences of care from nationally representative samples of people with Medicare to support consumers’ enrollment choices and enable the Centers for Medicare & Medicaid Services to monitor care quality and incentivize high quality patient-centered care. Since 2007, protocols for data collection, analysis, and reporting have evolved to address expanded Medicare coverage options and a shift from a single survey vendor to a model in which health plans hire approved vendors to administer the survey. During that time, response rates for all types of surveys have declined; increasing effort has gone toward increasing survey participation, especially among people whose preferred language is not English. In this paper, we describe the history, goals, and current use of the Medicare CAHPS surveys. We also summarize key methodological issues, such as sample design, field implementation and data cleaning, adjustment, scoring, and report production. Additionally, we discuss issues that may arise more generally in managing a large, annual national survey that has direct impact on policy, and consider how a long-running survey of this nature may need to evolve to reflect changes in health care delivery and promote standardization in survey administration while maintaining survey content.

Administrative claims datasets have great potential for health services researchers who wish to evaluate patient care on a large scale across providers, but categorizing patients’ primary health conditions from these data can be challenging. The goal of this work is to describe and evaluate a methodology to assign workers compensation claimants to meaningful groups within back and shoulder injuries using claims data. Claims data from a large multi-state workers compensation insurance dataset were used to assign eligible claimants to condition and subcondition groups using available ICD9 codes. Assignments were evaluated against body part indicators, severity indicators, resource utilization, and specific clinical interventions. Of the 575,967 claimants who met inclusion criteria, 54,066 claimants were designated as shoulder injuries and 118,772 were designated as back injuries. Within back and shoulder injuries, claimants were assigned to more specific groups known as subconditions. For both back and shoulder injuries, there were statistically significant differences between subconditions in several categories of resource utilization (p < 0.01 for all). For each of nine specific clinical interventions, the hypothesized corresponding subcondition had statistically significantly higher utilization than other subconditions (p < 0.01). This methodology could be an important tool to health services researchers who wish to target interventions or examine trends in cost and service utilization among meaningful groups of claimants.

Medical error is one of the most critical challenges facing medical services. They pose a substantial threat to patient safety, and their costs draw attention from policymakers, health care planners and researchers. We aim to make a realistic estimation of medical error incidence and related costs and identify factors influencing this incidence in Iranian hospitals. In the first phase of this multi-method study, through two reviews of systematic reviews and a meta-analysis, we will estimate the incidence of medical errors and the strategies to reduce them. We will extract available data among 41 hospitals supervised by the East Azerbaijan University in the second phase. We will also develop a model and use a Delphi method to predict medical errors incidence and calibrate our model output using the Monte Carlo simulation. We will compare this estimation with the incidence rate based on meta-analysis results from the first phase. In the third phase, we will investigate the relationship between several factors potentially influencing medical error incidence. In the fourth phase, we will estimate costs associated with medical errors by conducting a patient records review and matching those with claims related to medical errors. In the fifth phase, we will present a policy brief related to strategies for medical errors and associated costs reduction in Iran. Our findings could benefit Iranian and policymakers in other countries to reduce medical errors and associated costs.