How to Build Confidence in Analytic Results

Distilling millions of records into meaningful measures requires meticulous attention to detail; from initial specifications to final programming, precision is everything. As an analyst, my reputation rests on the accuracy of my work, and I need to feel confident in my results before releasing them to others.

One of the most effective ways I build that confidence is by benchmarking my statistics against reliable, external sources – i.e., doing a cross-check. For example, I needed to identify members of the 2022 Minnesota Senior Health Options (MSHO) plans for a new project, but I did not have a definitive list of contract and plan identifiers. I tried searching plan names in CCW’s Plan Characteristics file, but they were not useful for identifying MSHO. After using deductive reasoning to zero in on what I believed to be MSHO contract and plan IDs in that file, I needed to know if my logic held water. After calculating MSHO enrollment based on beneficiaries having those contract and plan IDs in the Master Summary Beneficiary File (MBSF), then I cross-referenced my totals against enrollment figures from the Minnesota Department of Health:

Image 1. MSHO enrollment by plan (June 2022) based on my work

Image 2. MSHO enrollment by plan (June 2022) according to the enrollment figures from the Minnesota Department of Health

Although my enrollment figures did not match MDH’s exactly, the small variations can easily be assumed to stem from slight differences in data sources (i.e., the MBSF vs. MDH’s database) Some plan names in my data pull were not an obvious match to MDH’s column headings, but a Web search confirmed my presumed alignments, e.g., “SeniorCare Complete” = “South Country Health Alliance”).

To further validate my project, I compared my 2022 percentage of full duals aged 65+ in Minnesota (8.4%) against KFF's most recent Dual-Eligible Individuals as a Share of Medicare Enrollment by Age figure from 2021 (8%). Despite the one-year difference, I felt the consistency between the two figures provided support for the quality of my work.

Image 3. Percentage of full dual eligibles in Minnesota (2021) according to KFF.org

Finding a perfect comparison can be difficult. When I am having difficulty, I try to find something at least in the same vein. After digesting available methodology, I ask myself, given what I know about differences in data, methods and timeframes, are these two sets of numbers consistent with each other? If the difference is difficult to explain, I do more checking of my work or look for other data points to compare to. When I dig deeper, something usually comes to light that leads me to either feel more comfortable with my results or more convinced that there is something wrong. Either way, the process of cross-checking gets me closer to the high-quality work that I strive for.

High Quality Cross-Check Sources

If you are looking for high-quality sources of metrics to which you can compare your work, here are some of my go-tos:

• Internal Historical Reports: Always compare new data against previous versions of a vetted report to spot unexpected values and trends.
• Data.CMS.gov: Data from Data.CMS.gov is a gold standard for Public Use Files (PUFs) and aggregated Medicare administrative data.
• Kaiser Family Foundation (KFF): KFF.org is an essential, reliable source for health policy statistics and trends.
• State Health Departments: State Health Departments offer publicly available reports as shown in the example above.
• Google Scholar: Leverage the power of the Google Scholar search engine for finding peer-reviewed journal articles that contain vetted baseline figures.

Turning complex data sets into meaningful measures involves countless logic decisions. By finding a reliable comparison as a cross-check, you can move forward with confidence that your results—and your reputation—are secure.