Confronting the Measurement Challenge in Healthcare as AI Use Expands

How Do We Measure the Value of Something That Never Happened?

Healthcare organizations are under constant pressure to improve outcomes while making the most of limited resources. Providers track readmissions, emergency department visits, quality scores, screening rates, medication adherence, and countless other metrics. Yet some of healthcare's most valuable outcomes are the events that never occur: a hospitalization avoided, a complication prevented, or a condition identified before it escalates.

These questions are becoming increasingly relevant as healthcare organizations expand patient engagement through emerging technologies, including agentic AI solutions. More frequent outreach creates new opportunities to identify risks earlier, support patients between visits, and extend the reach of care teams. But it also introduces a new challenge: how do organizations measure the value of prevention?

Traditional activity metrics can tell us how many patients were contacted or how many conversations occurred. Determining whether those interactions ultimately lead to improved outcomes, however, requires a broader approach to measurement.

Healthcare Has Historically Measured Events

Traditional healthcare quality programs focus on observable outcomes. We can measure when a patient completed a screening, if a diabetic patient has a recent A1C test, or whether a patient was readmitted within 30 days of discharge. These metrics matter because they provide tangible evidence of performance, but they tell us more about what happened than what was prevented.

A patient who returns to the hospital is a measurable event, while a patient who receives the intervention needed to avoid returning often disappears into the background. Success becomes harder to quantify because the wrong outcome never materializes.

The Value Hidden Inside Conversations

One observation continues to stand out: there is so much that can be learned from a conversation. Not every meaningful patient insight comes from a structured assessment or a risk score.

A patient may mention challenges accessing medications, transportation barriers, difficulties adhering to care plans, or symptoms that have not risen to an urgent level. Conversations can also surface social determinants of health, such as housing instability, food access, or financial pressure that shape a patient’s ability to follow a care plan.

These insights may seem minor in isolation, but they can serve as early indicators of larger issues. A transportation barrier may lead to missed appointments and delayed follow-up care. Confusion about discharge instructions can increase the risk of medication errors or avoidable readmissions if left unaddressed.

The challenge is that meaningful conversations take time, and time has always been one of healthcare's scarcest resources. As healthcare organizations look for ways to expand patient engagement, many are exploring AI technologies that can support more outreach and help uncover hard-to-detect concerns. Increasing the number of touchpoints creates more opportunities to identify barriers to care earlier, before small issues become larger problems.

What matters next is how organizations respond to those insights. The real value comes from connecting patients with the resources, care teams, and interventions that can help address those issues.

Early Outcomes Point to Real Potential

The challenge, of course, is that healthcare organizations cannot wait years to understand whether a new model of patient engagement is making a difference.

That is why many organizations are paying close attention to early indicators of engagement and participation. Activities such as completing Health Risk Assessments (HRAs), attending annual wellness visits, closing care gaps, and maintaining regular contact with care teams have long been associated with better long-term outcomes. While these measures don't tell the whole story, they can provide early signals that patients are becoming more engaged in their care.

Several healthcare organizations using 51% more likely to complete a Health Risk Assessment have reported encouraging results. For example, Guidehealth reported that patients engaged through AI outreach were 51% more likely to complete a Health Risk Assessment, 11% more likely to complete an annual wellness visit, and contributed to a 6% increase in quality and care gap closure rates. Similarly, Medical Mutual reported a 360% increase in its capacity to reach members through AI-enabled chronic care management conversations, including medication adherence outreach, health check-ins, care gap interventions, and annual wellness visit scheduling.

While these results do not prove a specific hospitalization was avoided, they do demonstrate increased participation in activities associated with improved long-term outcomes.

Looking Beyond Traditional Metrics with AI Assistance

If earlier identification leads to faster intervention, engagement may function as a leading indicator of future outcomes. Rather than relying on a single outcome metric, healthcare organizations should evaluate the full engagement journey, from initial outreach through clinical and operational outcomes.

Organizations can evaluate progress across several stages of the engagement journey:

• Access and reach: Contact rate, completed outreach, callback rate, language preference match, ability to reach patients outside traditional office hours.
• Engagement quality: Completed assessments, patient satisfaction, conversation completion, patient-reported understanding of instructions.
• Clinical signal detection: New or worsening symptoms identified, medication concerns surfaced, social determinant barriers documented, care gaps identified and closed.
• Care team response: Escalation rate, time to review, time to callback, follow-up completion, successful warm transfer or documented handoff.
• Preventive and quality actions: High Risk Assessment (HRA) completion, annual wellness visit scheduling, screening completion, immunization follow-through, quality measure and care gap closure.
• Downstream outcomes: Readmissions, ED utilization, avoidable admissions, no-shows, care gap closure, medication adherence, patient experience trends.

The goal is not to claim that every call prevented an adverse outcome. A stronger approach involves defining a baseline, looking at the expected impact, and tracking whether AI improves the steps that are known to support better outcomes.

Setting the Stage for Responsible AI Implementation and Measurement

As healthcare organizations scale five-minute AI Readiness Self-Assessment, success will depend on more than adoption metrics alone. Leaders need a framework for evaluating whether AI-driven interactions are improving engagement, accelerating intervention, and supporting better clinical and operational outcomes.

At CTG, we help healthcare organizations establish the governance, measurement frameworks, and operational processes needed to assess AI initiatives responsibly and at scale. Whether you're evaluating an AI-enabled outreach platform or expanding an existing program, defining success from the outset is critical.

The question is no longer whether AI will play a role in healthcare, but whether your organization is prepared to realize its full value responsibly and at scale. To better understand your current position, take CTG's five-minute AI Readiness Self-Assessment. You'll receive a personalized readiness score, insight into your strongest and weakest areas, and practical recommendations to help prioritize the next phase of your AI journey.