AI isn't a black box — It's a workforce and it's time to govern it like one

Most health care organizations I speak with are not opposed to AI — they’re uncertain about it. They want to understand what they’re agreeing to, who’s responsible if something goes wrong, and whether AI will truly help their teams. These are reasonable questions, and they deserve clearer answers than what most companies provide.

To address this uncertainty, we should reframe how we view AI in health care. Instead of treating it as a black box, we should integrate it like a new team member, with clear expectations, guidance and support. This approach builds trust and helps AI become a reliable part of care delivery, not a source of concern.

Though the biggest question I hear is not what AI can do for health care, but how we can trust it to act responsibly. That trust comes from transparency, accountability, and thoughtful integration of AI into clinical teams, not just installing new tools and hoping for the best.

The deployment mindset is holding organizations back

When organizations treat AI as just another project to roll out, they focus on features and speed. They ask if it can cut down documentation time, find care gaps faster, or handle prior authorizations for clinicians. These are good goals, but they miss a key issue: who is responsible for the system’s actions, and what happens if it makes a mistake?

Without a clear governance framework, there is no good answer to that question. Recommendations appear without explanation, mistakes go unnoticed because there is no process to catch them, and clinicians get results they did not request and have no formal way to respond. This impacts trust in these tools and the technology meant to help ends up making things harder.

This is often taken as doctors resisting technology. But in reality, it’s that clinicians don’t want to be held responsible for decisions made by systems they cannot question or change. As such, we must move the conversation from this being a technology problem to a governance problem that leaders need to address.

Treating AI as a governed workforce, not a black box

It’s more helpful to think about AI the way a well-run organization thinks about its staff. In a well-run clinic, everyone knows their role, and there’s a clear way to ask for help when things get tricky. People know what’s happening, can look back at decisions, and see how things are going. And if something goes wrong, there’s a way to figure out what happened and fix it together.

The same structure needs to apply to any AI system you bring into the workflow. Before deployment, each system should have the equivalent of a job description — a clearly defined scope that specifies what it is authorized to do and not do. Implementation should be treated like onboarding — the time invested upfront to make sure clinicians understand how the system works, what its outputs mean, and when to push back. Once the system is live, performance should be monitored continuously, not just reviewed when a problem surfaces, so that issues can be caught and corrected before they affect patient care. And when the AI produces output that could change a clinical decision, there should be a clear chain of command — someone with the right context and authority to review it, weigh in, and take responsibility for what happens next.

Organizations that have established these governance structures are already seeing positive results. For example, prior authorizations are processed more quickly and encounter fewer disputes, while care managers can dedicate more time to patients who truly need their support rather than being overwhelmed by administrative tasks. Similarly, clinicians are seeing AI recommendations as genuinely useful input, rather than something to disregard.

Ultimately, this progress is the result of both technological advances and thoughtful and effective governance.

Human oversight is not a limitation. It’s what makes AI trustworthy.

Often, people see human oversight as a temporary first step that can be removed once everyone feels comfortable with the tool — but in health care AI, this is the wrong approach. Having humans involved is how AI earns the trust it needs to take on more responsibility over time.

When clinicians can see the reasoning behind a recommendation, speak up if they disagree, and know that someone is paying attention to their feedback, their perspective on the system changes. They begin to view suggestions as support, especially as they’re able to keep their voice and expertise at the center of care. However, clinicians don’t accept every suggestion or piece of advice; they push back and ask questions, knowing that this feedback only helps the process get stronger.

In a peer-reviewed study, physician override rates initially reached 87%. When AI systems clearly displayed their confidence levels and reasoning, the override rate dropped to 33%, with high-confidence recommendations seeing overrides as low as 1.7%. This suggests that what clinicians needed was not a better model, but a clearer window into how the model was thinking. When that visibility was present, clinicians were far more likely to engage with the AI's output rather than dismiss it, and that shift in behavior is exactly what a well-governed AI system is designed to produce.

Good oversight makes sure these new tools are used in ways that actually help patients and teams.

A practical starting point

Health care organizations don’t need to change everything at once to get started. Focusing on one well-defined workflow, like prior authorization, care gap identification, or clinical documentation, can be enough to set up a governance model that can grow over time. The most important thing for teams to remember is that being disciplined with your first step is key.

Before launching any system, make sure everyone’s clear on what it’s supposed to do and what it shouldn’t touch. Build in checkpoints where real people can review and weigh in, especially when decisions affect patient care. Get everyone involved early, especially the clinicians who will be using these tools day-to-day and make it easy to track progress from the start.

The future of these tools isn’t about the flashiest tech or the most complex models. It’s about whether you’re putting the right guardrails and routines in place to make sure these changes are helping the people doing the work.

Sundar Subramanian is the CEO of Zyter|TruCare