Shadow AI is already in your practice

Chances are, your staff is already using artificial intelligence (AI), you just don't know which kind. If you haven't given them a sanctioned tool, they've found one on their own — and if nobody told your front desk to stop using ChatGPT, they're still using it.

Across independent and small group practices, clinicians are regularly turning to consumer AI tools, like ChatGPT or Claude, to draft notes, think through clinical questions and cut down on administrative work.

Even though most are doing so with good intentions, most of their employers have no idea. That blind spot is the core problem with shadow AI.

Asha Palmer, senior vice president of compliance solutions at Skillsoft, has spent considerable time advising organizations on exactly this challenge. She argues that bans aren’t the solution, and practices that refuse to sanction AI use don't eliminate it, they just lose visibility over it.

Medical Economics spoke with Palmer about the real risks of shadow AI in clinical settings, what a practical governance plan actually looks like for smaller practices, and why the conversation with clinicians has to come before the policy does.

The following interview has been edited for length and clarity.

Could you tell us a bit about yourself and Skillsoft?

My name is Asha Palmer. I’m a lawyer by background, a compliance professional who is now in the technology space, because I believe technology has a lot of power and opportunity. It can help compliance professionals build more effective and efficient programs, and that is one of the things we help do at Skillsoft.

We’re a learning company, and we provide compliance learning, metrics and analytics — really, a defensible, scalable compliance learning program that regulators are looking for.

What is the problem with using unapproved “shadow AI” tools? And what risks does it create?

I always like to start with the opportunity those tools create, which is why we see clinicians using them. No one wakes up and says, “I have enough hours in the day,” or, “I’m not overloaded in my daily job.” So we know the opportunity AI presents is effectiveness, efficiency and acting as a thought partner for clinicians who may feel like they are on an island by themselves or have seen something they do not quite understand.

They are often using these tools with the best of intentions, trying to create efficiencies and opportunities in care delivery. The challenge is that if their organizations have not embraced the same attitude — the idea that AI can create those opportunities and efficiencies — then you get shadow use. That is people using it in the shadows because no one knows about it, but they need it as a companion in their work.

Obviously, the challenge is: What is happening to that data? Where is it going? How much can you trust it? Are we using models that are actually trained to support the delivery models these clinicians are providing? We do not have those answers when we are letting clinicians cherry-pick whatever tool they decide to use.

The huge risk is what the tool is doing with the data that you put into it, whether it is protected and whether your patient’s data is protected. So there is both an input concern and an output concern with shadow AI. And when it is being used in the shadow, you cannot see it, so there is a lack of knowledge and visibility for organizations as to what it is being used for, how it is being used and where that data is going.

Why are clinicians repeatedly drawn to shadow AI tools outside official systems?

We see this in health care and beyond. The number one reason people use shadow AI is because the organization has not sanctioned the use of AI in the delivery of its services.

Whether it is a clinician or a lawyer or someone else, organizations need to step up and say: Here are the use cases we know you may want to use it for. Here are the efficiencies and opportunities we know you can gain. Then they need to create a governance structure around how you are actually able to use it.

Where we see the biggest gap, and the largest use of shadow AI, is where organizations do not have a risk appetite or risk tolerance level for these applications, so they tend to ban them entirely. Then people go behind the scenes and use them anyway.

For a small practice, what should a governance plan for AI actually look like?

Lawyers and compliance professionals can overcomplicate governance, but it can actually be quite simple. And in some ways, it is more simple for smaller practices that have visibility into their workforce and the opportunity to bring clinicians to the table.

It starts with bringing people to the table and saying, “Here is how we know you can use it. Have we covered everything?” The first step is establishing use cases. I always say organizations need to think about how people are already using it, how people want to use it and how the organization wants them to use it. Those are three separate buckets that need to be defined.

Then you have to think about the risks associated with those use cases. That could include exposing patient data, inconsistent clinician outcomes and the usual list of possible risks. What we find with most of our customers at Skillsoft is that the risks are not always as broad as people think when they are looking at these use cases in a vacuum.

Once you have the use cases and the risks, the question becomes: How do you mitigate and manage those risks? That is something for the risk professionals, leadership and the clinicians themselves to participate in. Are you going to procure a private version of the tool? Are you going to maintain oversight over inputs? What controls are you going to put in place so you can say, based on those controls, that you are comfortable with clinicians using it?

And in every governance structure, you also have to have testing and monitoring. As you let the technology out into the organization, you have to go back and check whether it is working the way you intended, whether the risks you were worried about are being managed and whether they are lower or higher than you thought. Then you recalibrate the governance structure based on that.

Are there risks here that health systems or practices may not be thinking enough about?

I don’t know that they’re unexpected, but in the clinical environment, AI is still very much in its infant stage, particularly generative AI. It’s still figuring itself out. There are still a lot of mistakes it can make.

From a health care delivery standpoint, the biggest risk probably is not patient data access. I think many physicians and nurses understand that piece. The bigger risk is inconsistencies and inaccuracies in of outcomes or recommendations.

How do you really guarantee that the outputs of AI are aligned with the care model of the organization? That is the biggest concern.

Is it hallucinating? Did it make something up that does not exist? I looked something up the other day and knew it was wrong. It was drawing summaries from the data that created an inaccurate conclusion.

That is a real risk, particularly in health care.

There can be a to treat AI as just an IT problem. How do you make the case that this is also a compliance, workflow and training issue?

IT often procures systems and monitors them. It can tell you, “You can’t paste your organization’s data here,” or push out phishing simulations. But a lot of that is reactive.

There is always a role for reactive governance, for testing and monitoring, for having someone looking over the company’s shoulder. But AI creates such a powerful opportunity for enablement that practice leaders have to look at it from a business standpoint too.

If you are a practice manager responsible for growing the business, looking for new business and creating faster turnaround times for patients, those are use cases that AI can help with. So when the practice manager looks at AI, it has to be through the lens of how it can create opportunity and accelerate the business. Then compliance helps make sure it is being leveraged safely, and information technology helps procure the tools and make sure their actual use is safe.

We always look at it as a multidisciplinary opportunity or problem, depending on how you look at it. Everyone has to be at the table when deciding how the organization wants to use this technology to accelerate without making itself vulnerable.

When a new AI tool does get formally adopted by a health system or a practice, what compliance infrastructure needs to be in place before it goes live?

In compliance, we are very passionate about what we call third-party due diligence. That is really just a fancy way of saying you have to check out the people and companies you are doing business with, and you have to ask them questions about how they are building their product, how they are creating their technology and how they are training their models.

One of the things we really encourage is that as you are onboarding vendors, you ask tough questions about where your data is going, how the models are being changed or trained, whether they are monitoring for fairness and what the accuracy rate is. If you are talking about scribes, then the question becomes how accurate the output is and whether the vendor is testing for that.

What is their governance structure that can help support your governance structure? That is very important before anything goes live. Everyone in the organization has to be comfortable with the answers they get from those vendors.

If a practice leader suspects staff are already using AI tools they do not know about, what should they do first?

Fire them. Just kidding, no way. There is still a huge opportunity to have a conversation with those clinicians and ask why they are using it. What are you not providing that they feel is so necessary to deliver their standard of care that they need this tool?

Listening to the clinician is so important right now, because I do believe most clinicians are trying to uphold the standard of care required by their profession. If the use goes beyond what the organization allows, and there are consequences for that, that is one thing. But there is still an opportunity to ask why and be receptive to the answer, and then see where you can help them do what they need to do in a safer way.

Another thing compliance professionals love is policy. A policy on the safe, accountable and responsible use of the organization’s approved AI tools is an important step to have before any tool goes live.

And when you discover shadow AI use, that is also an opportunity to revisit the policy, if you have one, and ask: What am I missing here? What do I need to expand in my own thinking, and in the organization’s thinking, so that clinicians feel they have what they need to provide care?

Is there anything else physicians and practice leaders should keep in mind?

Using AI is inevitable right now. We are using it in our daily lives, in our professions and in our analysis. So providing a safe, responsible and accountable way for clinicians to use it is the best route to go.

We encourage all of our customers not to ban AI, because that is not a sustainable strategy. Either I am going to use it on my phone or I am going to use it in a way where you can actually see how I am using it.

Right now, the most important thing practice leaders can do is have visibility into how their employees are using AI, because that will give you additional use cases, opportunities and data you may be able to leverage. Providing a sanctioned tool gives you the visibility you need to build the right governance structure.

If people are using it in the shadows, you are losing control of your data, your people and eventually your practice. Nobody wants that.

So the best thing leaders can do is embrace the reality of it, talk to clinicians about how they want to use it, how they could use it and how they are already using it, and then figure out how to protect both the individuals and the organization from the harms that may come with those use cases.