SHADOW AI IN HEALTHCARE
Finding the AI in your hospital that nobody told the security team about
Shadow AI in mid-market hospitals is more pervasive than most security teams estimate. SaaS vendors embed AI features into tools clinicians already use. Departments adopt AI-augmented tooling without IT or compliance review. Clinical staff paste PHI into public LLM interfaces. Armorstack’s shadow-AI discovery work, Pillar 1 of the AI Adoption Security Framework, produces a complete healthcare-specific inventory in 30 days.
Where shadow AI hides in mid-market healthcare
Shadow AI in hospitals appears in four distinct places:
- SaaS-embedded AI features in tools the organization already licenses. Microsoft 365 Copilot, Google Workspace Gemini features, Slack AI, Notion AI, Zoom AI Companion, Canvas, Webex AI Assistant — all touching organizational data including PHI in many cases. The features were enabled by the vendor without an explicit security review and may not be visible in the organization’s standard SaaS administration consoles.
- Departmental AI-augmented tooling adopted outside IT review. Clinical departments adopting AI scribing tools, marketing departments adopting AI content generation, HR departments adopting AI resume screening, finance departments adopting AI invoice processing — each potentially touching PHI in workflows the security team did not approve.
- Clinical and operational staff use of public LLM interfaces. ChatGPT, Claude, Gemini, Perplexity, and other public LLM tools accessed from work devices or personal devices used for work. Clinical staff paste discharge summaries into public LLMs for summarization, paste documentation for editing assistance, paste patient questions for AI-generated draft responses.
- Third-party clinical AI vendors integrated into Epic or Oracle Health (Cerner) via FHIR APIs or specific EHR partner programs. Each integration potentially processes PHI under Business Associate Agreement terms that may not contemplate the vendor’s AI feature set.
What healthcare-specific shadow-AI discovery looks like
Pillar 1 discovery work, applied to mid-market hospitals, combines four signal sources:
- API-based discovery against the major SaaS administrative consoles your organization uses — Microsoft 365, Google Workspace, Slack, Zoom, and the specific clinical SaaS in your environment
- Network telemetry analysis against known AI service domains, identifying traffic to public LLM endpoints and AI-vendor APIs from organizational devices
- Endpoint telemetry including browser extension and application usage data identifying AI tools in use
- A structured staff survey calibrated to clinical operations — not the typical “do you use AI” question but a workflow-specific battery that surfaces use cases staff may not consciously categorize as “AI”
The output is an inventory classified by department, by data type (PHI / non-PHI), by vendor authorization status, and by clinical workflow involvement. Mid-market hospital discovery exercises typically find an inventory two to four times larger than what the IT team estimated before the exercise began.
Frequently Asked Questions — Shadow AI in Healthcare
How does shadow AI become a HIPAA issue?
Three primary paths: (1) PHI is disclosed to an unauthorized AI service that is not under a Business Associate Agreement, becoming an impermissible disclosure under the Privacy Rule and potentially a reportable breach; (2) PHI processed by an AI vendor is used to train the vendor’s models without explicit authorization, raising 164.502 use questions; (3) AI-generated content based on PHI is shared inappropriately, replicating disclosure paths the security team’s existing DLP rules don’t recognize because they were built for human action patterns, not AI output patterns.
Can we just ban AI use by clinical staff?
Bans rarely work in practice and often create worse outcomes than governance. Clinical staff under workflow pressure will find ways to use AI to accelerate documentation and decision-making; if the organization has banned approved tools, staff use unapproved tools on personal devices, removing all visibility. The more durable approach is governance — an Acceptable Use Policy that defines what AI use is permitted, with what data, on what devices — combined with visibility into actual use.
Will the discovery work identify our clinical AI vendors?
Yes. Pillar 1 discovery is explicitly scoped to enumerate vendor-supplied AI in addition to staff-use AI. The discovery work surfaces vendor AI features that may have been enabled without explicit organizational authorization and produces the inventory of clinical AI vendor relationships your governance committee can review.
How does shadow-AI discovery feed into our existing Joint Commission preparation?
Joint Commission information management standards require organizations to know how clinical information is being used and protected. The shadow-AI inventory becomes documentary evidence that the organization knows what AI is touching clinical information, classified appropriately, with governance documented. Pillar 4’s deliverables feed into Joint Commission survey preparation directly.
What happens with the inventory after the assessment?
The inventory is yours. Some organizations operate it as a living artifact maintained quarterly. Others integrate it into their existing IT asset inventory or vendor risk management program. Armorstack can operate the inventory as a managed service for organizations that prefer ongoing managed shadow-AI discovery, but the assessment itself produces the inventory as a deliverable regardless of whether the relationship continues.
See what AI is actually inside your hospital.
Apply for the free 30-day AI Risk Assessment. Pillar 1 alone is worth the engagement.