A clinician types a patient summary into ChatGPT to draft a referral letter. It takes 30 seconds and saves 10 minutes.
It may also constitute an unauthorized disclosure of protected health information under HIPAA. One prompt. One violation. One mandatory breach notification.
This is the gap that most healthcare AI guidance does not address. The question is not whether AI is useful in clinical settings. It clearly is. The real challenge is how to use it without crossing compliance lines that carry significant legal and regulatory consequences.
This guide explains what HIPAA actually requires, which AI tools are and are not compliant, and how to structure prompts that deliver useful results without exposing your organization to unnecessary liability.
What HIPAA Says About AI
HIPAA predates modern AI by decades. But its requirements are intentionally technology-neutral, and they apply fully to any AI system that processes protected health information.
Three rules govern this space.
The Privacy Rule controls when PHI can be used or disclosed. The Security Rule requires safeguards, including access controls, encryption, and audit logs for electronic PHI. The Breach Notification Rule requires organizations to report unauthorized disclosures, including accidental ones, through consumer AI tools.
The proposed HIPAA Security Rule updates would strengthen cybersecurity requirements, including multi-factor authentication, encryption, and more rigorous risk analyses. AI systems that process PHI must be included in these assessments.
As AI adoption grows across healthcare, the free AI in Healthcare course by Great Learning Academy offers a practical introduction to key concepts and applications in healthcare AI.
Everything becomes mandatory, including encryption, multi-factor authentication, and comprehensive asset inventories that must now explicitly cover AI systems. AI tools that process PHI must be included in your organization's formal risk analysis.
The enforcement trend reinforces this. The HHS Office for Civil Rights resolved 21 HIPAA enforcement cases in 2025, with 76% of those cases including penalties for risk analysis failures. The shift is from identifying compliance risks to proving you acted on them with documented remediation.
Which AI Tools Are Not HIPAA-Compliant
This is where most healthcare professionals make mistakes.
Consumer versions of ChatGPT (Free, Plus, Pro, and Team plans) cannot be used with PHI under any circumstances. No internal policy changes that. No BAA exists for consumer tiers. A single patient detail entered into these tools can trigger breach notification obligations.
The same applies to standard consumer versions of Claude, Gemini, and similar tools. The AI detecting patterns in your prompt is one thing.
The data logging, model training pipelines, and third-party integrations behind the interface are another. You cannot verify what happens to PHI once it enters a consumer product.
Vector embeddings created from patient data also count as PHI. They require the same encryption and access controls as source records. This catches many technical teams off guard.
Which AI Tools Can Be HIPAA-Compliant
Enterprise configurations with signed Business Associate Agreements change the equation.
Microsoft Azure OpenAI Service can be configured for HIPAA-eligible workloads, but the specific services covered by the BAA must be verified. Not every Azure feature is covered by default.
In early 2026, OpenAI launched ChatGPT for Healthcare as a dedicated enterprise product with BAA support, data residency controls, and audit logging. This requires enterprise procurement, not an individual subscription.
A BAA alone is not enough. You still need to confirm how the vendor uses or trains on data, whether PHI leaves the protected environment through logs or third-party integrations, and whether your specific use case falls within the scope of the agreement.
Beyond selecting compliant tools, healthcare organizations also need a clear adoption strategy. Understanding the fundamentals of implementing AI in healthcare can help teams move from experimentation to responsible deployment.
How to Prompt Without Using PHI
The most practical compliance approach for clinicians is de-identification before prompting. HIPAA's Safe Harbor method requires removing 18 specific identifiers before data can be used freely.
These include names, geographic data below the state level, dates (except year), phone numbers, email addresses, Social Security numbers, medical record numbers, and device identifiers.
Once those identifiers are removed, the data is no longer PHI under HIPAA and can be used with any AI tool for drafting, summarizing, or analysis.
Here is what this looks like in practice.
Non-compliant prompt: "Summarize the clinical notes for John Martinez, DOB 04/12/1968, MRN 7734821, admitted to St. Mary's on May 3rd with chest pain."
Compliant prompt: "Summarize the following clinical notes for a 57-year-old male admitted with chest pain. [paste de-identified notes]".
The second prompt produces the same output. Nothing in it constitutes PHI. No BAA required.
Even compliant AI usage depends on how instructions are structured. A prompt engineering guide can help healthcare professionals learn the basics of crafting effective prompts, enabling them to generate more accurate, relevant, and useful outputs.
Practical Prompting Rules for Clinical Settings
1. Remove All 18 HIPAA Identifiers Before Prompting
Age over 89 counts as an identifier. Specific dates beyond the year count. Geographic data more specific than state counts. Build a de-identification checklist and use it before any AI interaction involving patient data.
2. Use Role-Based Framing Instead of Patient-Specific Context
Instead of describing a specific patient, describe a clinical scenario. "A 62-year-old diabetic patient with stage 3 CKD presenting with..." gives the AI everything it needs to generate useful clinical content without identifying anyone.
As organizations explore more advanced AI capabilities, the rise of AI agents in healthcare further highlights the need for strong governance and human oversight.
3. Never Paste Raw EHR Output
EHR exports almost always contain identifiers, even when you intend to share only clinical data. Copy specific clinical observations manually rather than pasting full records.
4. Verify Your Tool's BAA Coverage Before Use
If your organization has an enterprise agreement with a HIPAA-eligible AI platform, check which specific features are covered. A signed BAA does not automatically cover every product feature the vendor offers.
5. Document Your AI Use in Risk Analyses
The proposed 2025 HIPAA Security Rule update explicitly requires AI tools to be included in organizational risk assessments. Every AI tool that processes PHI in your environment needs to be included in your risk analysis, with documented controls.
6. Do Not Use AI for Final Clinical Decisions Without Human Review
This is both a compliance and patient safety requirement. "The AI said so" is no longer a defensible answer for clinicians, patients, or regulators. Document the human review step in your workflow.
State Laws Now Layer on Top of HIPAA
HIPAA sets the federal minimum. Several states now impose additional requirements, and organizations operating across state lines must comply with the most stringent applicable standard.
| State | Law | Key AI Requirement |
| Texas | TRAIGA | Healthcare providers must disclose AI use to patients no later than the date of treatment; failure may constitute a deceptive trade practice |
| Colorado | AI Act | Imposes additional healthcare-specific governance requirements for high-risk AI systems; effective 2026 |
| California | AB 489 | Additional healthcare AI governance requirements; effective 2026 |
Multi-state healthcare organizations should conduct a jurisdiction-by-jurisdiction review of state AI and privacy laws, as this regulatory landscape is evolving rapidly.
What This Means for Healthcare Leaders
The compliance burden here sits with organizations, not just individual clinicians. A nurse using a consumer chatbot to draft a patient letter is not acting outside the bounds of normal professional behavior. They are doing what people everywhere do with AI tools.
The failure is organizational — no clear policy, no compliant alternative provided, no training on where the line is.
Healthcare leaders and compliance teams need to address three things:
- Documented AI Use Policy: Establish clear guidelines and approved AI tools for staff use.
- De-Identification Workflow: Ensure clinicians remove patient identifiers before using AI tools.
- Formal Risk Analysis: Assess AI tools for privacy, security, and compliance risks before deployment.
The JHU AI in Healthcare Certificate Program addresses this directly. Week 4 covers LLM fundamentals and clinical considerations, including the limitations of AI in healthcare workflows and the regulatory framework around responsible deployment. Week 9 covers health data ethics, HIPAA, GDPR, India's DPDP Act, and EHR data governance.
Masterclass 3 takes this further with a live n8n workflow session on AI-powered pre-visit clinical risk assessment: building compliant, automated intake pipelines that handle patient data correctly from the start.
For healthcare professionals who want to move beyond compliance awareness to lead compliant AI implementation, the AI in healthcare course by Johns Hopkins University provides the practical and regulatory foundation to do so responsibly.
Transform healthcare with AI. Apply now for Johns Hopkins AI in Healthcare Program and enhance patient outcomes with cutting-edge skills
Frequently Asked Questions
1. Can I use ChatGPT for clinical documentation under HIPAA?
Not with consumer plans. ChatGPT Free, Plus, Pro, and Team plans cannot be used with PHI under any circumstances. OpenAI's ChatGPT for Healthcare, launched in early 2026, supports HIPAA-eligible use through enterprise procurement with a signed BAA.
2. What is a Business Associate Agreement in the context of AI?
A BAA is a contract between a covered entity and a vendor that processes PHI on its behalf. It specifies the vendor's obligations to protect that data. Without a signed BAA, using a vendor's AI tool with PHI constitutes an unauthorized disclosure under HIPAA.
3. How do I de-identify patient data before using AI tools?
HIPAA's Safe Harbor method requires removing 18 specific identifiers, including names, dates beyond the year, geographic data below the state level, contact information, and medical record numbers. Data with all 18 identifiers removed is no longer PHI and can be used freely.
4. Do HIPAA rules apply to AI-generated embeddings or summaries?
Yes. Vector embeddings created from patient data count as PHI and require the same encryption and access controls as source records. AI-generated summaries that retain identifying information are also PHI.
5. Is Azure OpenAI HIPAA-compliant?
Microsoft Azure OpenAI Service can be configured for HIPAA-eligible workloads, but you must verify which specific services are covered by your BAA and confirm that PHI does not leave the protected environment through logs or integrations.
6. What happens if a clinician accidentally enters PHI into a consumer AI tool?
It constitutes an unauthorized disclosure under HIPAA and triggers breach notification obligations. Organizations must report the incident to HHS and notify affected individuals depending on the scope and risk of the disclosure.
7. What does the 2025 HIPAA Security Rule update mean for AI?
The proposed update removes the distinction between required and addressable safeguards, making all security specifications mandatory. It also explicitly requires AI tools that process PHI to be included in organizational risk analyses and documented risk management activities.
