Description

The AI Nobody Approved: Governing Vendor-Delivered AI in Clinical and Regulated Environments

The session follows one composite case from deployment through harm to remediation. Kestrel Health System is a fictional organization assembled from patterns that recur across real deployments. The case opens where these conversations usually open in practice, with a board asking a simple inventory question that the organization cannot answer, and it proceeds to the adverse event that makes the unanswered question expensive.

The case is used to expose five seams where AI governance fails silently:

Validation. Whether the model was tested against a population resembling the one it now serves, and who holds that evidence.

Decision. Whether a human retains meaningful authority to override the output, and whether the workflow makes overriding realistic rather than theoretical.

Drift. Whether anyone monitors performance after go-live, on what cadence, and against what threshold.

Detection. Whether the organization would recognize a model-contributed adverse event as model-contributed.

Attribution. Who is accountable when the model, the vendor, and the clinician each hold part of the causal chain.

Each seam is paired with the specific question to ask, the evidence that constitutes an acceptable answer, and the contractual or procedural mechanism that makes that evidence obtainable rather than optional. The closing segment translates the findings into language that moves executives, using a worked example that expresses AI exposure in frequency and magnitude terms rather than heat map colors.

Areas Covered in the Session:-

• How AI enters clinical and regulated workflows without passing through an AI review, and why the procurement record does not show it
• A practical method for inventorying models already in production, including those embedded in tools already owned
• The five seams where AI governance fails silently: validation, decision, drift, detection, and attribution
• The specific evidence to demand from a vendor at each seam, and what an unacceptable answer looks like
• Why questionnaire-based assessment does not establish model safety, and what replaces it
• Contract language and procedural controls that make post-deployment evidence obtainable rather than discretionary
• Where the current regulatory picture reaches and where it stops, across device oversight, privacy and security rules, state legislation, and voluntary AI frameworks
• A worked example expressing AI exposure in frequency and magnitude terms suitable for board and audit committee reporting
• A one-page assessment checklist attendees can apply to a live deployment the following week

Background:-

Artificial intelligence is now embedded in clinical and regulated workflows across provider organizations, pharmaceutical manufacturers, medical device firms, and the vendors that serve them. Most of it did not arrive through a formal AI purchase. It arrived inside imaging platforms, scheduling engines, documentation assistants, prior authorization tools, and safety surveillance systems that were procured for other reasons and later updated to include a model. The purchase record shows software. The workflow contains a model.

The result is an accountability gap. Compliance, privacy, quality, and risk functions carry responsibility for outcomes produced by systems they were never given a method to assess. A standard vendor questionnaire establishes whether a supplier has an AI policy. It does not establish whether a model was validated against a population resembling the one it now serves, whether performance has degraded since deployment, who holds accountability when its output contributes to harm, or whether the organization would detect that failure at all.

The regulatory picture reinforces the gap rather than closing it. Device oversight does not reach most administrative and operational AI. Existing privacy and security rules address confidentiality and integrity but were not written for model behavior. State legislation is fragmenting. Voluntary frameworks describe what good governance looks like without specifying what evidence to demand from a named vendor about a named model in a named workflow. This session closes that distance.

Why Should You attend:-

Most organizations cannot answer a basic question: how many AI tools are currently operating inside clinical or regulated workflows, and who approved each one. That gap is not a failure of diligence. It is a failure of method, because the instruments in common use were built to assess software rather than models.

Attendees leave with a working method rather than a set of principles. The session supplies a way to inventory the AI already running in production, including the models embedded in tools the organization already owns; five diagnostic questions that surface governance failure before it produces harm; the evidence standard that separates a real vendor answer from an attestation; and the language that turns a finding into budget and executive attention.

The material comes from operating an enterprise third-party risk and AI governance program at scale inside an academic medical center, not from framework commentary. Every recommendation has been tested against vendors who pushed back on it.

Who Should Attend:-

• Healthcare compliance officers, compliance managers, and compliance analysts
• Privacy officers and privacy program leads
• Healthcare risk managers, patient safety officers, and quality leaders
• Information security governance, risk, and compliance professionals
• Third-party risk, vendor management, and supplier assurance leads
• Procurement and contracting professionals supporting clinical and regulated purchases
• Internal audit and enterprise risk management professionals
• Regulatory affairs and quality professionals at pharmaceutical, medical device, and health technology companies
• Legal counsel supporting healthcare technology and data agreements
• Executives with AI oversight responsibility, including chief compliance officers, chief risk officers, chief information security officers, and chief medical information officers.