In recent years, the discourse around artificial intelligence in healthcare has increasingly emphasised model performance, often measured through metrics such as accuracy, AUC, or F1-score. While these metrics are important, they can also be misleading when used in isolation, particularly in high-stakes clinical environments.

In a recent exploratory analysis using a publicly available heart disease dataset (see my project on Github), I compared the performance of three commonly used machine learning models: logistic regression, decision trees, and random forests. The results were, at first glance, unsurprising—tree-based models significantly outperformed logistic regression in predictive accuracy, achieving approximately 98% accuracy compared to 79% for logistic regression.

However, a deeper examination reveals why such results should be interpreted with caution. Tree-based models, particularly decision trees, are highly flexible and capable of capturing complex, non-linear relationships within data. This flexibility allows them to achieve high performance on structured datasets. However, it also introduces a well-known risk: overfitting, where models capture dataset-specific patterns that do not generalise to real-world populations. Random forests mitigate this risk by aggregating multiple decision trees, improving robustness and generalizability. Yet even ensemble methods are not immune to deeper structural issues.

This is where insights from Sociotechnical Challenges in ML systems become particularly relevant. A recent paper (link to the full text below) argues that machine learning systems should not be evaluated as isolated technical artefacts, but rather as components of complex sociotechnical systems. It highlights several key challenges that directly inform how we should interpret model performance in healthcare:

1. Data is Not Neutral

Datasets are shaped by:

• historical practices
• institutional processes
• measurement constraints

In my project, the heart disease dataset represents a simplified and potentially biased snapshot of reality. High model accuracy may therefore reflect patterns specific to that dataset, rather than clinically valid or generalisable insights.

2. Generalisation is Context-Dependent

A key argument in the paper is that model performance does not transfer reliably across contexts due to:

• population differences
• clinical practice variation
• data collection inconsistencies

This directly challenges the assumption that a model achieving 98% accuracy in a controlled dataset would perform similarly in real healthcare environments.

3. Human–AI Interaction Matters

The paper highlights that machine learning systems are ultimately used by people, and their effectiveness depends on:

• clinician trust
• interpretability
• usability within workflows

This reinforces the importance of models like logistic regression, which—despite lower accuracy—offer greater transparency and interpretability, making them more suitable for clinical decision-making contexts.

4. Hidden Failure Modes

Another critical insight from the paper is that models can fail in ways that are not visible through standard evaluation metrics. For example:

• systematic bias against certain patient groups
• sensitivity to missing or noisy data
• unintended consequences in deployment

These risks are not captured by accuracy alone, yet they are central to patient safety.

Reframing Model Performance

Taken together, these insights fundamentally change how we interpret the model comparison results. While tree-based models demonstrated superior accuracy, this advantage must be evaluated within a broader context:

• Are the results generalisable?
• Are the predictions explainable to clinicians?
• How does the model behave under real-world data conditions?
• What are the risks of failure, and who is affected?

A marginal gain in predictive performance may not justify the loss of interpretability, especially when decisions impact diagnosis, treatment, or resource allocation.

Toward a Sociotechnical Evaluation Framework

As both the empirical findings and the referenced research suggest, model evaluation in healthcare must move beyond purely technical metrics toward a multi-dimensional, sociotechnical framework that incorporates:

• Predictive performance
• Interpretability and explainability
• Data quality and representativeness
• Clinical workflow integration
• Human factors and usability
• Ethical and regulatory considerations

The key takeaway is not that complex models should be avoided, but that model selection in healthcare is not purely a technical decision.

It is a governance decision, shaped by the interaction between technology, people, and systems. As highlighted in Sociotechnical Challenges in Machine Learning Systems, the success or failure of AI in healthcare depends less on achieving marginal gains in accuracy and more on how well these systems align with the complex realities of clinical practice.

For years, the “Human-in-the-Loop” (HIL) has been the safety blanket of the digital health industry—a comforting slogan used to reassure regulators and clinicians that a person is still “in charge.” However, as we embed AI into complex sociotechnical healthcare organisations, it is becoming clear that HIL is often a misnomer that masks a lack of true agency.

Moving from Automation to Collaboration

In a traditional Human-in-the-Loop model, the system is designed around automation. The AI drives the inference process, and the human expert is relegated to the role of a safety net or a mere labeller who validates the machine’s output. This often leads to automation bias, where clinicians—under immense institutional pressure, reflexively accept algorithmic recommendations.

To build truly resilient health systems, we should pivot toward AI in the Loop (AI2L). As argued by Natarajan et al. (link to the full article below), AI2L is a collaboration-centric approach where the human expert remains the primary decision-maker, and the AI serves as a co-adaptive assistant that provides supplementary insights and perception.

The Sociotechnical Reality: Remote Monitoring as a Case Study

Nowhere is this distinction more critical than in Remote Patient Monitoring (RPM). Consider a continuous glucose monitor (CGM):

• The HIL View: An autonomous agent adjusts insulin levels, seeking human intervention only for calibration.
• The AI2L View: The device provides a rich data stream that assists a clinician and patient in co-developing a care plan.

In this broader context, the loop isn’t just about a single data point; it’s about a Learning Loop that oversees model drift and equity, and a Governance Loop that manages the ethical trade-offs of deployment. When we focus on AI2L, we prioritise the Clinical Loop, ensuring that point-of-care decisions are informed by the patient’s contextual daily life, data that AI alone often misses.

If we evaluate systems solely on AI-centric metrics like precision and recall, HIL systems will always appear superior. But healthcare isn’t a static benchmark. AI2L requires a holistic approach to evaluation that measures:

• Human-AI Interaction Quality: Is the clinician empowered or overwhelmed?
• Sociotechnical Alignment: Does the system adapt to local clinical workflows and cultural nuances?
• Fairness and Trust: Are the data sources credible and the outcomes equitable?

Automation is most effective in well-defined tasks, such as identifying drug-drug interactions. But in the undefinable context of general medical diagnosis, human wisdom is irreplaceable. We need to stop asking if the human is “in” the loop and start ensuring the AI is properly integrated into the expert’s workflow.

We're increasingly relying on third-party Model-as-a-Service (MaaS) solutions in the cloud for everything from radiology scans to patient risk stratification. But this raises a critical governance question:

When a hospital pays for a specific AI model, how do we verify that the cloud vendor is actually running that exact model? What's to stop them from using an older, cheaper, or unvalidated version to cut costs? And how do we prove that the sensitive patient data (PHI) we've sent to the cloud hasn't been silently corrupted by a server error or network glitch?

In healthcare, trusting our vendors isn't enough. We need to be able to verify.

I've been digging into the complex world of AI auditing and came across a fascinating (and highly technical) paper by Wang et al. (2025) titled “AI-Auditor: A Data Auditing Framework for Enhancing the Trustworthiness of AI Models.” It proposes a framework for exactly this: a cryptographic “challenge-response” system. It allows an organisation (like a hospital) to remotely and efficiently audit its cloud provider. It can verify both data integrity (is the data safe?) and model alignment (is this the correct AI model we paid for?).

This moves us from a “trust me” model to a “prove it to me” model, which is precisely what patient safety demands. It's a technical solution, but it solves a fundamental trust, safety, and liability problem. How is your organisation currently validating the third-party AI models you use?