3.1. The human–machine association as the fundamental decision-making unit

This article argues that the nature and functional positioning of clinical decision-making AI systems are not determined by fixed physical attributes, but rather are constituted through human–machine interaction (see Ji, Reference Ji2020, pp. 60–61). Analysis of such systems should focus not on their ontological reality, but on how they appear to humans during interaction, and on humans’ corresponding perceptions, experiences, and responses (see Wang, Reference Wang2024, p. 20). In the decision-making process, system outputs and clinicians’ judgements become intertwined, and the final decision cannot be reduced to either party’s individual decision or a simple aggregate of both. Therefore, the human–machine association should be regarded as the fundamental unit of clinical decision-making behaviour (see Beckers and Teubner, Reference Beckers and Teubner2023).

In practice, the patterns and degrees of human–machine decision integration vary. Medical practitioners’ diagnostic workflows typically encompass rule-based and heuristic processes. Rule-based processes follow explicit clinical guidelines, offering standardisation and reproducibility, and are often applied to conditions with established research. Heuristic processes, however, rely primarily on patterned cognitive frameworks—even intuition—developed through prolonged learning and practice, resulting in a black-box decision-making process. In this process, clinicians rapidly form preliminary hypotheses by associating similar diseases, recalling past cases or events, or anchoring specific information. Heuristic approaches are often applied to complex conditions (see Terry, Reference Terry2019, pp. 139–140), holding significant implications for medical advancement. However, conclusions derived through this method still require reflective validation. Notably, these two diagnostic approaches align with AI systems based on rule-based algorithms and deep learning. Generally, simpler conditions involve more well-established medical knowledge and clearer diagnostic standards. In such cases, AI systems should possess the capability to perform a stable, validated task within an explicit use envelope, and independently execute complete decision-making processes, while clinicians’ subsequent judgements can often build upon the diagnostic pathways provided by the AI system. Here, the relationship between AI and clinicians resembles a linear process of AI autonomous decision-making + clinician review. The system exhibits strong autonomy, with minimal interpenetration between human and machine judgements, closely approximating an agency model (where AI independently replaces human decision-making). Yet, since clinicians remain involved, this does not constitute true agency. In comparison, the diagnosis of complex conditions relies more heavily on deeper interaction and communication between AI and medical professionals, with final conclusions emerging from mutually stimulating cognitive processes. Current technological trends also validate this approach. For instance, RDguru, a conversational AI diagnostic model, narrows differential diagnoses through multi-round dialogue mechanisms, aiding rare disease diagnosis (see Yang et al., Reference Yang, Shu, Duan and Li2025). Similarly, the KAMAC system achieves breakthroughs in complex clinical scenarios by emulating multidisciplinary expert team collaboration in real medical decision-making. This enables large language model agents to initiate knowledge-driven discussions with one or more expert agents, dynamically assemble and expand expert teams based on evolving clinical contexts, review and update decisions, and ultimately reach conclusions (see Wu et al., Reference Wu, Huang, Deng, Qiao, Razzak and Xie2025). Similarly, the core concept behind Microsoft’s newly developed MAI-DxO clinical decision support system draws inspiration from the collaborative patterns observed in real medical teams (see AIGC Open Community, 2025). This demonstrates that communication and interaction models play a pivotal role in advancing technological development and enhancing clinical decision-making capabilities. As AI technology evolves and is applied to increasingly complex problems, its interactive characteristics will inevitably become increasingly prominent.

In summary, the relationship between clinical decision-making AI systems and their users forms a spectrum ranging from partnerships approaching agency to deeply interactive partnerships. The deeper the interaction, the less distinct the division of labour between humans and machines becomes, and the less reducible the final decision.Footnote ¹ It is precisely through this focus on the unique characteristics of human–machine interactive decision-making that we can more appropriately understand the auxiliary functional positioning of clinical decision-making AI systems. Auxiliary function does not establish a master–servant relationship between humans and machines, nor does it mean that medical professionals’ treatment plans and value standards override AI judgements. Auxiliary function signifies, on one hand, recognition of AI’s autonomous decision-making capabilities, and, on the other hand, the need for medical professionals to make the final judgement on the understandability and acceptability of treatment plans. This process of understanding and acceptance is not disconnected from the AI system’s decision-making. Through this interaction process, medical professionals’ information and knowledge have been expanded, and their diagnostic decision-making assumptions have been calibrated. Therefore, the auxiliary position of AI ensures the presence of medical professionals while avoiding rigid adherence to their final decision-making authority. Instead, it calls for context-specific configurations of responsibility, calibrated to the characteristics of human–machine interaction.

3.2. Establishing a relational responsibility system

Furthermore, treating the human–machine association as the basic decision-making unit does not mean that the association as a whole should bear responsibility. Otherwise, it risks repeating the pitfalls of schemes like “establishing independent compensation funds,” which serve merely to satisfy financial compensation requirements without improving practice. Simultaneously, the framework must avoid atomised responsibility arrangements that fragment the actions of various actors and their causal contributions to final outcomes.

Relational responsibility offers a third approach to overcome these challenges. This analytical approach emphasises that while we commonly express responsibility as “being responsible for something,” its essence lies in “being responsible to others” (see Painter-Morland, Reference Painter-Morland2006, p. 93). A responsible actor is not an atomised entity but one embedded within relational contexts. Social facts emerge through the interactions of various agents and cannot be reduced to a simple aggregation of the causal forces attributed to each party. The relational responsibility system has two key implications: first, it determines the content of responsibility within the specific context of relational interaction, thereby transcending a mechanistic understanding of responsibility attribution. It treats the content of responsibility as a social construction grounded in particular values, rather than a natural fact. Second, it establishes relational regulation as the value principle of the responsibility system. The system therefore operates both retrospectively and prospectively. It is concerned not only with tracing causality, distinguishing right from wrong, and assigning blame, but also with improving the interactive process itself. Even compensation for harmful consequences is viewed as a component of relationship regulation. People do not merely seek redress for specific outcomes through the responsibility system. They focus on the expectations, understandings, and attitudes towards the other party reflected in the specific acts of assuming responsibility. Both the practice itself and the compensation for damages should facilitate the possibility of sustained interaction and cooperation among parties, rather than exacerbating confrontation and shifting blame. This shifts the value principle from first-order morality to second-order morality, and the justification for assuming responsibility from outcome-based standards of right and wrong to process-based standards that assess whether actions create space for future interaction. As social networks become increasingly complex, individual actors exercise less control over the specific outcomes generated by interactive processes. Rather than persisting in the pursuit of linear causality, we should focus on the relational interaction process itself, as this is an effective strategy for addressing the emergent nature of outcomes and the ambiguity of fault. This approach helps stimulate actors’ potential to respond to risks through interactive practice (see Gergen, Reference Gergen2011).

In the specific application scenario of clinical decision-making AI systems, the emphasis on relational responsibility implies acknowledging that specific outcomes result from the interplay between the technology provider’s development and production, the system’s operational mechanisms, and the usage practices of healthcare institutions and clinicians. Nevertheless, it still holds the technology provider and users accountable as concrete responsible parties, requiring them to exercise their agency. In joint decision-making scenarios, the causal contributions of each party may not be clearly delineated. However, society cannot function through mutual shirking of responsibility and indifference. Such an approach neither addresses the pressing reality of scarce healthcare resources nor enhances medical standards through beneficial human–machine interaction, thereby safeguarding patient interests. Therefore, the central task is defining the desired mode of interaction and advancing its realisation through a framework of responsibility. Guided by relational responsibility theory, responsibility is oriented towards the relational patterns people expect, with the institutional goal of regulating how and with what attitude relational actors treat each other. The specific content of responsibility must be determined in light of particular human–machine interaction relationships and contexts, rather than being an explicit, static, object-centred standard, as exemplified by product defects. Moreover, it is precisely people’s focus on the specific meanings and attitudes underlying relationships and actions that indicates the insufficiency of current AI systems, which lack specific volition, attitudes, and emotional response, and by extension, the insufficiency of the human–machine association itself to bear the responsibility people expect. Only by directly addressing active agents such as technology providers and users can the system achieve regulation of the relationship itself.

This conception should also be distinguished from two established relational models in law. Fiduciary law begins from entrusted discretionary power, vulnerability, and confidence, and responds principally through duties of loyalty and remedies for betrayal of the beneficiary’s trust. Its bilateral and asymmetrical structure illuminates the physician–patient relationship, but it does not by itself organise the polycentric interaction among developers, institutions, practitioners and patients across an AI lifecycle (see Unver, Reference Unver2023, pp. 198–200). Relational contract theory, by contrast, correctly treats exchange as embedded in an ongoing social relationship sustained by cooperation, adjustment, and norms that exceed a discrete promise (see Feinman, Reference Feinman2000). Yet its primary field remains the continuing exchange relation between contracting parties. The relational responsibility proposed here borrows the fiduciary concern with vulnerability and the relational-contract concern with continuing cooperation, but extends beyond both: it includes non-contracting patients and institutional actors, makes the human–machine association the analytical unit, and evaluates responsibility by whether lifecycle practices construct, maintain, and repair relationships necessary for clinical decision-making. Its distinct contribution is therefore not the claim that every AI relationship is fiduciary or contractual, but the use of a broader, future-oriented responsibility principle to coordinate the legal duties that arise from several different sources.

In summary, the preceding analysis treats the human–machine association as the fundamental unit of clinical decision-making, adopting a spectrum-based understanding of how AI systems for clinical decision-making operate. It then embeds decision stakeholders within their interactive networks and imposes relational responsibility. This two-step approach offers three advantages and provides direction for constructing concrete responsibility arrangements: first, treating the decision association as the basic unit helps prevent responsibility shifting. Neither the inherent complexity and black-box nature of the technology nor the healthcare provider’s duty of independent review can justify mutual shifting of responsibility between technology providers and users. This approach avoids perpetuating the attribution logic of traditional product liability systems while addressing a core concern in existing research: including all decision-involved parties within the scope of responsibility. Second, a spectrum-based account of the depth of human–machine decision integration refines the interpretation of auxiliary role, transcending the master–servant relationship analytical framework. This avoids framing technological advancement and medical autonomy as opposing forces in a zero-sum dynamic, instead promoting their mutual empowerment and coordination. Third, the specific interaction model adopted within the decision-making association relates not only to the development of AI systems and their practical implementation tailored to clinical needs, but also to society’s overall expectations for human–machine interaction and the judgements and decisions made by clinicians in specific clinical practices. Therefore, the design of a responsibility system for human–machine association extends beyond compensation for harmful consequences and should encompass the practical process itself. This also represents a breakthrough from both technocentrism and anthropocentrism, thereby neither reducing real-world dilemmas to mere technical advancement nor viewing technology solely as a tool manipulated by humans. Rather than focusing on the static attributes of AI systems themselves, this framework prioritises the interaction between technology and society, thereby constructing an appropriate responsibility system grounded in specific interactive contexts and genuine human needs (see Li, Reference Li2016, p. 44).

4.1. From atomic responsibility allocation to trust relationship building

As previously noted, the traditional principles, which emphasise clear linear causation and atomised responsibility, are inadequate for regulating human–machine interaction, and may encourage responsibility shifting. Overcoming this bottleneck requires attention to the reality of human–machine interaction and a shift in analytical perspective from individuality to relationality. From a relational perspective, the arrangement of duty of care can be reinterpreted as determining the scope of mutual trust among relational participants. Specifically, the duty of care imposed on one responsible party should correspond to the degree to which it can trust other responsible parties, that is, the extent to which it need not exert additional vigilance. It is precisely upon this foundation of trust that all parties act, and collectively form the final decision.

It is worth clarifying the meaning of trust to demonstrate the value of trust relationships perspective. As Simmel noted, trust is one of the most important synthetic forces in society (see Simmel, Reference Simmel2002, p. 251), which is a mixture of knowing and not knowing, meaning that even when the parties are aware of the lack of complete control over trustees’ actions, they are still willing to assume that trustees will act in a manner beneficial to them (see Dai and Wang, Reference Dai and Wang2025, p. 40). Thus, a trust relationship implies accepting the existence of information asymmetry, opacity, and uncertainty while maintaining rationality as the basis for action under these conditions. Trust does rest on some degree of assessment, but it does not depend entirely on epistemological factors; that is, it is not founded on complete controllability or explicability. A leap of faith is inevitably required between acquiring knowability and explicability and taking cooperative action. With trust relationships as its normative goal, legitimacy for actions undertaken by all parties derives not from explicit causality or individual fault. Rather, it lies in the capacity of such actions to promote a constructive trust relationship oriented towards the practical process itself and future development. The construction of responsibility systems does not presuppose transparency and controllability and is not aimed solely at compensating for harmful consequences. Rather, it seeks to advance the appropriate interactive process itself, enabling technology providers, users and patients to engage in diagnostic and therapeutic activities with mutual trust despite the realities of algorithmic black boxes and medical risks. Technological maturity, the establishment of clear standards, and the assumption of liability for damages are essential elements and the rational basis for building trust. Therefore, the preceding shift in the legitimacy basis of responsibility does not displace requirements of traditional liability systems. However, considering the practical context of AI applications in clinical decision-making, responsibility frameworks should not merely regulate actions with certainty. They must also ease value tensions within conditions of uncertainty, thereby creating space for practice to explore the optimal blend of knowing and not knowing.

By contrast, the weakness of existing frameworks lies precisely in their failure to ground the system in this hybrid of knowing and not knowing. When prior research treats algorithmic black boxes as grounds for exempting medical providers from liability, it accordingly pursues the explainability of AI systems. Such explanations primarily manifest in three modes: first, contrastive explanation, justifying why the system produced a specific outcome rather than others, and demonstrating why the generated result is more reasonable; second, functional explanations, showing what output the system derives from a given input to perform a specific function; third, transparency explanations, disclosing the system’s internal structure and operational mechanisms. These explanatory approaches reflect efforts to address algorithmic black boxes. For example, functional explanations can be provided even without understanding the decision logic. However, these approaches all focus on assessing the correctness of decision outcomes. They assume users will become willing to use the system based on such outcome judgements. The problem lies in users’ limited ability to assess outcome accuracy. Such post-hoc explanations are also easily manipulated as a rhetorical tool. For instance, comparison items can be manipulated to create the illusion of credible results. In contrast, the advantage of building responsibility systems around trust relationships has two dimensions. First, the trust relationship framework aligns with the current reality of diverse, intertwined actors and algorithmic opacity, advancing technological practice on this basis. It does not advocate eliminating the inherent knowledge and information gaps between system users and developers, or between medical professionals and patients. Instead, it seeks to guide cooperative directions within these disparities, creating space for technological application. Second, establishing trust relationships concerns not only the rationality of decision outcomes and liability after harm occurs, but also the interactive process from system production to use. This is particularly significant for addressing the development and practical challenges of AI technology (see Bao and Zeng, Reference Bao and Zeng2024).

As previously noted, the responsibility framework here encompasses both physician–patient trust and algorithmic trust in clinical decision-making AI systems. The former involves external responsibility towards patients, fostering collaboration to advance treatment plans. The latter constitutes internal responsibility arrangements between technology providers and users through human–machine interaction, enabling the human–machine alliance to deliver effective clinical decisions. This distinction does not imply separate operational processes for the two trust relationships. Rather, they interact and reinforce one another in practice. For example, only when trust exists between physicians and patients can physicians access more effective patient information and specific treatment needs, thereby enhancing the responsiveness of human–machine interactive decisions and consequently strengthening algorithmic trust. Simultaneously, only by establishing algorithmic trust can healthcare professionals provide clearer and more persuasive explanations of clinical decisions to patients, which in turn helps deepen physician–patient trust.

In practice, diverse foundations for trust have emerged. Specifically, trust may rest on the following bases, ranging from personalised experiences to macro-level social structures, and from particular to universal contexts: (1) based on understanding of others and calculation of interests, such as determining through assessing the trustees’ motivation to honour commitments and safeguard the parties’ interests; (2) based on specific interactive relationships, such as gradually entrusting one’s interests to another through long-term engagement; (3) based on others’ group identity, social roles, and position within social networks, such as trust arising from shared ethnicity or professional affiliation; (4) based on institutional arrangements, which provide the foundation for individuals to advance relationships and make decisions when social complexity makes full interaction difficult yet decision-making is necessary; (5) based on behavioural norms, such as the belief in a universally accepted code of mutual trust within society. Importantly, these forms of trust are not grounded solely in rational analysis but are also influenced by emotions. For example, trust developed through long-term interactions is both a rational judgement derived from inductive knowledge and a leap of faith ultimately realised through emotional impetus. The trustee does not decide to honour commitments based solely on rational calculation. If trust is betrayed, the trustees feel they have violated mutual expectations for the relationship, thereby damaging their self-identity. Moreover, specific trust and general trust are not interchangeable. Institutional arrangements, for instance, cannot replace the specific trust formed through concrete interactions. Some scholars even contend that relationships built on institutional arrangements—relying on the clarity of rules and predictable consequences—differ from trust relationships that exist in the grey area between knowing and not knowing. Therefore, institutions alone cannot serve as the driving mechanism for trust relationships.

For physician–patient and algorithmic trust, the respective foundations can be found in the preceding account. Trust between physicians and patients may rest on: (1) patients’ understanding of physicians’ professional competence; (2) long-term interactions between physicians and patients during the diagnosis and treatment process; (3) physicians’ tendency towards responsible medical practice due to their professional identity, manifesting both as the constraining effect of professional ethics as role norms and as a reflective self-identity possessed by physicians; (4) institutional norms governing the physician–patient relationship and a prevailing social climate of trust between physicians and patients. Algorithmic trust, meanwhile, may rest on the following factors: (1) people’s understanding of artificial intelligence systems, specifically the system’s level of explainability; (2) the accumulation of long-term human–machine interaction experiences; (3) Relevant institutional regulations and a socially prevalent environment of trust in algorithms. Overall, algorithmic trust primarily rests on functional attributes and inductive judgements formed through prolonged interaction, whereas physician–patient trust involves emotional factors beyond rationality and hinges on holistic assessments of the practitioner’s benevolent intent. This emphasis on emotion and intent is crucial in motivating individuals to engage in risk-taking activities.

Because of the distinctive features of AI applications in clinical decision-making, three points warrant particular attention: first, we should acknowledge the reality of algorithmic black boxes. While high accuracy in long-term human–machine interactions can enhance trust, such trust is fragile and prone to collapse when biases emerge. Moreover, the critical nature of medical issues means they cannot be evaluated solely by the probability of algorithmic accuracy. Therefore, the building of algorithmic trust must transcend a model that relies solely on evaluating the technology itself. Second, regarding physician–patient trust, patients place trust in physicians not only based on specific interactions but also on physicians’ inherent benevolent intent associated with the medical profession. Accordingly, healthcare professionals derive a sense of self-identity from their role as physicians, engage in self-reflection and develop emotional attitudes based on whether they have earned and maintained this trust. The inherent emotions and attitudes reflect the distinctiveness of human-to-human interaction compared to human–machine interaction. In the absence of knowledge certainty, this is particularly significant for patients’ willingness to trust and cooperate with medical care. Therefore, in arranging responsibilities for medical institutions and practitioners, it remains essential to reinforce this unique foundation of physician–patient trust, rather than advocating for the withdrawal of physicians solely on the grounds of technological advancement. Moreover, the reflexivity and emotional elements inherent in interpersonal trust can compensate for the limitations of algorithmic trust, which is primarily based on cognition. This, in turn, supports technological development by enabling healthcare professionals to mediate and foster patient trust in the application of artificial intelligence systems within medical processes. Third, the establishment of a responsibility system as a macro-level institutional arrangement should not merely aim to regulate the behaviour of responsible parties from the top down, providing rule-based certainty for legal consequences. Rather, it should facilitate spontaneous interaction among all parties, fostering flexible trust relationships responsive to real-world contexts. This aligns with the agency inherent in the relational subjects discussed earlier and the emergent nature of human–machine collaborative decision-making.

4.2. External responsibility to foster physician–patient trust

With respect to external responsibility, it is particularly crucial to leverage AI systems to support trust-building between physicians and patients, and avoid the emergence of new forms of medical paternalism due to technological integration (see Huang and Tan, Reference Huang and Tan2024, pp. 51–52). This responsibility system has two dimensions: first, the dialogic responsibility of medical providers during the diagnostic and treatment decision-making process; second, the primary responsibility of medical providers as the accountability hub when harm occurs.

First is the issue of responsibility allocation in the clinical decision-making process. It can be observed that the introduction of relational responsibility has updated and deepened the meaning of informed consent, which gives practical effect to Article 1219 of China’s Civil Code.Footnote ² Specifically, prior to making a clinical decision, healthcare institutions should explicitly explain to patients the specific interaction model between AI and medical personnel during AI-assisted diagnosis and treatment, as well as the associated benefits and risks. To be concrete, the validity of informed consent under AI opacity cannot depend on explaining source code and every parameter. Such a requirement would overwhelm patients without necessarily improving autonomy. Nor, however, should the difficulty of technical explanation justify silence. A relational approach calls for layered, continuing communication. At the baseline, the physician should disclose that AI materially participates in the recommendation, the function it performs, the degree of human review, known material limitations or population mismatch, reasonably available non-AI alternatives, and the consequences of accepting or declining its use. Communication intensity should increase where the system is unusually opaque and novel, where the clinician cannot independently verify a material output, or where the patient asks about AI involvement. This empowers patients to autonomously decide whether to incorporate AI-assisted diagnosis and treatment. During the treatment process, if a patient consents to using a clinical decision-making AI system, the system’s deep learning capabilities and algorithmic black-box nature still do not constitute grounds for exempting medical personnel from liability. Medical professionals should adhere to the principles of relational responsibility, engaging in dialogue with patients about treatment decisions. This involves understanding the patient’s expectations regarding treatment goals and concerns about the proposed plan, and providing further clarification based on this understanding.Footnote ³ Implementation of dialogic responsibility offers the following advantages: first, dialogic responsibility facilitates healthcare providers’ reflection on their own decision-making processes, which prevents them from blindly trusting AI conclusions based solely on historical accuracy, thereby reducing bias. Second, strengthening doctor–patient dialogue aligns with the practical demands arising from the increasing prevalence of chronic diseases. In such cases, curing the disease is not—and often cannot be—the ultimate goal of treatment. Patient expectations for treatment are diversifying—some may prioritise pain reduction over cure, while others seek to maintain functional abilities like eating or mobility. Treatment plans cannot be predetermined without patient input. Rather, they must emerge from collaborative, iterative dialogue to identify and reflect patients’ value priorities in clinical decisions. Such dialogue also facilitates feedback to AI systems, preventing them from generating treatment plans that contradict patients’ actual wishes. Third, direct doctor–patient dialogue strengthens trust between them. While AI systems may exhibit high decision accuracy in statistical and probabilistic terms, in medical contexts, each patient’s rights are paramount. Therefore, trust cannot be built solely through probability. As previously noted, the mechanisms for building interpersonal trust and algorithmic trust differ. Concrete physician–patient communication can stimulate human reflexivity and emotional elements. This is crucial for trust-building and aligns with the emerging medical ethics concept of narrative medicine. Narrative medicine aims to enhance patient engagement in their illness, providing opportunities to articulate the broader social contexts underlying their treatment needs. This encourages physicians to listen to and understand patients’ stories, thereby transcending the limitations of the biomedical model, which focuses solely on combating disease while neglecting the patient as a whole (see Jiang and Lin, Reference Jiang and Lin2024). Overall, the introduction of dialogic responsibility implies that informed consent is not a one-way information dump but an iterative relationship-building practice. The physician–patient relationship can no longer be conceived in unidirectional term, and thus cannot be reduced to an either/or choice between physician-based and patient-based disclosure standards. Instead, shared decision-making can be achieved through physician–patient dialogue.

One possible objection to this dialogic responsibility is that in traditional medical practice, healthcare providers are not obligated to disclose to patients the literature or case studies referenced in their decision-making, nor to reveal their reasoning process. Given this, why should healthcare providers be obligated to explain their specific interaction patterns with decision-making AI systems to patients (see Cohen, Reference Cohen2020, p. 1442)? This article argues that the process of a physician conducting literature searches based on their preliminary analysis of clinical symptoms differs from their direct interaction with AI systems. The former involves a thinking framework dominated by the clinician, while the latter represents an interaction between two thinking frameworks, making clinicians more susceptible to anchoring effects. This influence is often difficult to avoid and can be mitigated only through the preceding reflective dialogue. Therefore, although both the healthcare professionals’ literature search and the application of an AI system serve an auxiliary role, the nature and operation of assistance remain fundamentally different. In scenarios where AI systems are used to support decision-making, this dialogic responsibility plays a crucial role in strengthening both the rational and emotional foundations of trust between physician and patient.

The second issue concerns liability allocation when medical decisions result in harm. As previously discussed, the human–machine association is the fundamental unit for clinical decision-making. However, its liability is limited to monetary compensation. To treat the human–machine association alone as the liable party is to reduce the responsibility system to a punitive instrument, obscuring its more fundamental role as a vehicle for cultivating relational trust and fostering constructive engagement among the parties involved. Therefore, this article advocates a two-step approach to implementing liability for harm: first, theoretically establish the human–machine association as the fundamental unit for decision-making liability. In this case, patients need not prove the factual causation and degree of fault of individual technology providers, sellers, medical institutions, and healthcare personnel to seek separate compensation. Instead, they only need to demonstrate a causal link between the final treatment decision made by the human–machine association and the resulting harm. They must also demonstrate that this final decision was negligent, considering the prevailing medical standards and the patient’s specific condition as reflected in the history of physician–patient interactions. Second, regarding the specific entity liable for compensation, the medical institution serves as the pivotal hub within this system, centrally assuming external liability. Subsequently, the medical institution may internally allocate responsibility among other liable parties based on specific human–machine interaction. The internal allocation of liability will be detailed later and is not elaborated upon here.

The rationale for this two-step approach rests on two considerations: first, treating the human–machine association as the theoretically liable entity aligns with the emergent nature of interactive decision-making, which cannot be reduced to the individual actions of any single participant. Moreover, even if medical practitioners assume dialogic responsibility during the diagnostic decision-making process by explaining their interaction patterns with AI systems to patients, patients still struggle to comprehend the actual internal mechanisms of human–machine interaction. Requiring patients to pursue compensation based on relational characteristics among various entities is impractical. Designating the human–machine association as the theoretically liable entity substantially reduces the burden of proof for patients. Second, designating the medical institution as the primary recourse entity not only facilitates patient compensation but also serves as an institutional solution for building trust between patients and healthcare providers, as well as trust in algorithms. This creates a consistent framework for physician presence, encompassing prudent reflection on treatment decisions during active dialogue and taking the lead in assuming responsibility after treatment-related harm. This approach adheres to the human-in-the-loop model of human–machine interaction, ensuring that while AI systems permeate human decision-making, human judgements retain primacy in specific decisions. The legitimacy of healthcare institutions assuming centralised liability in the first instance stems neither from institutional fault nor punitive intent, but rather from fostering trust-building through this system and empowering algorithmic trust through physician–patient trust. This mechanism is also supported by current legislation. According to Article 1223 of China’s Civil Code, when a patient suffers harm due to a defective medical device, the patient may first seek compensation from the medical institution. After paying compensation, the medical institution may then seek reimbursement from the responsible manufacturer or other liable parties. Admittedly, this provision of the Civil Code is grounded in a clear dichotomy between medical device defects and medical malpractice. Yet even in cases where fault can be distinctly categorised, the law still supports patients requesting medical institutions to bear responsibility first. By extension, imposing such liability requirements on medical institutions in interactive decision-making scenarios is justified, reflecting a consistent consideration for safeguarding patient rights and building trust between physicians and patients.

Naturally, the preceding mechanism does not preclude medical institutions from establishing dedicated funds for compensation. In the longer term, a specialised compensation fund or pooled coverage may address diffuse, latent, or scientifically indeterminate injuries. However, it should be clarified that although insurance and funds are valuable simplification and loss-spreading devices, they remain relationally legitimate only when designed to return information and incentives to the actors capable of improving future interaction, rather than purchasing an exit from responsibility. Payment should trigger, rather than erase further investigation and internal accountability.

In summary, the medical provider’s fulfilment of dialogic responsibility and its primary assumption of liability for damages encourage careful review of treatment decisions. This approach also enhances the acceptance of medical technologies through strengthened trust between physicians and patients, thereby balancing the relationship among technological innovation, medical autonomy, and patient rights.

4.3. Internal responsibility arrangements to foster algorithm trust

A further question is how AI systems and their providers should interact with healthcare institutions and practitioners as users to foster algorithmic trust and achieve mutual empowerment between technological innovation and medical practice? In scenarios involving algorithmic black boxes, algorithmic trust cannot be built solely on the transparency and interpretability of AI systems. Instead, it requires focusing on the interaction process itself between technology providers and users. Accordingly, the internal allocation of specific liability for damages resulting from decisions must align with both parties’ expectations regarding the scope of mutual trust.

As previously noted, the human–machine association is not monolithic. Its modes form a spectrum ranging from near-agency relationships to deep interactive partnerships, dynamically evolving with technological advancement. This inherently implies varying degrees of trustworthiness. Algorithmic trust within the team should therefore be understood as calibrated reliance by clinicians, not as general confidence in technology. Regarding the interactive process itself, the design intent of technology providers, system capabilities and risks, alongside practitioners’ operational needs and concerns, can only be mutually clarified and aligned through dialogue. Therefore, equal emphasis should be placed on the dialogic responsibility between technology providers and users. Specifically, technology providers should articulate the underlying value norms, functionalities, and risks of the technology, the efforts made during development to address challenges, and how the technology’s clinical decision-making aligns with or extends beyond healthcare practitioners’ decision-making reasoning. This provides guidance on how practitioners should apply and interact with the system. Healthcare professionals, as users, should articulate their specific clinical practices for the target condition, particularly the challenges and needs encountered during practice. This enables verification of how well the technology provider’s solution aligns with actual requirements. When individual clinicians decide to adopt or override the AI recommendations, the decision should be well-reasoned and logged. These logs, however, must be used for further learning rather than for automatic punishment that encourages concealment. This dialogue process must be iterative throughout subsequent practical use to drive technological advancement and evolve human–machine interaction models. For instance, if discrepancies exist between the clarity of medical knowledge and the accuracy of decision-making AI systems (e.g., AI failing to accurately diagnose conditions with well-defined clinical standards), designers should provide explanations. This helps medical practitioners understand obstacles in translating medical knowledge into algorithms while enabling technology providers to grasp clinical practice needs and clarify trust levels in system decisions during human–machine interactions. This dialogic responsibility fosters shared objectives between technology providers and users, engaging users and conveying providers’ concern for their needs. As previously noted, this constitutes a critical element in building trust.

It follows that interface design is also part of the dialogic responsibility arrangement: the interface should disclose the system’s validated task, confidence and data limits, present clinically meaningful alternatives or counterfactuals, permit efficient correction and override, and record the reasons for adoption, rejection, and escalation. Explanations that are technically rich but poorly fitted to workflow may increase cognitive burden or become superficial badges of competence rather than triggers for reflection (see Naiseh et al., Reference Naiseh, Al-Thani, Jiang and Ali2023). The logic of human collaboration with coding agents provides a useful but limited analogy. Reliable teams do not equate the human’s final click with meaningful control. Instead, they decompose work into inspectable intermediate outputs, preserve version histories, permit rollback, and require heightened review before consequential deployment. Clinical AI systems should similarly expose intermediate reasoning artefacts appropriate to the task, make overrides reversible where possible, and place explicit review gates before high-risk recommendations become orders. The analogy has limits because clinical decisions concern bodily integrity, time pressure, and patient values rather than software artefacts. Precisely for that reason, the human approval point must represent substantive, supported review rather than a formal transfer of blame to the physician.

The dialogic responsibility differs from the warning duty under traditional product liability. The warning duty centres on the product itself, relying on defined design and manufacturing standards alongside foreseeable risks. It manifests as a non-interactive, one-way statement aimed at ensuring users operate within the product’s intended functionality. Furthermore, proper fulfilment of the warning duty can constitute grounds for reducing or eliminating the manufacturer’s liability. The dialogic responsibility, however, does not necessarily presuppose foreseeable risks or fixed usage guidelines. The appropriate manner of human–machine interaction is clarified and continuously adjusted through dialogue between technology providers and users, based on users’ needs and technical feasibility. This implies that vague usage instructions provided by technology providers, such as “system-generated decisions are for reference only,” cannot be considered adequate fulfilment of dialogic responsibility, and cannot negate specific commitments about validated performance or institutional directions that encourage reliance.Footnote ⁴ Furthermore, the dialogic responsibility differs from the interpretability of AI systems. The latter focuses on knowledge-based explanations of technology, building algorithmic trust by demonstrating the accuracy of decision outcomes. The former, however, centres on the proactive, responsible attitude manifested in the dialogue process itself and its corresponding role in constructing trust relationships. Here, trust stems not merely from users’ understanding of technology but from the collaborative, interactive relationship between technology providers and users. It is worth noting that the specific consensus reached regarding the level of trustworthiness can be formalised in the purchase and usage agreements signed between technology providers and users. Given that the explicit definition of trustworthiness levels directly impacts liability arrangements in the event of subsequent harm, both technology providers and users will have sufficient incentive to practice this dialogic responsibility. Overall, by reshaping organisational conditions, these measures implement relational responsibility with a focus on the interaction process itself, rather than merely waiting to identify a single blameworthy actor after harm.

Furthermore, post-damage liability arrangements should correspondingly revolve around the calibrated level of trust. It must be reiterated that in scenarios where the human–machine association serves as the fundamental decision-making unit, such liability arrangements are not based on the explicit attribution of fault or the blameworthiness of a specific party in causing harm. A party is not responsible merely because it appears somewhere in the causal chain, nor exempt merely because another party retained the formal “final decision.” Instead, the party that breaches the previously established trust baseline, that is, that fails to perform the dialogue, monitoring, documentation, updating, or reflective verification reasonably expected of it, should bear responsibility for achieving relationship adjustment. Generally, if an AI system employs rule-based algorithms to mimic healthcare professionals’ rule-based diagnostic processes, it implies the system possesses clear diagnostic criteria. Compared to humans susceptible to irrelevant factors and cognitive biases, such a system should inherently offer more precise diagnostic capabilities, thereby providing a stronger rational foundation for algorithmic trust. In such cases, medical professionals have greater justification for freeing up attention. Clinicians should principally check eligibility, data quality, and exceptional contraindications. Reliance within the validated envelope is more justified, so an unexplained failure within that envelope weighs more heavily in internal recourse against the AI system provider. Conversely, if the system aligns with medical professionals’ heuristic diagnostic approaches through deep learning modes, the system’s diagnostic decisions may either exacerbate the anchoring effect experienced by medical professionals or compensate for their oversights, offering valuable insights into treatment directions. In these instances, medical professionals should engage in more thorough deliberation and reflection on the diagnostic outcomes. Evaluation should focus on the quality of iterative questioning, independent clinical synthesis, preservation of the interaction record, disclosure of model and data limits, and the use of feedback from observed failures to inform system updating. In sum, as warranted reliance and machine autonomy increase, provider commitments and traceability should intensify. As uncertainty and interaction deepen, clinicians’ reflective and dialogic responsibilities should intensify. This mechanism helps avoid both under-trust and over-trust. It is worth noting that the more controllable an AI system is, the stronger its tool-like attributes become, aligning more closely with the regulatory scope of traditional product liability frameworks. The greater the degree of independent decision-making and trust bestowed upon the technology, the greater the human dependence on it. Human cognitive frameworks may consequently become dominated by technology, with human–machine interaction increasingly resembling a principal-agent model. Conversely, when AI systems for clinical decision-making lack controllability due to their deep learning characteristics, human trust diminishes, and reflective verification of their decisions increases. In fact, this further reveals the limitations of subject–object dichotomy in understanding human–technology relations. Anthropocentrism and technocentrism often coexist. The more emphasis is placed on human control over technology, the greater the dependence on technology becomes, paradoxically subverting the subject–object relationship. The power dynamics between humans and technology are not characterised by mutual exclusion, but rather by a tense equilibrium that transcends unidirectional domination.

Thus, the central task is determining the consensus reached between technology providers and users regarding the trustworthiness of algorithms. The practice of the dialogic responsibility and related contractual agreements serve as crucial benchmarks. For example, consider a scenario where prior dialogues and agreements established that an AI system operates on rule-based algorithms and delivers outcomes consistent with current medical standards, yet the final diagnosis proves erroneous. In such a case, after assuming primary responsibility towards the patient, the medical institution may seek recourse from the technology provider. A potential concern arises: could technology providers circumvent liability by incorporating system alerts requiring medical personnel to independently review decisions? As previously established, the dialogic responsibility is not a vague, blanket duty to warn. The technology provider must clearly specify the achievable accuracy level, identify which aspects require clinicians’ verification, and clarify which aspects can be fully trusted. Otherwise, technology would not only fail to enhance clinical efficiency but become an additional burden. If the technology provider cannot make any commitment regarding freeing up clinicians’ attention, clinicians would have no incentive to adopt such systems. Therefore, to achieve widespread adoption, technology providers naturally have an incentive to continuously refine usage guidelines and engage in communication with healthcare providers, thereby effectively addressing the preceding concerns.

Similarly, when employing deep learning-based systems, the generation of incomprehensible decisions does not equate to erroneous decisions violating medical standards. Rather, it represents an inherent probabilistic nature of the deep learning process, often accompanied by the system’s ability to provide insightful guidance. This should not be regarded as a system flaw or a fault of the designers. In these circumstances, the scope of medical professionals’ trust in AI systems should be bounded by the understandability of system decisions. Otherwise, reliance on system decisions would lose its fundamental rational basis and could no longer be termed trust. Therefore, if medical personnel, after reflecting on and verifying the diagnosis and treatment outcomes by integrating existing medical standards, their own clinical experience, the rationale and insights from the system’s decisions, and the patient’s actual condition, still deem the system’s recommendations incomprehensible and choose not to adopt its conclusions, this does not constitute negligence on their part—even if future medical advancements validate the system’s conclusions. Conversely, if medical personnel knowingly adopt the system’s incomprehensible diagnostic or treatment decisions and cause harm, this constitutes negligence. In such a case, the medical institution and the healthcare provider shall bear liability for damages. It must be emphasised that, given that the vast databases of AI systems help medical personnel overcome temporal and spatial constraints to access medical information, their judgement on whether the system’s decisions have a rational basis should be grounded in verification and reflection on the information provided by the system, rather than in their existing medical knowledge and experience alone.

Notably, regarding legal attributes of the dialogic responsibility in both the physician–patient relationship and the AI provider-user relationship, it should be clarified that a breach of dialogic responsibility does not automatically create a new, independent tort. Future statutes or binding standards could turn specified dialogue practices into direct duties of care. Until then, “relational responsibility” (and dialogic responsibility as its manifestation) is best understood as a principle for interpreting and coordinating existing tort, contract, professional, and regulatory duties. In the patient-facing relationship, it may serve as evidence that the institution and clinicians failed to meet the diagnostic and treatment duties under Article 1219 and Article 1221. In the provider–user relationship, failure to fulfil the dialogic responsibility may constitute breach of documented agreements and the good-faith duties of notification and assistance under Article 509, and such a breach may serve as a basis for internal recourse. In short, the introduction of relational responsibility aims to advance the relational interpretation of the law, thereby maximising its vitality. Relational responsibility thus does not dispense with positive law. Rather, it connects otherwise fragmented duties to the interaction that gives them practical meaning. It is also necessary to address a discussion in existing research: whether the development of clinical decision-making AI systems should incorporate medical personnel errors as a risk factor in system design. Existing research suggests that system developers should be able to anticipate the possibility of general negligence by medical personnel, but cannot be required to factor in gross negligence or even intentional misconduct (such as inaction in response to counter-intuitive machine judgements), as gross negligence constitutes an alternative cause that breaks the causal chain. This allocation of responsibility between humans and machines has two drawbacks: first, from a technical design perspective, the more a harmful outcome reflects gross negligence or even intentional misconduct by medical personnel, the more it indicates that the decision contradicts established medical knowledge and standards. Accordingly, this also implies that the decision-making process likely falls within an area where the AI system should have been capable of learning and providing accurate diagnostic conclusions. Conversely, matters that healthcare providers are prone to overlook due to ordinary negligence during re-evaluation may actually represent challenging aspects in algorithm design. This suggests that under the existing liability allocation standard in research, technical providers could be exempted from responsibility on the grounds that “the healthcare provider’s gross negligence constitutes a substitute cause” precisely where the system likely harbours design flaws. This approach hinders the full realisation of the system’s inherent decision-making capabilities, discourages technological refinement, and impedes improvements in accuracy. Second, discussions on whether “medical actions constitute a superseding cause” still implicitly follow the logic of linear causation and atomised attribution. This approach aligns with traditional product liability frameworks, limiting the provider’s responsibility to foreseeable risks. However, traditional product liability could adopt this standard because a product’s specific functions and production pathways are predetermined, providing clear guidance on subsequent usage and associated risks. If users act in ways exceeding expectations, this naturally constitutes an interruption of linear causation, thereby relieving producers of liability. However, in the context of AI applications for clinical decision-making, the relationship between system judgements and healthcare professionals’ reviews is one of mutual reinforcement. Given this characteristic, the decision-making capacity of all parties—their control over outcomes—should be fully leveraged and facilitated through comprehensive interaction, aiming to achieve optimal clinical outcomes through collaborative efforts.

By contrast, the proposed framework offers two key advantages: first, it distinguishes between external responsibility aimed at fostering trust between physicians and patients and internal responsibility focused on enhancing algorithmic trust. The complexity of internal arrangements does not affect patient compensation, thereby safeguarding patient rights. For example, in scenarios where machine judgements are clearly counter-intuitive yet go unnoticed by physicians, crises of both physician–patient trust and algorithmic trust coexist. Patients have reasonable grounds to believe medical personnel failed in their duty, while physicians may reasonably question whether the AI system delivers its promised accuracy. This proposal effectively prevents mutual interference between these crises. Medical providers cannot evade liability by citing system provider errors but must assume primary responsibility before seeking recourse from the system provider based on their relative roles. This approach incentivises responsible action from both medical providers and technology suppliers. Second, the degree of trustworthiness can be determined through dialogue between technology providers and users, naturally incorporating market flexibility into the institutional design. Agreements reached through the active engagement of all parties can be embedded in the responsibility determination process, helping to avoid rigid application of the liability system. Thus, technological development and healthcare professionals’ reflective verification become mutually reinforcing, providing feedback and refinement to each other.