Introduction
The rapid proliferation of GenAI has resulted in sophisticated technologies that are readily available and enable scammers to create convincing-looking content.Footnote 1 Deepfake scams are complex threats that target, deceive, and manipulate customers across industries through, for example, synthetic videos and imagery, while scammers exploit online social media platforms available to them and constantly adapt their methods to evade enforcement.Footnote 2 Scammers are increasingly using deepfake videos to impersonate celebrities and trusted professionals, raising growing concerns within the financial sector.Footnote 3 Such deepfakes can produce outputs that are not explicitly programmed and may contain inaccuracies, while also being deliberately created or used to cause distress to recipients.Footnote 4 A 2024 survey by Regula reveals that globally, half of all businesses have experienced fraud involving audio and video deepfakes.Footnote 5 Additionally, 66% of business leaders believe deepfakes pose a serious threat to their business; on average, businesses across industries have lost nearly $450,000 to deepfakes.Footnote 6 Deepfakes create multifaceted challenges for public authorities, including legal and ethical implications concerning consent, misleading content, and cybersecurity.Footnote 7
If deepfakes misrepresent crucial information when offering financial advice or support that is essential for the decision-making process, the effects on specific clients can be irreversible.Footnote 8 Deepfakes are working like a charm for scammers because some social media users simply do not know what GenAI is capable of when it comes to making convincing impersonation videos.Footnote 9 Some users watch deepfake videos and believe they are real because many do not have the information or knowledge to question them. As reported, ‘designers should pay far more attention to how ordinary people might use their services’.Footnote 10 Deepfakes are increasingly being used on video calls to impersonate senior staff at corporate organisations to persuade other staff members to process payments.Footnote 11 As the technology progresses, deepfakes have the potential to make romance scams even more convincing, and could be used to manipulate images of friends and family members making requests for money.Footnote 12 This practice creates a social problem and could threaten resilience assets such as networks, human relationships, individual education, and media trust.Footnote 13
Scammers are relentless and continuously evolve their tactics to try and escape detection. This pushes firms to develop new ways, including the use of facial recognition technology, to make it harder for fraudsters to deceive others.Footnote 14 There is a general assumption that business-to-consumer uses of digital platformsFootnote 15 can be sufficiently regulated through existing contracts, tort (eg fault-based liability based on negligence), and consumer protection laws.Footnote 16 Contractual mechanisms for limiting the scope of responsibility for professional advice are considered in terms of consumer protection policy.Footnote 17 Extant debates focus on holding deepfake content creators accountable.Footnote 18 There are, however, challenges with this. Copyright and data protection laws seem inadequate to impose liability on creators of deepfakes due to questions of anonymity and authorship, as well as difficulties in bringing an action for copyright infringement in relation to any underlying copyright works.Footnote 19 The question at stake is how copyright applies to the use of protected works in training generative AI models, particularly at the data collection and model training stages. Addressing this question requires legislative frameworks that strengthen licensing, transparency, and enforcement.Footnote 20 Investigations into who is behind the deepfakes are very costly and time-consuming.Footnote 21 Criminal penalties would not be suitable to deter deepfake content creators, as they require proof beyond reasonable doubt of the intention to deceive consumers.Footnote 22 Similarly, social media platforms release their own policies for removing or banning deepfakes, also using ‘intent’ as their barometer for deciding whether to remove a deepfake. However, defining ‘intent’ is highly subjective, since it is based on the assessment of individual actors.
At the EU level, the General Data Protection Regulation introduced a set of penalties for companies that fail to protect the data of citizens.Footnote 23 However, the European Artificial Intelligence Act (EU AI Act)Footnote 24 and Digital Services Act (DSA 2022)Footnote 25 show limitations in accommodating liability for AI-generated harms.Footnote 26 The EU legislation has not provided a clear definition of malicious deepfakes, making any responsibility of the deepfake supply chain meaningless.Footnote 27 Under the UK’s Online Safety Act (OSA 2023),Footnote 28 providers of relevant online services are required to establish proportionate systems and processes to swiftly identify and promptly remove illegal content when brought to their attention. Providers also need to assess and minimise the risks arising from algorithmic or recommender systems, which could be exploited to enable the proliferation of illegal content. Ofcom’s codes also envisage testing and safety indicators for changes to recommender systems.Footnote 29 The OSA 2023 introduced offences prohibiting the creation, sharing, and threatening to share intimate images, including deepfakes. In the context of financial scams, the Act imposes a duty on digital platforms to ‘use proportionate measures relating to the design or operation of the service’Footnote 30 and ‘operate the service using proportionate systems and processes designed’Footnote 31 so that users cannot view fraudulent content and platforms remove such content as soon as possible. Nevertheless, the OSA 2023 does not impose direct liability on digital platforms to victims of fraudulent scams.
A shared liability regime for GenAI scams would target the deepfake AI supply chain. The supply chain involves multiple actors such as the creator of the video, the developer of the deepfake software used to fabricate the altered or distorted image, the online platforms that intermediate the dissemination of the video, and digital users who upload, view, and share the content. Supply chains are complex networks of interconnected third parties, cloud services, and IT systems, all of which may be vulnerable to cyberattacks targeting AI systems or exploiting datasets.Footnote 32 The creator of the video or perpetrator usually makes non-consensual deepfake materials that are claimed to be lawful and legitimate, while they are the result of ‘an unauthorised exploitation and reengineering of other people’s images and thus, their personal data’.Footnote 33 The software providers are responsible for writing the code, integrating third-party components, and ensuring that the development of software is immune from vulnerabilities and threats. Developers of software have the delicate task of implementing coding practices that ensure the security and integrity of the decision-making process.Footnote 34 Several episodes of machine learning supply chain compromise have resulted in undesirable outputs because a malicious actor accessed a supply chain and jeopardised the algorithm.Footnote 35 AI models are more exposed to manipulation, as is the case with the technique of feeding adversarial data into the training dataset of the model in a way that introduces biases or errors without triggering security mechanisms.Footnote 36 Poisoning AI models with malicious code within software components to control the results is a common method in deepfake scams where large databases are exploited to alter key automated procedures.Footnote 37
Such a liability regime would require AI software developers and online social media platforms to apply for a licence before the implementation of software applications. Regulatory sandboxes offer a suitable tool for testing AI-generated systems in a legally controlled environment while providing accuracy in AI recommendations and screening potential malicious deepfakes in financial products.Footnote 38 The Singapore courts have adopted a comprehensive binding Guide on GenAI use to verify the accuracy and appropriateness of all AI-generated outputs.Footnote 39 The Guide attributes full responsibility for the content of AI tools to court users; however, the question is: who should be liable for the losses from deepfake scams when the supply chain for the scam is highly complicated?
The UK Financial Conduct Authority’s (FCA) consumer duty rule, which established regulatory standards against firms’ unfair practices, can provide a solution to the use of GenAI tools in the context of financial scams.Footnote 40 The duty requires financial firms to evaluate their products, services, and processes with a view to protecting customers in business relationships and remedying foreseeable harms.Footnote 41 Its purpose is to set higher expectations for the standard of care that firms afford to customers while, in parallel, making firms responsible for addressing fraudulent practices that impede consumers from achieving reliable outcomes. However, the consumer duty rule is primarily a financial regulation principle and does not explicitly apply across all sectors. It might not be suitable to mitigate detrimental outputs generated by deepfake scams.
We argue that the shared liability regime for deepfakes would combine with the consumer duty rule as a regulatory conduct of business requirement for firms. This requires firms to assess the suitability of their financial advice services and monitor the results of AI-generated content that customers receive. This could require online social media platforms to share liability for losses with financial firms which, in turn, would lead to the development of a preventive enforcement mechanism for the consumer protection framework. We contend that a shared liability regime for malicious financial deepfakes among content creators, software developers, and online social media platforms would be effective as a form of deterrence and as a tool to hold these actors liable for fraudulent scams.
The originality of the proposed shared liability framework lies in the analysis of various actions and activities among multiple parties, which serves to identify potential scenarios where the apportionment of responsibilities involves each actor in the AI supply chain. Although the idea of shared liability has been shaped in various contexts according to the main doctrinal views, the proposed model advances existing debates on AI while providing critical reflections on the need for further regulatory intervention and redress policy options. However, the proposed framework reveals the difficulty of determining each actor’s share of liability for actions taken at every stage of the supply chain. Existing regulatory regimes do not allocate the loss when an AI system causes harm, and the degree of accountability for one’s behaviour is not equivalent to the attribution of responsibility, which mostly depends on context and the AI application used. Furthermore, the victim bears the onerous burden of proving which decision-making process in the supply chain led to the specific outcome, a particularly difficult task, especially in a system where procedures are not fully reliable or detectable.
The article proceeds as follows. We begin with an introduction to GenAI techniques, illustrating how they generate content and produce outputs that are not explicitly programmed. The second section examines the regulatory framework applicable to deepfakes in the UK, which is the focus of the article. We also discuss the regulatory frameworks for deepfakes in the EU and Singapore for comparison. In the UK, we pay particular attention to the OSA 2023 and its main effects on regulated services. The EU comparative analysis is useful because the main piece of legislation, the DSA 2022, adopts a systems-based approach in contrast to the content-based approach in the OSA 2023. The comparison with Singapore is instructive because its legislative framework establishes a lower threshold for scams and malicious cyber activities than that of the UK. The Singaporean government only needs to suspect that there is online harm to issue a direction, while for other activities, reasonable suspicion is required. This contrasts with the ‘reasonable grounds to infer’ threshold that the content amounts to an offence in section 192(5) of the OSA 2023 for all offences, including scams and cyber activities. Further, the Singaporean legislation provides scam victims with faster remedies than those available in the UK. The third section depicts the liability contours of deepfake financial scams, analysing the challenges in identifying the responsible party for GenAI fraud. This section addresses the attribution of liability to deepfake content creators and the interaction with online social media platforms: it investigates the applicable regulatory standard for holding fraudsters liable and for ensuring effective consumer protection through compensation schemes for scam losses.Footnote 42 Platform responsibility in investment scams is also considered along with potential challenges in imposing liability for losses suffered by scam victims. This liability attribution leads to the elaboration of a framework which establishes obligations on the various actors in the deepfake supply chain. The penultimate section advances a policy proposal for a shared liability regime to prevent multiple actors from disseminating malicious deepfakes in financial investments. We identify potential scenarios in which responsibility is allocated to content creators, software developers, and online social media platforms as the main actors involved in online fraudulent activities. In this context, we define content creators as the authors and creators of deepfake scams advertised as posts on regulated online social media platforms such as Facebook or Instagram, which host the deepfake content. Software developers are the individuals and businesses that create the deepfake software enabling deepfake scams. The final section provides concluding remarks.
The regulatory landscape for deepfakes in the UK
The OSA 2023 represents the most significant attempt to protect consumers by regulating harmful online content, including deepfakes. It regulates search services that allow users to post content online or to interact with each other. Regulated services must have ‘links’ with the UK, either because they have a ‘significant number’ of users or they view the UK as a target market or they can be accessed from the UK and there ‘are reasonable grounds to believe that there is a material risk of significant harm to individuals’ from their content.Footnote 43 While it creates new duties for internet service providers,Footnote 44 the OSA 2023 has several limitations when it comes to deepfakes. First, the Act focuses primarily on protection from illegal content and content harmful to children. Illegal content is defined in schedules 5-7, and the priority offences (which include fraud and financial services offences) are broadly defined in schedule 7. However, there are far more non-financial-related offences under schedule 7 such as pornography and sexual abuse.Footnote 45
Further, different types of harms call for different legal treatment. Where generative AI tools, such as deepfake voice cloning or synthetic identity creation, are intentionally used to defraud, harm arises from malicious user behaviour. Here, the actor(s) with fraudulent intent to deceive and harm others should be liable, while platforms and service providers face obligations under the OSA 2023 to implement proportionate systems to detect, mitigate, and remove such illegal content. Regulators focus on content moderation and criminal enforcement here.
Meanwhile, unintended harms from faulty AI outputs, such as incorrect information generated through hallucinations, involve questions of product reliability, safety, and duty of care.Footnote 46 The relevant legal areas are negligence and product liability. Product developers need to ensure model accuracy, explainability, and user safeguards. The regulatory concerns here are algorithmic accountability and transparency. Thus, while both intended and unintended harms involve AI-generated content, the legal treatment is different. Intended harms involve monitoring intentional misuse by users, whilst unintended harms concern rectifying design defects in AI outputs and non-deliberate failures.
Second, critics such as Nash and Felton argue that legislation should focus more on regulating digital platforms as systems rather than just hosts of content.Footnote 47 The OSA 2023 currently adopts a content-based regulatory approach, imposing duties on regulated services regarding the content uploaded on platforms. The OSA 2023 tackles the aspect of removing illegal content and activity harmful to children (section 1(2) of the OSA). However, this neither directly tackles human vulnerabilities nor holds the responsible actor(s) liable under a criminal standard. Criminal liability would fall under existing criminal offences such as fraud by false representation (section 2 of the Fraud Act 2006) in the UK.
Taking the example of a celebrity’s image being used in a fraudulent deepfake advertisement for a cryptocurrency investment scheme uploaded on an online social media platform,Footnote 48 the OSA 2023 criminalises fraudulent advertising on regulated services. A service may be classified as either a Category 1 or Category 2A service depending on the threshold conditions.Footnote 49 In the case of a Category 1 service, internet service providers must comply with more duties than service providers in Category 2A or 2B. For instance, companies must put in place proportionate systems and processes to prevent (or minimise in the case of search engines in Category 1A) the publication and hosting of fraudulent advertising on their services. Companies must also remove such content when they are made aware of it. Non-compliance can lead to criminal liability for named senior managers.
Despite the promise to regulate scam advertisements, the OSA 2023 is unable to cope with the ‘cat and mouse’ game of removing harmful online content. TikTok removed 85.8 million pieces of content in Q4 of 2021Footnote 50 while Instagram removed 43.8 million pieces of content in the same period.Footnote 51 Even with content moderators working at full speed, it is difficult to keep pace with the vast amount of content uploaded every day. Further, although deepfakes can be malicious and harmful, deepfakes cause the most damage when they are operating in conjunction with other technological tools such as cryptocurrency wallets, QR codes, and manipulation of platform engagement metrics.Footnote 52 We illustrate this with the example of Elon Musk’s image being used in a deepfake cryptocurrency scam. On 18 June 2024, scammers illegally obtained $50,000 in cryptocurrency in just two hours with a deepfake video of Musk soliciting investments via a YouTube livestream. Viewers who invested were first deceived through the deepfake video of Musk. The video was streamed on YouTube channels with a high number of subscribers. Scammers then rebranded the YouTube channels to impersonate the official Tesla YouTube channel before launching a livestream.Footnote 53
The scammers directed the investors offline through a QR code to invest. Moving investors offline and onto their mobile phones has the benefit for scammers of less content moderation and scam detection. Therefore, effective regulation requires thorough processes by user-to-user services to identify the risks when deepfakes are working alongside generative AI, QR codes, and search engines.Footnote 54 The highly convincing deepfake video was only one element that contributed to the fraud. Anonymity and opacity arising from moving cryptocurrency through several wallets, manipulating subscriber numbers, and diverting investors offline through a QR code all contributed to the sophisticated fraud. This case illustrates that deepfake investment scams often operate within a landscape of other technological tools and platforms, which makes it difficult to precisely attribute liability to the individual actors involved.
The modus operandi of deepfake-enabled financial scams is varied, though. Deepfake scammers can directly target senior company leaders, as illustrated by the widely reported case of a deepfake financial scam involving the Hong Kong office of a UK company called Arup. In the case of Arup, the company lost $25 million to a deepfake conference video.Footnote 55 This case did not involve the use of social media platforms, so the deepfake supply chain is shorter but nonetheless cross-border in nature. Therefore, there are varied methods of committing financial deepfake scams, which ultimately exploit human vulnerabilities, not organisational system vulnerabilities.Footnote 56
Farrand posits that the OSA is too narrow since it focuses on individual ‘psychological or physical’ harms to specific users and not systemic harms that affect wider society.Footnote 57 Deepfake financial scams do not fall within this definition because they are non-linear, networked, algorithmically amplified, and spread across multiple platforms.Footnote 58 The OSA 2023 does not tackle the systemic conditions in the fraud ecosystem. By contrast, the EU DSA 2022 requires very large online platforms to assess and reduce systemic risks arising from algorithmic design, amplification dynamics, and manipulation techniques (recital 69; article 34 DSA 2022). As a result, the OSA 2023’s content-based approach struggles to address the infrastructural and behavioural drivers of deepfake financial scams, whereas the DSA 2022’s systemic-risk framework is expressly designed to regulate the platform-level mechanisms that make such harms scale.
The legal framework in the EU
Article 50(4) of the EU AI Act sets out disclosure requirements for ‘deployers of an AI system that generates or manipulates image, audio or video content constituting a deep fake’.Footnote 59 Labelling AI-generated or -manipulated content improves transparency, especially when several studies demonstrate that customers find it difficult to establish whether AI-generated media is real or not.Footnote 60 Like the EU AI Act, the US AI Labelling Act 2023 requires developers of generative AI systems to disclose ‘clear and conspicuous information indicating the content includes the use of AI’. Wittenberg and others show that labelling can reduce an individual’s chance of relying on and engaging with misleading AI-generated images.Footnote 61
With the public possessing varying levels of knowledge and familiarity with generative AI, designing effective labels with the appropriate content is key to preventing deepfake fraud.Footnote 62 However, empirical studies have demonstrated that authorship labels can be effective in enhancing transparency, although they are unlikely to affect the persuasiveness of the labelled content which requires alternative techniques to address risks posed by AI-generated information.Footnote 63 Wittenberg and others opine that labels indicating whether content has been generated or manipulated by AI are process-based and are not as effective as labelling content as ‘manipulated’ or ‘false’, which is impact-based. The latter is more effective in changing people’s behaviour.Footnote 64 Ternovski and others’ experiments reveal that general labels warning the public about videos led the public to distrust all videos as fake, even if they were real.Footnote 65 Further, the participants in the experiments did not show any improvement in identifying manipulated videos. Therefore, research to date shows that targeted labelling appears to be more effective than general labelling.
Articles 34 and 35(1)(k) of the EU DSA 2022 also mandate very large online platforms to conduct thorough risk analyses, curb the dissemination of illegal content, and label deepfakes and false information. Although this systems-based framework has its advantages, there are weaknesses. The ‘notice and action’ mechanism in article 16 of the Act allows individuals or entities to ask online platforms to remove illegal content. However, article 16(2) sets out conditions for activating the ‘notice and action’ mechanism, one of which requires the individual or the entity to provide ‘a sufficiently substantiated explanation of the reasons’ why they think the content is illegal. This seems rather difficult for the public to determine, especially given that the definition in article 3(h) of the EU DSA 2022 is broad and vague. Whilst child abuse content is clearly illegal, the public needs to provide reasons why this is illegal. This means searching for suitable legal authorities. For non-lawyers, this can be an onerous task. Indeed, de Streel and others report that many stakeholders, such as non-governmental organisations, are of the view that the users ‘are not necessarily capable of accurately identifying illegal content online’.Footnote 66 The users can only make assumptions that the content is likely to be illegal.
Content moderators compare the content on platforms against databases storing illegal content such as the Image Watch List by the Internet Watch Foundation.Footnote 67 This process, called hash-matching, is made more difficult because generative AI can reproduce edited versions of the same illegal content very quickly, bypassing databases which hold illegal content.Footnote 68 Further, auto-detection software such as watermarking and labelling methods used by online platforms has weaknesses.Footnote 69 In particular, such software is not very effective with shorter texts, content translated from other languages, or factual replies and it implicitly treats synthetic media as false and deceptive.Footnote 70 Whilst there are databases of illegal child abuse (IWF Watch List and Interpol),Footnote 71 there do not appear to be similar ones for fraud. Even for established databases for illegal content, Stockwell and others suggest that it is important that these repositories use standardised metrics, such as harmonised labelling and classification methods.Footnote 72 Standardisation of repositories would be helpful given the global nature of the internet and its related problems.
The regulatory approach in Singapore
Singapore adopted a piece of legislation specifically targeting deepfakes in political elections. The Elections (Integrity of Online Advertising) (Amendment) Act 2024 bans the publication, boosting, sharing, and reposting of deepfake content depicting election candidates. This adds to the protection given in the Online Safety (Miscellaneous Amendments) Act 2022 and the Online Criminal Harms Act 2023 (OCH Act). The Online Safety (Miscellaneous Amendments) Act 2022 amends the Broadcasting Act 1994 by imposing legal duties on social media platforms. There are two main duties. First, social media platforms with a significant impact must comply with certain Codes of Practice. Secondly, the Infocomm Media Development Authority (IMDA) can require social media platforms to disable access or stop ‘egregious content’ being transmitted to users.Footnote 73 ‘Egregious content’ broadly covers sexual content, child abuse, and terrorist content but does not refer to financial scams.
The OCH Act covers a range of offences, including scams and malicious cyber activity offences under Part 2 of Schedule 1. The most notable difference between the UK’s Online Safety Act and Singapore’s OCH Act is that the latter imposes legal duties on government officials, not online platforms. Section 3(1) of the OCH Act establishes the ‘competent authority’ as ‘a public officer from a Ministry or department of the Government; or an employee of a public authority’. Their legal duties are then set out in section 3(2) of the OCH Act. The government can issue five types of directions under the OCH Act, namely: stop communication, disabling, account restriction, access blocking, and app removal directions. A study by the IMDA in Singapore shows that online platforms are often slow to remove harmful content: on average, they take at least five days to respond to the victims.Footnote 74 This is a possible explanation for the additional role the government plays in dealing with harmful content.
Regarding scams and malicious cyber activities, the government only needs to suspect that online harm has occurred to issue a direction. For other activities, reasonable suspicion is required. This contrasts with the requirement for ‘reasonable grounds to infer’ that the content amounts to an offence in section 192(5) of the OSA 2023 for all offences, including scams and cyber activities. The lower Singaporean threshold of merely suspecting that a scam or malicious cyber activity has occurred should make it easier to remove such content. This may be explained by the fact that Singaporeans are particularly vulnerable to scams because many of them are ‘affluent, digitally literate and compliant’.Footnote 75 Further, scam victims in Singapore suffer some of the highest losses in the world.Footnote 76
Interestingly, a new government agency named the Online Safety Commission, would provide faster remedies for victims.Footnote 77 Under the Online Safety (Relief and Accountability) Bill, scam victims can request perpetrators’ information to commence legal proceedings. They can also sue online platforms and page administrators for deepfakes, cyberbullying, and sexual harassment. Online platforms must remove offensive content reported by users and victims. The Singapore government has also required app stores to carry out age verification through technology such as facial scans so that only users over eighteen years old can use the apps.Footnote 78 These additional protections for scam victims, along with the Broadcasting Act and Protection from Harassment Act 2014, indicate that the Singaporean legislative framework has adopted a strict regulatory approach to online scams.
The challenges of liability attribution for malicious deepfake scams
Establishing liability for GenAI deepfakes poses multiple challenges. Investigations to identify malicious content creators are very costly and time-consuming.Footnote 79 For example, a CEO fell victim to a deepfake video and the company detected the fraud quickly but was unable to catch the fraudster or recover financial losses.Footnote 80 Further, many deepfake investment scams involve the use of cryptocurrency. Fraudsters use deepfake videos, free cryptocurrency, and promotion codes to lure victims to fake websites.Footnote 81 Once the victims enter the promotion codes, they are asked to pay Bitcoin and share their personal details, and they never get anything in return. This is particularly problematic, given that the number of generative AI-enabled scams has increased by 78% from 2023 to 2024 and then by 456% from 2024 to 2025.Footnote 82 In the Elon Musk cryptocurrency scam of 2024, Rizzuto used a blockchain explorer and was able to trace the addresses of the scammers’ wallets.Footnote 83 It was found that Bitcoin, Ethereum, and Dogecoin wallets were used in the scam and new wallets were used to hide the transactions. Yet, such cryptocurrency platforms are currently unregulated, which shield bad actors from liability to scam victims.
Liability of deepfake content creators
Content creators who use deepfakes to defraud victims should in theory bear responsibility for their crimes. Malicious content creators have the intention of misleading others and are using AI technology as a weapon to cause harm. As Mirsky and Lee posit, ‘for deepfakes, the objective of the generated content is to fool a human’.Footnote 84 In the UK, there is a gap in legislation concerning the accountability of malicious content creators. As mentioned above, deepfake scams are intended harms, the prevention of which involves monitoring misuse by users and addressing malfunction, whilst unintended harms concern rectifying design defects in AI outputs. Deepfake scam victims will need to rely on existing UK criminal legislation such as section 2 of the Fraud Act 2006 (fraud by false representation) as a potential basis for criminal prosecution. However, gathering clear evidence of intent to deceive by the deepfake content creators can be very challenging.
If a liability framework is to be created, we submit that the focus should be on holding content creators liable for failing to implement verification or safeguard measures such as disclaimers as to how much content has been altered.Footnote 85 The challenge, though, is that many such content creators are anonymous because of their nefarious activities. There are significant financial and jurisdictional barriers in uncovering the identity of these anonymous content creators.
Law enforcement techniques in the form of metadata analysis,Footnote 86 network traffic analysis,Footnote 87 financial trails,Footnote 88 and behavioural analysisFootnote 89 can all assist in uncovering the identity of the content creator. Courts can also issue search warrants for the police to enter and search premises for evidence of serious offences, including evidence from electronic devices under section 8 of the Police and Criminal Evidence Act 1984. Metadata analysis of Ross Ulbricht’s Gmail address and a financial investigation by the Federal Bureau of Investigation led to the successful prosecution of Ulbricht, the mastermind behind the Silk Road online black market.Footnote 90 This case involved Ulbricht selling illegal drugs worth $214 million.Footnote 91 He disguised his identity by using the username Dread Pirate Roberts. The scale of the operation was unprecedented, with buyers and sellers from all over the world.Footnote 92 Although the US law enforcement agencies do not disclose the costs of investigations generally, the Silk Road case involved multiple federal agencies, complex digital forensic work, and a three-week federal trial.Footnote 93 Thus, it would have been a very expensive investigation. The Silk Road case was of a huge scale and led to drug-related deaths. Not every case will justify the significant investigative costs. Therefore, imposing liability on content creators poses real challenges.
The interaction between content creators and online social media platforms
Online social media platforms integrate GenAI technologies into their product offerings to enhance content creation workflows and empower content creators. For example, ChatGPT and Copilot are AI-powered software applications powered by large language models (LLMs). They are designed to assist users in generating text through automated programming interfaces which analyse data patterns without human direction.Footnote 94 Companies largely embed generative AI models into their digital systems, eg to improve user engagement, enhance content quality, and support creators with limited resources or technical expertise.Footnote 95 However, the interactions between content creators and online social media platforms raise concerns about the employment of GenAI technologies to produce reliable outputs.Footnote 96 GenAI could significantly improve creators’ productivity and enhance the platforms’ profitability. When creators operate on online social media platforms and choose to use AI, the quality of content improves, thereby facilitating economic returns for the platform. As noted, ‘platforms offer creators visibility, enabling them to attract and engage audiences. In turn, platforms sell a share of this attention to advertisers’.Footnote 97
The social media platform captures GenAI’s potential to boost creators’ productivity while maintaining their incentives to participate in content creation.Footnote 98 The issue of AI-generated deepfake financial scams has caused widespread concern for investors and consumers; more sophisticated AI tools promote scams that cause harm on online social media platforms rather than protecting vulnerable customers from worthless investment schemes.Footnote 99 GenAI’s ability to catalyse deceptive or fraudulent content may reflect the appetite of big tech companies (eg Meta, Google, Apple, Microsoft) to publish misleading advertisements enabled by scammers in pursuit of substantial revenues. This agency relationship identifies potential collusive behaviour between content creators and online social media platforms, which can increase misinformation and individuals’ vulnerability to deception. It also supports Nash and Felton’s argument that legislation should focus more on regulating online social media platforms as systems rather than just hosts of content.Footnote 100
It is generally considered that online social media platforms provide the environment in which creators publish content and offer important monitoring capabilities for community management.Footnote 101 These platforms provide analytics tools that enable creators to measure their activities. However, it remains unclear how content creation, advertising strategies, and user responses are governed in light of such data. This can raise concerns about the platform’s responsibility for detecting, flagging, and removing harmful content to protect users and ensure trust in the distribution of information.Footnote 102 Social media companies have often been shielded from responsibility on the assumption that they provide innovative technologies and platforms for speech, such that imposing restrictions on them would amount to a form of censorship of technological progress.Footnote 103 Doctrinal debates have questioned whether social media platforms exercise their own freedom of expression rights in the course of content moderation decisions.Footnote 104
Section 230 of the US Communications Decency Act 1996 protects technology companies from legal claims arising from user-generated content on their platforms, treating them as neutral hosts (similar to telephone companies) rather than as publishers.Footnote 105 While US legislation affords immunity to platforms for removing offensive content, courts have begun recognising causes of action that address the weaknesses and risks of social media.Footnote 106 Section 230 provides that an entity is shielded from liability only in its capacity as a provider of an interactive computer service. However, the provision does not protect a company when acting as the publisher or speaker of information provided by another information content provider.Footnote 107 As a result, a service provider is not shielded from liability where it makes a material contribution to the creation or development of content.Footnote 108 It is worth noting that GenAI outputs are not protected by section 230, thereby undermining any defence available to companies. When a platform incorporates GenAI applications into features such as messaging and profile creation, it may jeopardise its section 230 protection.Footnote 109 Lawsuits against social media platforms have been successful on the grounds of users’ mental health harms, as illustrated by the cases State of New Mexico v Meta Platforms, Inc. Footnote 110 and KGM v Meta Platforms et al,Footnote 111 where courts have focused on how technology platforms design their products and services to protect users from misleading or harmful content.
The EU’s AI Act attempts to address accountability gaps between providers and deployers. Providers are responsible for the design and documentation of AI systems, while deployers manage their contextual use and oversight. For general-purpose AI and foundation models (eg chatbots), upstream obligations focus on technical documentation, systemic risk mitigation, and downstream information flows, recognising their infrastructural role beyond single use cases. However, these foundation models pose significant challenges for regulators and policymakers as they can be adapted to sophisticated automated tasks such as those enabled by large language models, which make it difficult to detect malicious scams and to allocate responsibility.Footnote 112 Article 50 of the AI Act regulates transparency obligations for providers and deployers of certain AI systems, requiring the provision of specific information about the content and outputs of the AI system. In general terms, providers of high-risk AI systems are required to adopt additional precautions and safety measures to mitigate the undesirable effects of unexpected outcomes.Footnote 113 These obligations are complemented by the Code of Practice on the transparency of AI-generated content, which sets out regulatory measures – whose adoption is voluntary – for providers and deployers of GenAI systems on how to comply with the requirements of the AI Act.Footnote 114 Risk management requirements cover predictable and reasonably foreseeable harms associated with the use of AI applications, but sophisticated computational systems may fall outside the taxonomy introduced by the Act. Application layers deployed on AI systems generate new risks within the value chain, raising questions as to the need to examine the broader network, as the operational adequacy of models cannot serve as a blanket justification for assigning responsibility.Footnote 115
Article 25 of the AI Act classifies distributors, importers, deployers, or other third parties as providers of high-risk AI systems, thereby prescribing responsibilities along the value chain. This provision must be read in conjunction with article 16 of the Act, which sets out specific obligations for the product manufacturer as the provider of the system, focusing on the disclosure of information concerning the tools, services, components, and processes used or integrated within a high-risk AI system. Specifically, article 16 establishes a set of requirements to ensure compliance with the applicable conformity assessment procedures, including a series of control mechanisms related to the quality management system. Furthermore, article 17 of the Act requires providers of AI systems to implement techniques, validation procedures, testing, examination, and systematic measures for the design, development, and data governance of the system. At first sight, the bulk of obligations placed on providers suggests that the legislation identifies them as the primary point of reference for allocating responsibility. However, unforeseen events – particularly where developers ought reasonably to have known of the existence of a risk prior to its materialisation − complicate this allocation.Footnote 116
The perpetrator and attacker feeding adversarial data and poisoning AI models introduce unforeseen risks, which may jeopardise the attribution of responsibilities and the grounds for duties that arise in relation to the associated actions necessary to prevent the negative consequences of such events. On this point, the doctrinal debate has questioned how to identify AI providers’ duties in relation to AI risk management and risk assessment.Footnote 117 Unintended consequences of unforeseen outcomes are context-based, spanning predictive analysis and decision-making processes across various stages of specific AI services (eg machine learning, cloud computing, and GenAI applications). These dynamics significantly affect the control mechanisms governing AI systems and may shift the role of providers in relation to customer-facing application functionality.Footnote 118
The adoption of safety measures and the ability to foresee certain risks are key to allocating liability along the AI value chain, although the level of participation of each actor, their expertise in identifying harmful effects of defective products, and the design of the AI model may alter the distribution of responsibility. As argued, ‘the manufacturers of key AI components, such as foundation model providers, may be best positioned to adopt safety measures and foresee certain risks and thus should be held liable, or jointly liable, when harm is caused by an AI system’.Footnote 119 Foundation models constitute a primary source of deepfake scams, as they are susceptible to data manipulation and cyberattacks by downstream developers. Articles 53 and 55 of the EU AI Act impose obligations on providers of general-purpose AI models to disclose information concerning the training and testing processes, as well as the results of their evaluation, and require the adoption of codes of practice aimed at mitigating systemic risks originating from upstream developers. However, deepfake financial scams also raise concerns in relation to downstream developers, whose modifications of foundation models by other actors within the AI value chain may generate unforeseen harms that render safety measures ineffective.Footnote 120
Downstream activities are difficult to detect within large datasets, given the ability of third-party developers, such as those operating through application programming interfaces (APIs), to conceal their identity when fine-tuning the original model.Footnote 121 Attributing responsibility and accountability to AI providers and digital platform operators for downstream harms may therefore constitute an arduous exercise, requiring a careful assessment of causation, foreseeable consequences, and applicable standards of reasonable care, which EU legislation has not yet fully addressed.Footnote 122 Imposing risk management requirements, together with a set of duties of care, skill, and diligence for senior board members of companies deploying AI models, may represent a viable option for allocating responsibility, although specific harms, such as errors in downstream activity, remain difficult to foresee.Footnote 123 Establishing a causal link between downstream activity and defective outputs requires a meticulous inquiry into whether the actors involved in the value chain possessed a sufficiently high level of expertise at a given stage of the process, as well as into the specific patterns derived from training datasets.
We argue that a shared liability framework can shed light on the technical complexities that the AI value chain presents for determining liability associated with fraudulent behaviour. Such a framework may operate as a deterrent mechanism by targeting each actor’s contribution across the system life cycle, thereby facilitating the foreseeability of risks and addressing regulatory gaps in the EU AI Act, which remains largely silent on how to allocate accountability for harm between upstream and downstream developers. However, a shared liability regime must be assessed on a contextual basis, and it may open the floodgates to users’ compensation claims, given the incentive to seek redress from multiple parties.Footnote 124
The idea of moral sense and self-restraint on the part of AI creators should not preclude the adoption of online regulations aimed at preventing the dissemination of misleading and fraudulent information.Footnote 125 Such regulations should not remain confined to the national level; rather, there is a need for a global consensus on how AI systems are assessed and managed, and on how liability is allocated for losses. However, as argued, the effectiveness of online regulation depends on the extent to which users can circumvent restrictions or substitute non-compliant platforms.Footnote 126 Creators know how to profit from their content, while users increasingly perceive platforms as providers of valuable goods and services, enabling them both to consume content and to engage in shopping activities. However, this interaction may give rise to a pernicious cycle, whereby creators are incentivised to maximise income generation and platforms, in turn, promote user engagement to accommodate creators’ demand for more predictable remuneration from content-based advertising.Footnote 127
Online social media platforms’ responsibility for deepfake scams
Sophisticated adverts, including AI-generated deepfakes, persuade individuals to disclose personal data or invest in fraudulent schemes. Online social media platforms are a favourite conduit for carrying such adverts and profiting from them, and big tech companies seem unable to curb the proliferation of deepfakes.Footnote 128 According to the US Federal Trade Commission’s Consumer Advice Report, ‘scammers use social media platforms to promote bogus investment opportunities, and even to connect with people directly as supposed friends to encourage them to invest’.Footnote 129 Frauds originating on social media (eg Facebook, Instagram) indicate that platforms are the main method of contact, and cryptocurrency is the most used method of payment.Footnote 130 We focus on regulated social media platforms such as Facebook and Instagram; largely unregulated platforms (such as Telegram, Signal, WhatsApp)Footnote 131 are outside the scope of our analysis. Public regulators and policymakers call for action to protect consumers as fraud causes severe harm to individuals and the stolen money goes to organised crime groups mostly hidden behind crypto platforms.Footnote 132
Technology-enabled frauds require greater collaboration between the public and private sectors, and a ‘systemic’ solution, as scammers adopt new methods and increase unauthorised losses which exacerbate a systemic problem.Footnote 133 Online social media platforms should bear a legal duty to check that investment advertisers are authorised, and a legal duty not to provide advertising space to fraudsters in the first place. They ought to be expected to ‘know their customers’ and be held liable, with proper enforcement and tough penalties, if they fail to block the dissemination of fraudulent advertising content.Footnote 134 Further, online social media platforms should be legally required to verify that advertisers are authorised by a regulator to sell financial services and block them in case they are not.Footnote 135 A mandatory risk assessment for providers of very large online platforms about the dissemination of manipulative content through their services is required by article 34(1) of the EU DSA 2022, although this is not equivalent to an obligation to control the use of the platform. It seems more like a due diligence test which could result in a ticking-the-box exercise.Footnote 136 This solution would not address the issue of who should be held accountable for the harm caused by such frauds.
Court rulings have begun to recognise the responsibility of technology companies in cases where they fail to control the products and services they place on the digital market. A German regional court found Google liable for false claims (including scams, links to dubious companies, subscription traps, and suspicious phone calls) and inaccurate information contained in an AI-generated search overview, which was treated as its own content rather than merely as a list of search results.Footnote 137 It made clear that Google has influence over the AI’s offering and the algorithms with which the AI operates, which essentially convert the function of generative search from a hosting service to a publishing service.Footnote 138 Another interesting case is the UK’s Competition and Markets Authority (CMA) action against Google under the Digital Markets, Competition and Consumers Act 2024.Footnote 139 The CMA imposed targeted rules on Google’s search services following its decision to designate the company as having Strategic Market Status (ie market power combined with a position of strategic significance in a digital activity) in the provision of general search services.Footnote 140 This designation allows the CMA to introduce conduct requirements governing Google’s search activities, provided that such measures are proportionate to the objectives of ensuring fair dealing, open choices, and trust and transparency.Footnote 141
We argue that a regulatory framework imposing shared responsibility for deepfake financial scams on malicious content creators and online social media platforms would provide a viable redress mechanism for victims’ losses and would impose corresponding obligations on these actors to compensate those affected by fraud. Such a solution may be particularly appropriate where a content creator intentionally trains an AI model to produce deceptive outputs, and where online platforms use such outputs to distort or manipulate information (rather than to moderate or report harmful content) relevant to individuals’ decision-making processes.Footnote 142
The conundrum of a liability regime in GenAI
The liability contours of deepfake financial scams involve multiple actors at multiple stages of the GenAI supply chain: malicious deepfake content creators, software developers, and online social media platforms. Regulatory challenges around the degree of liability for malicious scams have created concerns about suitable remedies.Footnote 143 In most disputes, responsibility is placed on consumers to exercise reasonable care in their decision-making process following the principle of ‘caveat emptor’.Footnote 144 However, this approach does not resolve the issue of liability attribution for unforeseeable harms, which could not be reasonably prevented, such as inaccurate outputs generated by AI tools embedded in GenAI and LLMsFootnote 145 that ‘hallucinate’ fabricated or false information about historical events, legal filings, news, and research articles.Footnote 146 AI hallucinations produce plausible but incorrect outputs, even with perfect data, due to fundamental statistical and computational limits which cause unintentional errors in the decision-making process.Footnote 147 The datasets used to train the models can be biased, inaccurate, or intentionally disrupted by content creators who provide faulty narratives to manipulate information.Footnote 148 As argued, corrupted data and information make GenAI outputs no longer so intelligent, profitable, or socially beneficial for market participants.Footnote 149 However, GenAI hallucinations not only produce false outputs but are also entangled in the cognitive experience (eg memories, emotions, behaviour) that characterises the human-AI interaction in conversational interfaces.Footnote 150 GenAI tools hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty: AI models do not acknowledge uncertainty, but they provide estimates as facts without further research into the validation of their statements.Footnote 151 The impact of uncertainty about the performance of GenAI is a matter of concern for legal practice and consumer claim disputes, where the widespread use of AI models introduces risks of misperception when accessing valuable advisory assistance.Footnote 152 There is a pressing need to address the harmful impact of outcomes due to the uncertainty and unpredictability of risks in the GenAI supply chain.Footnote 153 AI risk frameworks have proved inadequate for the reality of persistent hallucinations; they often underestimate epistemic uncertainty, so updates are needed to address systemic unpredictability.Footnote 154 In the doctrinal debate, it has been noted that additional concerns result from the lack of a universally accepted definition of hallucinations.Footnote 155 As a result, there is no reliable way of measuring their occurrence and evaluating the reliability of specific GenAI-based tools. Requiring disclosure or certification of GenAI use, such as a mandatory ‘hyperlink rule’Footnote 156 in legal citations and contentions to authoritative sources, would mitigate the risk of users’ reliance on hallucinated cases through verification at the point of citation.Footnote 157
Situations of unintended harms caused by faulty outputs (eg hallucinating critical information in providing advice or assistance as a result of genuinely defective AI behaviour) are distinct from intentional abuse of a dual-use technology to commit fraud (eg human-generated misinformation that influences decision-making). Human misinformation is technically and conceptually different from AI hallucinations in terms of motivation, beliefs, and intent.Footnote 158 This results in human-initiated inaccuracies which involve deliberate deception or flawed prompts, leading to disinformation as intentional falsehoods, and computational manipulation of individuals’ choices.Footnote 159 As a corollary, intentional misuse of GenAI may exacerbate the risk of deterioration of the information ecosystem with lower-quality, synthetic, or misleading content, effectively generating untruthful content.Footnote 160 Sophisticated forms of market manipulation come from misuse of AI techniques such as algorithmic trading, which faces ‘challenges to guaranteeing accountability and assigning responsibility, as human experts do not explicitly program the AI behaviour’.Footnote 161 The complexity and interoperability of automated trading practices as well as their autonomy in the decision-making activity (self-learning from their own experience based on market observation) make it difficult to identify the degree of liability (among the parties involved in designing, developing, using, and monitoring the AI system) for misconduct and harm when operating in financial transactions. An innovative proposal to implement a credible regulatory deterrence framework to achieve effective law enforcement of securities law has been suggested in the doctrinal debate with a view to addressing unlawful behaviours and conduct in the trading context.Footnote 162
Manipulated content in deepfake financial scams undermines technological advancements and exploits vulnerabilities in software design to facilitate intentional fraud, misinformation, and emotional harm.Footnote 163 The Financial Stability Board (FSB) warned about the potential of GenAI to increase financial fraud and enable malicious actors to generate and spread disinformation in financial markets, which underscores the need for accountability mechanisms and content moderation.Footnote 164 Generative AI visual text is produced by a human designer or creator, adapted to representational media for altering users’ beliefs, and distributed to mislead the audience.Footnote 165 It is the human who is at fault for deceiving and distorting reality through materials artificially created by the AI tool; this affects market structure and trading exchanges with negative implications for competition and transaction costs.Footnote 166 When AI tools self-learn and self-operate with disrupted data to achieve an unlawful objective, it is difficult for regulatory authorities and market users to understand how the outputs originate. Conduct of business rules can mitigate the risks of traders committing abuses, although the high-speed data used in financial products makes monitoring and supervisory activities meaningless with respect to technological errors.Footnote 167 Risk management and internal controls within financial firms are often inadequate to detect the potential harms created by malicious deepfake scams; in parallel, regulatory compliance frameworks and prudential standards adopted by financial authorities exhibit weaknesses in terms of skills and resources (including deficiencies in data science expertise and essential IT capabilities) necessary to assess the reliability of AI tools.Footnote 168 Model validation, ongoing monitoring, outcome analysis, and data quality assessment entail substantial compliance costs and require a thorough understanding of the vulnerabilities associated with AI models. This, in turn, calls for the development of policy frameworks that enhance engagement with private sector participants, AI developers, and other third-party service providers.Footnote 169
The shared liability model
We identify different scenarios for allocating responsibility among parties implicated in fraudulent AI-generated outcomes. We contend that a shared liability regime would provide a legal response to fraudulent content, as well as act as a deterrent by discouraging malicious content creators and online social media platforms from disseminating deepfakes. This approach stems from the wider discussion about the conceptual framework of shared single liability among multiple injurers.Footnote 170 Shared responsibility applies to multiple stakeholders proportionally responsible for a particular action or outcome.Footnote 171 The share of liability must be assessed on a case-by-case basis depending on the duty borne by each party involved in the illegal activity. The limitations of this proposed liability approach lie in the potential hurdles in identifying the specific harm associated with the deceptive action for each party involved in the AI supply chain. These limitations reflect the risk of misallocating the burden of responsibility among the actors engaged in the system life cycle, thereby both diluting accountability in the decision-making process and placing disproportionate compliance burdens on AI providers.
Andhov and Gardner have observed that complex algorithmic decision-making systems challenge the application of tort law as an appropriate private cause of action to ensure compensation for individual victims of financial harm.Footnote 172 Specifically, it could be difficult to demonstrate that the fault of the algorithm caused the harm experienced. In the context of the tort of negligence, risks of harm can be reasonably foreseeable, and the exact causal mechanism of the harm does not need to be reasonably foreseeable.Footnote 173 Howells and Twigg-Flesner have proposed a liability approach that allows users to seek redress without having to face the difficulties associated with identifying the correct party responsible.Footnote 174 Ben-Shahar offers a view on fault-based liability in tort law and apportionment problems by proposing safety score liability, which is commensurate with a party’s habitual propensity to behave unsafely.Footnote 175 This proposal relies on measurable and predictable actions of individuals, which are identified through a machine learning algorithm that calculates scores to assess the likelihood of causing accidents that result in harm.Footnote 176 The implementation of safety score liability would be suitable for assessing the responsibility of content creators and software developers, although the algorithmic system that calculates the scores should be trained with accurate data on the parties involved, which may prove difficult in the case of anonymous malicious deepfake scammers. Some commentators have found limitations in liability law, pointing out that it is ‘too blunt an instrument for allocating responsibility to meet society’s needs, and that any attempt to improve that allocation via law and regulation needs to focus on the implementation of appropriate governance systems by GenAI supply chain members and also to recognise that risk prevention is an impossible target, instead focusing on mitigation of risks to a socially acceptable level’.Footnote 177
We situate the shared liability regime within the law of causation in tort to identify responsible actors in the deepfake network among multiple causes of the same injury which arises in negligence and results from significant, foreseeable, and harmful risks.Footnote 178 This regime allows the identification of the burden of responsibility for manipulating the GenAI system across the supply chain while providing consumers with effective remedies to claim compensation where fraudulent practices compromise their decision-making process. The shared liability regime may have limitations in addressing the reasonable expectations of safety and reliability that an average consumer has regarding AI software.Footnote 179 The practical implementation of shared liability is analysed through the actions of the parties that increase the risk of faulty outcomes. The more fundamental problem is identifying certain actors at all, especially given limited enforcement resources, anonymity, and cross-border operations. These real-world constraints are essential to any feasible liability model.
Our proposal for a shared liability regime advances existing debates on the development of a more resilient AI risk framework, in which all parties are required to implement more robust verification technologies and protocols to protect consumers. In cases where victims are scammed, a shared liability regime would provide guidance on the allocation of liability among the relevant parties.
Liability for deepfake software developers
The missing parties being held liable in the deepfake scam supply chain are deepfake software developers. Such software can be used for legitimate and even beneficial purposes. This makes it difficult to impose blanket liability on all deepfake software developers. We consider the scenario of liability for software engineering in which developers hold responsibility for using methodologies which may be manipulated for harmful deepfake content (eg audio, visual images, advice). Developers of automated tools such as agentic integrated development environments (LLMs such as GitHub and Copilot X) can generate code to support human skills.Footnote 180 Software development methodologies require ongoing monitoring, adaptation, and refinement to avoid hacking and data leakage.Footnote 181 Developers can implement safeguards in software to prevent hackers and malicious users from executing cyberattacks. Although the safeguards can be evaded by a downstream user, a few proactive techniques can be adopted to monitor users performing harmful tasks, such as open-sourcing, pre-release stress-testing of models, structured model access for third-party auditors, and research APIs to facilitate external independent model evaluation.Footnote 182 These defensive strategies can be implemented by creating a group of ‘red-team professionals’ to provide effective safety evaluations of AI models.Footnote 183
The shared liability regime can be implemented for developers based on the tort of negligence for failure to exercise due care in ensuring the safety and reliability of the software (generated code and automation) and in providing software updates. Developers should be responsible for code review and debugging of software, although they often bargain with the online platforms, agreeing to distribute their software exclusively through big tech companies in exchange for reduced fees.Footnote 184 This suggested view is in line with article 50 of the EU AI Act, which places a number of obligations on providers and users of AI systems to enable the detection and tracing of AI-generated content.Footnote 185
An interesting accountability regime exists in medical law, where software is viewed as a service rather than a product. This perspective arises from the shared fault liability for injuries resulting from the integration of AI systems into healthcare practice, which is distributed among physicians, hospitals, and product developers.Footnote 186 The EU Product Liability Directive (PLD)Footnote 187 includes software as a product for the purposes of no-fault liability, reframing it under the strict liability regime.Footnote 188 The impact is that producers of software can be held responsible for damage caused by their defective products, regardless of fault.Footnote 189 In the academic debate, it has been suggested that software has become an important element of products and is even remotely interconnected with them.Footnote 190 Therefore, the case for considering software like any other component has become stronger with the result that producers of the software itself could be liable in their own capacity whether as a supplier of software components or as a supplier of standalone software. On this view, it is observed that ‘as AI is more pervasive and ubiquitous – just like other software – it might soon be found in a significant number of consumer products and services’.Footnote 191 However, the definition of software as a product remains cryptic in the PLD as it is not clear whether digital content with equivalent functions and software as a service are included in the scope of the revised product category.Footnote 192 Specifically, the Directive does not provide a clear demarcation between the concept of software as a service (eg computer programs and AI systems) and software as a genuine digital service (eg digital offerings) when it is treated as a product in its own right. Recital 13 of the PLD provides some indication that software may be considered either as a standalone product or as integrated into other products as a component; however, it does not distinguish between ‘software’ and ‘service’, nor does it classify genuine digital services as related services (eg the supply of data), which are de facto excluded from the category of products.Footnote 193
Software is capable of causing damage through its execution, and manufacturers should remain liable for defectiveness as a result of related services within their control.Footnote 194 As Howells and Twigg-Flesner observe, the term ‘producer’ is given a broad meaning under the PLD.Footnote 195 It goes beyond the manufacturer and can include even the producer of raw components, the manufacturer of components, and even importers of the products. We posit that deepfake software producers would fall within the PLD and would therefore be accountable for victims’ losses.Footnote 196
Article 12 of the PLD states that multiple economic operators can be held liable jointly and severally for the same damage; the liability is not reduced where the damage is caused both by the defectiveness of a product and by an act or omission of a third party.Footnote 197 Responsibility for the testing of high-risk AI systems is placed under article 60(9) of the EU AI Act, which requires that ‘providers should be held liable for any damage caused in the course of their testing in real world conditions’. Limitations of the shared liability framework of the PLD emerge in relation to financial harms caused by faulty algorithmic procedures, which require an effective right to compensation and a redress scheme. Specifically, the shared liability approach may fail to capture responsibility for immaterial harms (eg discriminatory outputs, breaches of privacy, emotional losses), which necessitate enforcement mechanisms capable of awarding damages for defective products.Footnote 198
Prevention measures to deter malicious actors also include government certification programmes and authorisation schemes for assessing the software before deployment on the platforms.Footnote 199 The licensing regime would test the technology used to ascertain whether the software components (eg, operating system, code, and data)Footnote 200 are aligned with safety and reliability requirements. It has been noted that ‘regulators should only grant such licenses if applicants can show that the use of generative AI will provide consumers with financial guidance that is at least as accurate and well-tailored as the guidance that is provided by more conventional symbolic AI’.Footnote 201 The rapid development of face-swapping AI technology raises questions about how to identify the scam.Footnote 202 The capacity for face-swapping to deceive is bolstered by voice-cloning applications. Voice-changing devices raise concerns about their use in various operations such as buying a product on the platform or creating voices and cloning them.Footnote 203 The significant challenge is that face-swapping can be used for both legitimate commercial purposes and malicious investment fraud. The question, therefore, is how the licensing system can regulate software with these dual uses. Proper safeguard measures such as regulatory requirements and compliance methods can be imposed on the software developer in order to detect the characteristics and purpose of synthetic media distribution and identify natural persons through fingerprint information.Footnote 204 It has been suggested that policy intervention should focus on the sociotechnical domain, such as the identification of content manipulations through authentication practices to verify the accuracy of the source of content and preserve the distribution chain.Footnote 205
The Australian Scams Prevention Framework Act 2025 requires digital platforms to check that all advertisers of financial products have an Australian Financial Services Licence and take specific steps to verify new accounts.Footnote 206 Along with these preventive regulatory tools, tech companies have developed trial processes such as regulatory and digital sandboxes to mitigate the risks of harmful outcomes of AI models. The European Council defined regulatory sandboxes as follows:
‘[C]oncrete frameworks which, by providing a structured context for experimentation, enable where appropriate in a real-world environment the testing of innovative technologies, products, services or approaches … for a limited time and in a limited part of a sector or area under regulatory supervision ensuring that appropriate safeguards are in place’.Footnote 207
Regulatory sandboxes such as a digital ‘scale up box’ can be implemented to monitor the quality and performance of GenAI applications.Footnote 208 These regulatory tools, launched by the FCA to test Fintech products, are designed to establish a safe environment to oversee technological software and foster innovation.Footnote 209 They aim to enhance cooperation between regulatory authorities and financial firms with a view to promoting best practices for transparency of automated systems.
An innovative regulatory approach against the risks of unexpected outcomes of technology applications is the FCA’s consumer duty, which concerns a firm’s conduct in relation to financial services.Footnote 210 The consumer duty rule sets the standard of care that firms should exercise towards customers in retail markets and holds manufacturers and distributors responsible for the assessment of foreseeable harms.Footnote 211 The duty applies across the distribution chain, including to the design or operation of products or services, and a firm's responsibilities depend on its role and ability to influence retail customer outcomes.Footnote 212 Manufacturers are required to monitor the characteristics of their products and whether they are fit for purpose, and distributors are under an obligation to have a clear understanding of the target market and the way products operate.Footnote 213 Software developers and digital platforms are expected to use watermarking techniques to deploy reliable content authentication and provenance mechanisms to enable users to identify AI-generated content.Footnote 214 Firms need to share information about harmful content in technological applications with other relevant parties, in particular where a distributor identifies foreseeable harm or problems with the way a product or service is operating in practice.Footnote 215 The consumer duty can impose a primary obligation for firms to report fraudulent behaviours in deepfake scams relating to GenAI products and implement internal controls to detect potentially harmful software.Footnote 216 The FCA duty rule can be implemented by firms to reduce the vulnerability of GenAI tools, although compliance requirements, such as risk management processes and transparency obligations, may not be effective in detecting (and avoiding) potentially harmful outputs.Footnote 217 Further, the consumer duty rule requires firms to conduct an assessment of the AI model and its use and operation before deploying the AI model in the market.Footnote 218
Risk management and verification measures for deepfake software developers and online social media platforms
We argue that deepfake software companies and online social media platforms should be held liable for negligence if they do not adequately implement risk management processes and verification measures to prevent harmful misuse such as deepfake scams, although we acknowledge that these solutions may prove ineffective in detecting in time the unintended malicious actions of users. Software developers should use biometric and identity verification to prevent unauthorised use of someone’s likeness by requiring consent verification before processing images or voice samples of identifiable individuals. In order to verify the integrity of content, watermarking techniques include hashing trace files with a small string of numbers that are lost if the audio or video is deepfaked.Footnote 219 Failure to employ digital forensic techniques in GenAI amounts to gross negligence, which may result in personal liability for individuals accountable for careless supervision of content authenticity.Footnote 220 Digital forensic techniques employ blockchain technology, which can verify the origins and distribution of videos by storing digital signatures in a ledger which is difficult to manipulate.Footnote 221
The UK Cyber Security and Resilience Bill 2024,Footnote 222 which updated the Network and Information Systems (NIS) Regulations 2018,Footnote 223 established a set of provisions that impose obligations on managed service providers (IT service providers and managed security services),Footnote 224 holding responsible those firms for failure to take effective technical and organisational measures (eg contractual requirements, security checks, or continuity plans) and to manage risks posed to the security of the network, infrastructure, and information systems.Footnote 225 The concept is similar to our proposal of imposing liability on those responsible for negligence. The Chinese Provisions on the Administration of Deep Synthesis of Internet-based Information Service 2023 imposed specific obligations and responsibilities on service providers, users, and online platforms for the management of synthetic media technologies.Footnote 226 Alongside these legislative frameworks, supervisory authorities should develop response programmes tailored to address the threats posed by deepfake manipulation. These programmes would establish procedures for identifying, assessing, and mitigating the harmful consequences of malicious deepfake content, supported by regulatory tools designed to enhance coordination between enforcement mechanisms, intelligence agencies, and other relevant stakeholders.
We submit that deepfake software developers and online social media platforms should implement risk management techniques to identify, assess, and mitigate the harmful impact of deepfake AI content. Where AI-generated content is detected but does not violate platform policies (eg parody), platforms should apply a prominent, immutable label (eg ‘AI-generated’ or ‘Synthetic Media’), which remains visible even when the content is shared or downloaded. Furthermore, if a user receives a message containing phrases such as ‘Bitcoin’, ‘investment opportunity’, or ‘send money’, a pop-up alert could warn: ‘Be careful. Scammers often promise guaranteed returns. Never send money to someone you have only met online.’ Similarly, if a user shares a video identified as a potential deepfake scam, a warning could appear stating: ‘This media has been identified as potentially synthetic. Learn more about deepfakes’.
Liability could also be imposed on software developers by implementing reasonable safeguards. For example, Microsoft launched safety benchmarks for developers to rank iterations from a range of providers with a view to building trust with cloud customers.Footnote 227 The safety metric evaluates whether a model can be used for malicious purposes. This testing programme based on rankings enables users to understand the risks posed by AI applications.Footnote 228 It is an innovative mechanism to monitor GenAI products by ensuring they cannot be used to create harmful content. The safety system evaluates the fairness, biases, and mistakes of AI products, and the risks of fraudulent hallucinated outputs in investment scams.Footnote 229
In 2025, the Singapore Government launched testing pilot schemes, namely the Global AI Assurance Pilot,Footnote 230 the Joint Testing Report with Japan,Footnote 231 and the AI Safety Red Teaming Challenge Evaluation ReportFootnote 232 to enhance the best practices of GenAI governance. Red teaming calls for scenario planning and adversarial testing of programmes in order to identify software developers’ deceptive behaviour.Footnote 233 The Monetary Authority of Singapore prescribes explicit duties for banks and telecommunications providers under the Shared Responsibility Framework, acting in coordination with the Infocomm Media Development Authority (IMDA) when allocating responsibility for losses arising from a defined scope of phishing scams.Footnote 234 The Supreme Court of Singapore has attributed liability for AI-generated content to users, thereby complementing the Shared Responsibility Framework and the Protection from Scams Act 2025.Footnote 235 In parallel, English courts have held that there is a professional duty to use AI tools ‘with an appropriate degree of oversight, and within a regulatory framework that ensures compliance with well-established professional and ethical standards’.Footnote 236
Under the OSA 2023, senior managers of online social media platforms may incur criminal liability if they fail to implement proportionate systems and processes designed to prevent the publication and hosting of fraudulent advertising. Within the shared liability model, we posit that, where the scale of harm justifies it (as in the Silk Road operation), content creators should also be held liable for deepfake scams. We acknowledge the challenges associated with uncovering the identities of anonymous content creators, which is why we have focused on the liability of software developers and online social media platforms. We further contend that deepfake software companies and online social media platforms should be held liable for negligence where they fail to adequately implement risk management processes and authentication methods designed to prevent harmful misuse, such as deepfake scams.
We suggest that online social media platforms should adopt protocol design and governance policy guidelines to mitigate the harmful risks posed by AI-generated content while, in parallel, assessing user expectations as to whether the purpose of such content is to mislead or distort beliefs. Corporate governance boards (including senior management, directors, executives, and compliance officers) of software companies and online platforms should be trained and fully informed about the reputational risks that the use of defective GenAI models may pose to business operations.Footnote 237 Poor monitoring of expected outputs and a limited understanding of AI decision-making processes can lead to compliance failures and flawed corporate reporting, thereby affecting the expectations of investors and market stakeholders.
Conclusion
Advances in generative computing software provide firms with greater opportunities to create smarter services for customers. However, they also provide scammers, fraudsters, and creators of deepfakes with more powerful tools.Footnote 238 GenAI can disseminate inaccurate information and unintended ‘hallucinations’ that are not factually reliable, and it is often difficult to trace how a particular output has been generated.Footnote 239 Malicious deepfakes operate at different stages of the AI supply chain through digital platforms, thereby complicating the identification of fraudulent activity.
The time is ripe for regulatory intervention. In this article, we propose a shared liability regime to attribute responsibility to different actors involved in the creation and dissemination of fraudulent AI-generated content. This regime would enable a more balanced approach to compensation, whereby the parties involved would bear losses proportionately, depending on whether and to what extent they contributed to the scam. The FCA’s consumer duty can complement the shared liability regime by requiring firms to avoid foreseeable harm to customers arising from the behaviour of AI models. However, the UK regulatory framework does not provide adequate responses to the risks of automated manipulation of information in synthetic media, as the impersonation of individuals by scammers exacerbates unfair practices against users. This suggests that UK legislation should consider specific liability approaches that account for the timing and nature of deepfake scams, which may undermine the safety and reliability of outcomes, while also imposing responsibility on online social media platforms to compensate victims. The normative experience of Singapore in tackling online scams and sharing the costs of fraud may offer a useful policy model for the UK Government in establishing mandatory rules for compensating losses and allocating liability among multiple parties involved in malicious deepfake activities.
Acknowledgements
The authors are grateful to the reviewers for their constructive and insightful comments on earlier drafts.
Competing interests
The author(s) declare none
Open access
