Civitai, an online marketplace for AI-generated content, has recently attracted attention for its facilitation of custom instruction files, particularly for creating deepfakes of public figures, including celebrities. Backed by the prominent venture capital firm Andreessen Horowitz, Civitai has garnered scrutiny for allowing users to purchase and sell instruction files, known as LoRAs, that enable mainstream AI models like Stable Diffusion to produce content that may not have been part of their training data. A recent analysis conducted by researchers from Stanford University and Indiana University revealed that a substantial portion of these requests involved deepfakes of real women, raising ethical concerns surrounding the platform’s governance and content moderation policies.
The study highlighted that between mid-2022 and the end of the year, a significant percentage of bounties—requests for specific AI-generated content—were aimed at creating deepfakes, particularly targeting female celebrities. Users often sought high-fidelity models that could accurately represent body features or modify specific attributes. While Civitai has implemented a ban on sexually explicit deepfakes, many requests submitted prior to this ban remain active. This has led to questions about the effectiveness of the platform’s moderation strategies, as the majority of weekly requests reportedly now lean toward NSFW content.
Moreover, Civitai’s structure complicates the ethical landscape of deepfake creation. The site not only allows the sale of LoRAs but also provides educational resources on utilizing external tools to enhance image generation, which can include explicit content. Despite the company asserting its commitment to curtailing harmful content, the challenge remains in balancing creative expression with the potential for misuse. As the dialogue surrounding deepfakes continues to evolve, Civitai embodies the complexities of navigating innovation while addressing significant ethical dilemmas in the digital age.
Source: Inside the marketplace powering bespoke AI deepfakes of real women via MIT Technology Review