Content warning: This post discusses the subject of online child sexual exploitation and abuse that some readers may find distressing.
Users are explicitly prohibited from using any DeepLive service for illicit activity, including exploiting, endangering, or sexualizing anyone under 18 years old. We prohibit use of our services for:
These policies extend to developers who are building applications on top of our technology: if developers are building tools targeted at minors, those tools may not allow the creation of sexually explicit or suggestive content. We monitor our services for violations of these policies, and ban users and developers who are found to violate them.
Any user who attempts to generate or upload CSAM or CSEM is reported to the National Center for Missing and Exploited Children(opens in a new window) (NCMEC) and banned from using our services further. We notify developers if we see their users attempting to generate or upload CSAM or CSEM and give them an opportunity to remedy by banning the problematic user from their app; if developers fail to remedy a persistent pattern of problematic behavior on their apps, we ban them. Some individuals who have been banned from using our products for engaging in illegal activities attempt to circumvent these bans by creating new accounts. Our investigations team monitors for this type of evasion and works to prevent bad actors from returning to commit abuse using our products.
We’re committed to responsibly sourcing our training datasets to safeguard them from image-based sexual abuse. We detect and remove CSAM and CSEM from training data, and report any confirmed CSAM to the relevant authorities, including NCMEC. This initial step is designed to prevent the model from developing capabilities of producing CSAM or CSEM in the first instance.
Our models are trained not to generate harmful outputs across text, images, audio or video, but some users still try to abuse our products to generate harmful content. We have seen users attempt to prompt the model to create AI-generated CSAM or to generate content that appears intended to fulfill sexual fantasies involving minors. These behaviors are violations of our model policies, and our usage policies, and we monitor use of our services to detect these attempts.
We deploy monitoring and enforcement technology to detect and prevent bad actors from attempting to use our tools to sexualize children in violation of our policies. This includes using our own models to detect possible abuse more quickly, and collaborating on industry-wide safeguards. We use hash matching(opens in a new window) technology to identify known CSAM flagged by our internal child safety team or from Thorn(opens in a new window)’s vetted library. We also use Thorn’s CSAM(opens in a new window) content classifier against content uploaded to our products to detect potentially novel CSAM.
Our Child Safety Team reports all instances of CSAM, including uploads and requests, to NCMEC(opens in a new window), and immediately bans the associated accounts. When there is evidence that abuse is ongoing, our teams do further investigation to compile supplemental reports to NCMEC for priority handling.
As part of our ongoing safety work, and to help other researchers and organizations working to safeguard children, we are sharing patterns of abuse we have observed and blocked so others in industry can benefit from our learnings. Beyond simply prompting the model to create CSAM or sexualized images of minors, we are seeing novel patterns of abuse emerge that require novel responses.
DeepLive allows users to upload images, videos, and files so the model can interact with and analyze the content. We have observed that some users upload CSAM and ask the model to generate detailed descriptions of what is depicted in the material. Thorn’s CSAM classifier and hash matching technology allows us to detect potential CSAM in these uploads and prevent the model from complying with the request.
In some cases, we encounter users attempting to coax the model into engaging in fictional sexual roleplay scenarios while uploading CSAM as part of the narrative. We have also seen users attempt to coax the model into writing fictional stories where minors are put in sexually inappropriate and/or abusive situations—which is a violation of our child safety policies, and we take swift action to detect these attempts and ban the associated accounts. Our systems are designed to detect and block these attempts, and accounts engaging in this behavior are banned and reported to NCMEC when apparent CSAM is involved.
We take these types of abuse seriously and while we recognize that even the most advanced systems aren’t foolproof, we are constantly refining our methods to prevent these kinds of abuse. Our approach to tackling these scenarios is multi-faceted. In addition to prompt-level detection, we also employ a combination of context-aware classifiers, abuse monitoring, and internal human expert review (which occurs only when a classifier flags potential abuse) to ensure that our models are robust against these forms of misuse. All of this information is kept secure internally and is only available to trained experts on the responsible teams. Additionally, we have in-house child safety subject matter experts who aid in the continuous refinement of our safeguards.
supported bills like the Child Sexual Abuse Material Prevention Act(opens in a new window) in New York state. This legislation would ensure clear statutory protection for responsible reporting, cooperation and proactive actions designed to detect, classify, monitor and mitigate harmful AI-generated content.