Is "Fake Information Factory" the Destiny of GenAI?
Researchers have found that most cases of GenAI misuse are normal uses of the system without any "jailbreaking" behavior, accounting for 90% of such "routine operations".
As the researchers explained later in the paper, the widespread availability, accessibility, and hyperrealism of GenAI have made possible numerous lower-level forms of abuse - the cost of generating false information is simply too low!
After reading this paper, you may involuntarily feel that people are not abusing GenAI, but are simply using it normally according to its product design.
People use generative AI to produce large amounts of fake content because it is inherently very good at completing this task.
Much of the fake content often has neither obvious malicious intent nor clearly violates the content policies or terms of service of these tools, but its potential harm is enormous.
This observation is consistent with previous reports by 404 Media.
Those who use AI to impersonate others, scale and spread harmful content, or create nonconsensual intimate images (NCII) are mostly not hacking or manipulating the AI generation tools they are using; they are using these tools within the allowed scope.
Two simple examples can be given:
The "fences" of AI tools can be cleverly bypassed with some prompts, and nothing prevents users from using ElevenLabs' AI voice cloning tool to highly realistically mimic the voices of colleagues or celebrities.
Civitai users can create AI-generated celebrity images, and although the platform has a policy prohibiting NCII, nothing prevents users from using open-source tools on GitHub (such as Automatic1111 or ComfyUI) to generate NCII on their own machines.
Posting this AI-generated content on Facebook may violate the platform's policies, but the act of generation itself does not violate the policies of the AI image generators they use.
Media: GenAI Abuse is Just the Tip of the Iceberg
Because Google's research material largely comes from media reports, this raises a question: Does this make the research conclusions biased by the media? After all, the media, as a spotlight, has its own bias in topic selection and reporting.
Those sensational events are more likely to be reported, which may lead to the dataset being biased towards specific types of abuse.
404 Media responded to this: Although the media can only report on events it can verify, one thing that can be certain is that there is a large amount of generative AI abuse that we are not yet aware of that has not been reported.
Even in the case of using AI to generate celebrity porn images mentioned above, which has been widely exposed by the media, there is still an issue of underreporting.
Firstly, this is because the topic is still taboo, and many publications are unwilling to report on it.
Secondly, it's because the media can only capture individual cases and cannot always focus on the overall event and its follow-up. An editor at 404 Media wrote:
Before I contacted Patreon officials for a response (after which the spokesperson closed his account), I wrote a report about a user profiting from NCII on Patreon, who had created 53,190 unconsented celebrity images. That report also mentioned two other NCII creators, and I later discovered others. The AI-generated nude images of Taylor Swift that went viral on Twitter were first shared in Telegram and 4chan communities, which were very active before and after that January report broke, and they have been posting NCII every day since then. Other reporters and I won't report on every single image and creator, because if we did, we wouldn't have time to do anything else.
When False Information is Rampant, It Becomes Easy to Blame AI
Let's make a projection: what will happen when the internet is flooded with AI-generated content?
The most direct consequence is that the chaotic situation on the internet will pose a huge challenge to people's ability to distinguish between truth and falsehood. We will fall into a state of frequent doubt, asking "Is this real?"
In the early days of the internet, a popular saying was, "On the internet, nobody knows you're a dog." Now this trend is intensifying. People are overwhelmed by false AI-generated content and are increasingly tired of dealing with it.
If left unresolved, the pollution of public data by AI-generated content could also hinder information retrieval and distort collective understanding of socio-political realities or scientific consensus.
Moreover, this could become a "shield" for some well-known figures. In certain situations, they could explain unfavorable evidence against them as AI-generated, easily shifting the burden of proof.
Google has played a role in fueling the proliferation of fake content brought by generative AI, and it can even be said to be the "originator". The bullet fired years ago has finally hit its own brow today.
References:
https://arxiv.org/abs/2406.13843
https://futurism.com/the-byte/google-researchers-paper-ai-internet