Gaatha Sarvaiya, an aspiring Indian law graduate in her early 20s, dreams of sharing her work on social media to build a public profile. However, the rampant rise of AI-powered deepfakes has cast a shadow over her ambitions. The fear that her images could be distorted into something grotesque or violating keeps her from posting online.
“The thought immediately pops in that, ‘OK, maybe it’s not safe. Maybe people can take our pictures and just do stuff with them,’” says Sarvaiya, who resides in Mumbai. This sentiment echoes across India, where women increasingly hesitate to engage online due to the potential misuse of their images.
The Rise of AI Harassment
India has emerged as a critical testing ground for AI tools, being the second-largest market for OpenAI globally. While the technology is embraced across various sectors, it has also become a tool for harassment. A recent report by the Rati Foundation, which operates a helpline for online abuse victims, highlights the growing use of AI to target women and gender minorities.
“It has become evident in the last three years that a vast majority of AI-generated content is used to target women and gender minorities,” states the report, co-authored with Tattle, an organization combating misinformation on Indian social media.
About 10% of the hundreds of cases reported to the helpline now involve AI-manipulated images, the report found.
High-Profile Cases and Their Impact
High-profile incidents have brought attention to this issue, such as the case of Bollywood singer Asha Bhosle, whose likeness was cloned and circulated on YouTube. Journalist Rana Ayyub faced a doxing campaign that led to the spread of deepfake sexualized images. These cases have sparked a broader societal conversation, with some public figures successfully fighting for legal rights over their images.
However, the impact on ordinary women, like Sarvaiya, is less discussed. “The consequence of facing online harassment is actually silencing yourself or becoming less active online,” explains Tarunima Prabhakar, co-founder of Tattle. Her organization conducted focus groups across India to study the societal effects of digital abuse.
“The emotion that we have identified is fatigue,” Prabhakar notes. “And the consequence of that fatigue is also that you just completely recede from these online spaces.”
Legal and Platform Challenges
In India, deepfakes operate in a legal grey area. While several laws could address online harassment, the process remains cumbersome. Sarvaiya argues that India’s legal system is ill-equipped to handle AI deepfakes, citing the extensive red tape involved in seeking justice.
The responsibility also lies with platforms like YouTube, Meta, Instagram, and WhatsApp, where such content is often shared. A report by Equality Now highlights the challenges in getting these companies to remove abusive content, describing the process as “opaque, resource-intensive, inconsistent and often ineffective.”
While some platforms have taken steps to limit the spread of nudify apps, Rati’s report cites instances of inadequate responses to online abuse. In one case, WhatsApp acted on an extortion incident, but only after the content had already spread widely.
“One of the abiding characteristics of AI-generated abuse is its tendency to multiply. It is created easily, shared widely and tends to resurface repeatedly,” Rati notes.
Looking Forward
Addressing AI-generated abuse requires greater transparency and cooperation from platforms. As Sarvaiya and women like her navigate this digital landscape, the need for robust legal frameworks and responsive platforms becomes increasingly urgent.
For now, the fear of deepfakes continues to silence many voices online, posing a significant challenge to digital empowerment in India. As technology evolves, so too must the measures to protect those most vulnerable to its misuse.
