San Francisco’s city attorney has demanded that Apple and Google remove a group of AI-powered “nudify” and face-swap apps from their mobile stores, escalating a long-running moderation problem into a direct legal test for the two biggest app platforms.
The letters, sent this week by City Attorney David Chiu and first detailed by WIRED, target 13 apps across the App Store and Google Play. The apps are described as face-swapping or image-editing tools, but investigators say they can be used to generate nonconsensual intimate deepfakes. Chiu’s office is asking the companies to remove the apps, cut business ties with the developers, and strengthen the review systems that allowed the tools to reach users in the first place.
The move matters because it shifts the pressure from individual app developers to the infrastructure around them. Apple and Google do not merely list apps. They review them, rank them in search results, process payments, collect developer fees, apply age ratings, and decide when developer accounts should be removed. San Francisco’s argument is that those control points make app stores part of the harm when abusive AI image tools are repeatedly surfaced, downloaded, and monetized through official channels.
Why app-store enforcement is the center of the fight
Nonconsensual deepfake imagery is often discussed as a model-safety or website problem, but mobile stores add a different layer of accountability. Apps distributed through the App Store and Google Play inherit trust from the platform. They can use built-in payment systems, appear in search results, buy ads, and reach younger users through ordinary phone workflows. That makes enforcement failures harder to treat as isolated developer misconduct.
According to WIRED, the city’s letters accuse Apple and Google of helping facilitate sales of explicit deepfake-generation services by continuing to host the apps and collect payment-related revenue. The San Francisco Chronicle reported that the city is also pointing to California law covering nonconsensual deepfake pornography, with potential civil penalties if companies fail to act after being notified.
The practical question is not whether Apple and Google have rules on paper. Both companies prohibit sexual abuse, harassment, and explicit content in their developer policies. The question is whether their review and enforcement systems can detect tools that present themselves as ordinary face-swap, avatar, beauty, or image-editing apps while preserving the ability to create abusive outputs once installed.
The dual-use problem is bigger than obvious “undress” branding
That disguise is central to the problem. A May preprint from researchers at Cornell University and Georgetown University, “Dual-Use AI Face Swap Apps Are Mostly Unsafe,” found that harmful capability is often hidden behind ordinary app-store presentation. The researchers identified 420 face-swap apps, manually tested 155 eligible apps, and found that 70 percent lacked technical safeguards preventing nude-image face swaps.
The finding helps explain why ordinary keyword bans are not enough. If an app never describes itself as a nudification tool, a store review process that relies heavily on app names, screenshots, descriptions, or stated terms may miss the underlying capability. Reviewers need to test what the app actually permits users to do, including whether it blocks prompts, templates, source images, or workflows that create synthetic intimate imagery without consent.
Earlier reporting has shown the scale of the issue. 404 Media, citing Tech Transparency Project research, reported in April that Apple’s and Google’s stores were not only hosting such apps but could push users toward them through search and advertising. WIRED reported that research this year identified around 100 apps across the two stores and estimated roughly 480 million downloads and about $120 million in combined revenue.
Apple and Google say they have removed offending apps
Apple and Google have both taken some action. WIRED reported that Google said it had removed hundreds of apps with nudifying features, including the five Android apps flagged by Chiu’s office, and had restricted related search terms. Apple said it had removed three of the apps flagged by the city attorney and was moving to terminate those developers’ accounts, while other developers were told to fix policy violations or risk removal.
Those responses show the enforcement gap rather than closing it. Removing named apps after an outside investigation is different from preventing repeat versions, clone apps, or lightly rebranded tools from returning under new names. For app stores, the durable fix would likely require more aggressive capability testing, developer-account linkage, payment-account enforcement, search-query monitoring, ad review, and age-rating checks.
Apple’s own 2025 App Store Transparency Report shows how large that enforcement machine already is. Apple said it reviewed more than 9.1 million app submissions, rejected more than 2 million, and removed 166,899 apps from the store that year. Sexual and pornographic content accounted for only 360 removals listed under guideline violations, a small number compared with the scale of general fraud, design, and spam enforcement. The nudify-app fight suggests that harmful AI image tools may not fit neatly into older app-review categories.
What stronger platform controls would look like
For Apple and Google, the next test is operational. A serious enforcement program would need to look beyond app descriptions and inspect real behavior: whether face-swap apps block nude reference images, whether their templates sexualize uploaded faces, whether they allow minors’ images, whether they advertise “uncensored” generations off-platform, and whether payment flows are tied to abusive output categories.
Search and recommendation systems also matter. If users typing obvious abuse-related terms still find face-swap tools, the platform has a discovery problem. If developers removed for deepfake abuse can relaunch under a new account, the platform has an identity and repeat-offender problem. If apps are rated for younger users while containing adult-generation paths, the platform has a classification problem.
The policy stakes extend beyond these 13 apps. Generative image tools are increasingly built into consumer editing apps, messaging features, camera filters, and social-media workflows. That makes the line between a benign creative tool and an abuse tool harder to patrol. Platforms will need rules that focus on capabilities, safeguards, and misuse patterns rather than only on whether an app openly markets itself as harmful.
What users and institutions should watch now
For parents, schools, and workplaces, the immediate lesson is that app-store availability should not be treated as proof of safety. AI face-swap and avatar apps should be reviewed with the same skepticism as other tools that can process personal photos. Schools handling deepfake harassment reports should preserve evidence, document app names and developer details, and use platform reporting channels quickly, because app listings and developer accounts can disappear once enforcement begins.
For app developers building legitimate image-editing tools, the pressure points are becoming clearer: strong content filters, abuse reporting, prompt and template restrictions, age-appropriate design, clear terms against nonconsensual imagery, and technical blocks against nude face swaps are moving from nice-to-have safeguards into basic platform-risk controls.
San Francisco’s letters will not solve the deepfake-abuse market on their own. But they mark a sharper phase in the debate. The question is no longer just whether abusive AI apps exist. It is whether the companies that control mobile distribution, payments, and discovery can be held responsible when those tools keep reaching users through trusted app stores.
