Phish Tycoon: phishing using voice cloning
Craig Albuquerque, Melwina Albuquerque · Team Phish Tycoon
Submitted to AI capabilities and risks demo-jam. Sprint projects are early-stage work by participants, not Apart Research publications.
This project is a public service announcement highlighting the risks of voice cloning, an AI technology capable of creating synthetic voices nearly indistinguishable from real ones. The demo involves recording a user's voice during a phone call to generate a clone, which is then used in a simulated phishing call targeting the user's loved one.
Reviews
Nice idea! Showing the entire workflow of audio collection→voice clone→scam phone call is quite helpful. I do recommend using better TTS and voice cloning. As is, the clone doesn’t really sound like you and the robotic TTS decreases the impact of the demo.
I really liked it. I would suggest using a better TTS during the conversational part, as this would have allowed for better immersion and to project what it could look like to have a conversation with someone over the phone only to have your voice stolen.The voice cloning was quite impressive although a bit off. With some polish I feel like this demo could really shine.
Fantastic project! This is a near-term risk, presented in a visceral way! I love the conciseness of the website, and the reveal of the generated audio message. Next steps for this project would be polish - you could use a better voice for the AI conversation (e.g. an ElevenLabs voice), the voice cloning could maybe use additional tuning too, and with more time you could improve the website presentation overall. Great work on this!
Cite this project
@misc{albuquerque2024phish,
title = {{Phish Tycoon: phishing using voice cloning}},
author = {Craig Albuquerque and Melwina Albuquerque},
year = {2024},
month = aug,
note = {Submitted to AI capabilities and risks demo-jam, an Apart Research Sprint},
howpublished = {\url{https://apartresearch.com/sprints/projects/phish-tycoon-phishing-using-voice-cloning}},
url = {https://apartresearch.com/sprints/projects/phish-tycoon-phishing-using-voice-cloning}
}More from AI capabilities and risks demo-jam
- 1st place by peer reviewView project: Speculative Consequences of A.I. Misuse
Speculative Consequences of A.I. Misuse
Team S.C.A.M.
This project uses A.I. Technology to spoof an influential online figure, Mr Beast, and use him to promote a fake scam website we created.
- View project: Demonstrating LLM Code Injection Via Compromised Agent Tool
Demonstrating LLM Code Injection Via Compromised Agent Tool
This project demonstrates the vulnerability of AI-generated code to injection attacks by using a compromised multi-agent tool that generates Svelte code. The tool shows how malicious code can be injected during the code …
- View project: Misinformational AI-Generated Academic Papers
Misinformational AI-Generated Academic Papers
The Fake Academics
This study explores the potential for generative AI to produce convincing fake research papers, highlighting the growing threat of AI-generated misinformation. We demonstrate a semi-automated pipeline using large …