Dec 28, 2025 06:05 PM
https://scienceblog.com/sciencechina/202...erability/
INTRO: An email lands in your inbox. The tone is familiar, the phrasing matches your manager’s style down to the occasional comma splice. It asks you to review a document on a shared drive. You click. By the time you realize the sender wasn’t your boss, your password is already logged on a server halfway across the world. This is high-fidelity phishing, and it runs on the same technology that helps you draft presentations.
Researchers from Shanghai Jiao Tong University and East China Normal University have mapped the security landscape around large language models in a review published in Frontiers of Engineering Management. After screening more than 10,000 documents and analyzing 73 key works, the team outlines how fluent text generation has become a dual-use tool: helpful for drafting emails, dangerous when weaponized for impersonation, phishing, and misinformation at scale.
The paper frames the problem across two fronts. One is misuse, where bad actors exploit the model’s fluency to automate fraud. The other is direct attacks on the model itself, including techniques that extract private training data, poison datasets, or manipulate outputs through what the researchers call prompt injection. That last method works like slipping hidden instructions into a conversation, causing the model to ignore its safety rules and follow the attacker’s script instead... (MORE - details)
INTRO: An email lands in your inbox. The tone is familiar, the phrasing matches your manager’s style down to the occasional comma splice. It asks you to review a document on a shared drive. You click. By the time you realize the sender wasn’t your boss, your password is already logged on a server halfway across the world. This is high-fidelity phishing, and it runs on the same technology that helps you draft presentations.
Researchers from Shanghai Jiao Tong University and East China Normal University have mapped the security landscape around large language models in a review published in Frontiers of Engineering Management. After screening more than 10,000 documents and analyzing 73 key works, the team outlines how fluent text generation has become a dual-use tool: helpful for drafting emails, dangerous when weaponized for impersonation, phishing, and misinformation at scale.
The paper frames the problem across two fronts. One is misuse, where bad actors exploit the model’s fluency to automate fraud. The other is direct attacks on the model itself, including techniques that extract private training data, poison datasets, or manipulate outputs through what the researchers call prompt injection. That last method works like slipping hidden instructions into a conversation, causing the model to ignore its safety rules and follow the attacker’s script instead... (MORE - details)