AI hallucinates software packages and devs download them – even if potentially poisoned with malware
The idea here being that someone nefarious could ask models for code advice, make a note of imagined packages AI systems repeatedly recommend, and then implement those dependencies so that other programmers, when using the same models and getting the same suggestions, end up pulling in those libraries, which may be poisoned with malware.
Last year, through security firm Vulcan Cyber, Lanyado
published research detailing how one might pose a coding question to an AI model like ChatGPT and receive an answer that recommends the use of a software library, package, or framework that doesn't exist.
"When an attacker runs such a campaign, he will ask the model for packages that solve a coding problem, then he will receive some packages that don’t exist," Lanyado explained to
The Register. "He will upload malicious packages with the same names to the appropriate registries, and from that point on, all he has to do is wait for people to download the packages."