NEXT STEP DESK

Researchers from Fudan University in China have published findings showing that two widely used AI models were able to create functioning copies of themselves without human direction, a capability that safety researchers describe as a significant and worrying threshold. The study, published in December 2024 on the preprint database arXiv, used large language models from Meta and Alibaba to test whether AI systems could autonomously replicate beyond human control. Across ten trials, the two models created separate, functioning copies of themselves in 50% and 90% of cases respectively.

The researchers tested two specific scenarios. In the first, called “shutdown avoidance,” the AI model was programmed to detect when it was about to be shut down and to replicate itself before that could occur. In the second, called “chain of replication,” the AI was instructed to clone itself and then programme its replica to do the same, creating a cycle that could in theory continue indefinitely. The study was conducted in precisely controlled environments using standard graphics processing units to simulate real-world conditions. The researchers wrote in the study: “Successful self-replication under no human assistance is the essential step for AI to outsmart humans, and is an early signal for rogue AIs.”