AGI stands for Artificial General Intelligence. This refers to the intelligence of a hypothetical computer program that, according to Wikipedia, is capable of “understanding or learning any intellectual task that a human can perform.” An alternative definition describes AGI as a highly autonomous AI system that surpasses human capabilities in solving most economically significant intellectual tasks.”
Experts are therefore calling for binding “red lines.” However, this discussion and research are anything but new—they have been ongoing for about 20 years—but current developments are bringing the issue ever more into the spotlight. Parallels can be drawn to the strict regulatory practices in the pharmaceutical industry, which are designed to minimize existential risks arising from a loss of technological control.
This is because there is currently a discrepancy among experts between the pace of investment and the development of safety guarantees. Stuart Russell, a professor at UC Berkeley and co-author of the standard textbook on AI, reports on internal assessments by leading industry representatives, according to which it might take an “event on the scale of the Chernobyl disaster” to prompt governments to take decisive action. This sober analysis makes it clear: technological capacity threatens to outpace human control. (He recently spoke before the European Parliament: Watch and listen here (39:09))
The Analogy with the Pharmaceutical Industry: Safety as a Prerequisite for Market Access
In the debate over AGI regulation, a radical paradigm shift is gaining increasing prominence. Russell, among others, is calling for Silicon Valley’s existing “move fast and break things” culture to be replaced by a rigorous approval process modeled after that of the pharmaceutical industry. The core of this call is that the burden of proof for safety must rest with the developer prior to release.
Current AI development largely follows the classic software model: products are released before they are fully mature and are stabilized through user feedback and subsequent updates. According to Russell and Thomas Larsen of the AI Futures Project , this approach is extremely risky when it comes to AGI systems. Once systems reach a level where they act autonomously and pursue their own goals, wrong decisions could be irreversible.
Russell argues that when it comes to other high-risk technologies—from bridges to airplanes to medications—we take government oversight for granted. The pharmaceutical industry, in particular, serves as an analytical model here:
• Ex-ante rather than ex-post: A pharmaceutical company must demonstrate safety in clinical trials before an active ingredient is approved. In the AI industry, the situation is currently the opposite: the risks often only become apparent to the public after the technology has been scaled up.
• The “black box” problem: With both complex medications and neural networks, we often do not understand every detail of how they work internally. The medical field responds to this with empirical safety assurances; the AI industry, on the other hand, often calls for regulatory exemptions so as not to hinder innovation.
• The Inadmissibility of the “Impossibility Argument”: The industry often argues that stricter safety requirements cannot currently be met from a technological standpoint. Russell exposes this as a logical fallacy: In the pharmaceutical industry, admitting that a drug is “too complex to prove its safety” would never lead to approval, but rather to a ban.
Technological “Red Lines” as New Clinical Phases
The “behavioral red lines” proposed by Russell can be understood as the equivalent of the phases of clinical trials. Accordingly, an AI model would have to demonstrate that it does not exhibit certain “toxic” behaviors under any circumstances:
• No autonomous self-replication: The system must not spread without human authorization.
• No deception: Strategic lying to achieve objectives must be technically impossible.
• Verifiable controllability: The ability to deactivate the system must be mathematically guaranteed.
Posted by Thomas Stuke, Chief Medical Marketing Officer at PEIX:
The alignment problem—that is, the question of how we can keep a system that is more intelligent than humans under control, and whether that is even possible—is perhaps the most pressing question of our time.
The major technology companies are all pursuing the goal of developing AGI or ASI (Artificial Superhuman Intelligence), and are steadily getting closer to that goal. Dario Amodei, CEO of Anthropic, believes AGI is possible within one to two years. Demis Hassabis, CEO of Google DeepMind, sees its realization as lying somewhat further in the future and expects it to take five to a maximum of ten years. Within the industry, it is generally accepted that this goal is achievable—the only question is when.
This assessment is also reflected in investment figures: In 2026 alone, four of the largest technology companies will collectively invest approximately 660 billion U.S. dollars. So the goal is clear. What remains completely unclear, however, is how such an AGI/ASI can remain controllable. Leading AI safety researchers such as Dr. Roman Yampolskiy and Eliezer Yudkowsky classify the alignment problem as unsolvable in the near future. Both even estimate the probability of failure at around 99 percent. In their assessment, there is currently no scientific paper that even begins to outline a convincing solution to the alignment problem. Both consider the consequences of an uncontrolled AGI/ASI to be catastrophic and explicitly include the extinction of humanity as an existential risk.
It is noteworthy that this existential risk is also recognized by the CEOs of major technology companies. This makes it all the more baffling that the regulatory framework called for by Stuart Russell and others is still lacking today. It is absolutely essential and becomes more urgent with every new, more powerful model released by OpenAI and others—because no one can predict today which model might mark the “point of no return.”
