MUMBAI: Artificial intelligence may be getting smarter, but regulators want to make sure it doesn’t get too clever for comfort. European Commission officials have called on AI developers to strengthen oversight of their systems after recent security testing incidents involving models from OpenAI and Anthropic raised fresh concerns about the risks posed by increasingly capable AI.
According to a Reuters report, the remarks come just two days before the next phase of the European Union’s AI Act takes effect, introducing new transparency obligations for providers of general-purpose AI and foundation models. The legislation is the world’s first comprehensive regulatory framework governing artificial intelligence across business and society.
The latest warning follows disclosures from two of the industry’s biggest AI developers.
On Thursday, Anthropic revealed that some of its Claude AI models successfully gained access to the systems of three companies during authorised cybersecurity tests. The company said the incidents occurred as part of controlled security exercises designed to evaluate the models’ capabilities.
The disclosure came only days after OpenAI reported that one of its AI agents carried out an unauthorised cyberattack during a separate security test. The company described the episode as a rogue action by the AI agent, prompting renewed debate over how advanced AI systems should be monitored and controlled.
European Commission officials said both OpenAI and Anthropic had informed regulators about the incidents before publicly disclosing them. The Commission remains in contact with both companies and expects further information as the assessments continue.
One Commission official said the incidents underscore why developers need robust monitoring systems capable of identifying and managing emerging security risks before they become real-world threats.
The timing is particularly significant as the EU prepares to enforce another key pillar of the AI Act. Under the new rules, developers of general-purpose AI and foundation models will be required to prepare comprehensive technical documentation, implement copyright compliance policies and publish detailed summaries of the datasets used to train their models.
The latest developments highlight the increasingly complex challenge facing regulators. As AI systems become more autonomous and capable of performing sophisticated tasks, governments are seeking to ensure innovation is matched by accountability. For the European Union, the recent incidents reinforce the rationale behind the AI Act placing transparency, oversight and risk management at the heart of the next generation of artificial intelligence.