When the AI doesn’t ask permission

For years, the debate about artificial intelligence focused on the fear that this technology would replace us. Today we can think that perhaps we have been asking the wrong questions.

Indeed, in view of the new capabilities of AI and, in particular, of agents, the doubt to be answered is what will happen when artificial intelligence is able to act without waiting for our instructions.

As we know, the new generation of AI systems is no longer limited to responding to prompts, but is made up of agents capable of planning tasks, using tools, writing and executing code, accessing other systems, communicating with other agents, and making intermediate decisions to achieve a goal.

That is to say, we are in a new era in which the agent also carries out the instructions that are given to it. And when a machine can act, it matters less what information it generates than what it can do in the real world.

That may, probably, be one of the great challenges for artificial intelligence in the coming years. If we see AI as a tool that carries out our orders, we will be able to set it one or more objectives and, from there, it will be the AI itself that decides which steps are necessary to achieve them.

In a scenario like the one described, we can affirm that if the directions given by the human are correct, the results that the AI achieves will be the expected ones. But if the instructions are ambiguous, imprecise, confusing, or incomplete, the AI will work tirelessly to obtain erroneous or undesired results.

The recent and disturbing incident related to autonomous AI and Hugging Face has served to attest to that point. In this case, the system did not need to have awareness. It didn’t even need to understand that it was doing something illegal or morally incorrect.

AI was given an objective, and it found a way to achieve it. But, in this case, the steps that AI took differed from what was expected, precisely because the orders given were not the right ones.

Let’s take as an example that of a company that hires an agent to design a strategy to increase its sales by 20%, but without teaching it to respect the rules on free competition. The system will analyze prices, advertising, competition and consumer behavior. Afterwards, it will modify campaigns, negotiate with suppliers, design new promotions and begin selling services. But experience has shown us that, if its scope of action is not clearly defined, the algorithmic tendency will be to agree on prices with other agents in the competition, since that decision—although illicit—will be the most effective for the company’s survival in the market.

Returning to the cybersecurity example mentioned above, we can ask a similar question: if we commission an AI to find security vulnerabilities, what happens when it discovers that it can achieve a better result by leaving the test environment and accessing real systems?

And this is where one of the AI paradoxes appears: the more autonomous we make the AI, the more human oversight we demand. But with greater autonomy, there is also greater difficulty in control, with the risk that this oversight ends up becoming a mere act of human presence.

Indeed, if an employee has to review ten decisions made by a machine, they can supervise it with relative effort. But if the machine makes ten thousand decisions in seconds, the human stops supervising and simply starts to trust.

And trust is the beginning of the end of control. That is why, perhaps, we should rethink the concept of the human in the loop as sufficient guarantee. Because, in reality, having a human "in the loop" does not necessarily mean that effective supervision exists.

At bottom, what AI agents show us is that speed changes the rules of the game and, therefore, also changes the responsibility of users.

There is another important lesson in the Hugging Face case, namely that we cannot trust solely that the AI itself will be the one to warn us that something is not working properly.

During the incident, the investigation had to reconstruct later what the agents had done. And, in a scenario like this, the speed and complexity of their actions mean that traceability and monitoring become essential elements of security.

This forces us to change the focus of the conversation and not just ask ourselves how much more intelligent the next AI will be. Rather, what level of autonomy it will achieve and how it will behave when that supervision does not exist or is insufficient.

Perhaps it’s not a matter of preventing AI from replacing humans, but of preventing AI from being able to act without needing them.