The perennial concern about AI, and in particular Advance General Intelligence (AGI), has been that it will become self-aware and take over, making choices for humans that are in the best interest of the machines rather than the people. It was HAL 9000 (HAL being one letter off of IBM) and later Skynet.

What I’ve experienced lately is a much more subtle but equally disturbing trend: the return of Code is Law in the form of customer service folks who can’t help you because “the system won’t let me do that.”

Choices have been made in the building of whatever software they are using, which determine what it allows and does not allow. The choices about what use cases to prioritize, which edge cases to account for, and what workarounds to allow are increasingly opaque and perhaps made by AI?

Don’t Contact Us

The customer service industry has long talked about “right channeling:” design choices that funnel customers to the channel the company wants to serve them in, which is typically the most cost effective. Reduce the number of calls to customer care, reduce the length of each of those calls, and save the company money.

Combine this with AI, and you get AI chat agents, who are perfectly capable of many of the things you might have needed to talk to a human for in the past, until they aren’t. At the moment you cross into edge case territory, or want to do something that a human would find perfectly reasonable but for whatever reason was not accounted for in the training of the bot, you have to escalate to a human.

The challenge is increasingly all the person on the other end of the line can do is access the same system—sometimes literally the same interface, other times a light reskin with a few more options—and tell you the same thing the web (or the phone tree automated system) was already telling you.

Is AI Making This Better or Worse?

In theory, as generative and agentic AI together make companies (allegedly) more productive, they would: a) be able to cover more use cases and have more powerful self-service options, and b) be able to empower the remaining human agents to handle the exceptions, giving them more capable systems.

If the bot can handle 90% of requests that previously required some human intervention, that ought to make it easier to handle the 10% that actually require human judgement. But that only works if the system (and the set of corporate policies/practices it encodes) actually empowers those humans to make smart choices.

But so far I’m not convinced. The trend seems to be in the opposite direction (less capable software, harder to reach humans, and less-empowered human agents across the board).

Have Your Agent Call My Agent

Will I need to create an AI agent that represents me against the agents deployed by companies I purchase goods and services from?

I’m reminded of the figure of the gestor, a kind of professional who exists to help individuals work through bureaucracy in some Latin American countries. I see a future in which AI agents of varying degrees of sophistication and leverage interact on behalf of humans with AI agents of varying degrees of sophistication and authority acting on behalf of companies.

I’m sure everything will be just fine. What could possibly go wrong?

I imagine a dystopia something like the love child of Brazil and The Matrix: with Smith, Brown, and Jones arguing with each other in gibberlink about whether Buttle or Tuttle was supposed to fill out form 27B stroke 6.

If there are enough choices, consumers may influence this by selecting providers who provide better service. Someday I hope we will return to competing on who can be most customer centric rather than just who can be most ruthlessly efficient.