The intelligence paradox: AI may make markets less rational

Even the smartest algorithm has to operate within its limitations on risk and capital.

The evolution of artificial intelligence raises profound questions for financial markets.

The evolution of artificial intelligence raises profound questions for financial markets. Will human portfolio managers become obsolete as AI algorithms become smarter? Will markets become perfectly efficient and reflect the ultimate equilibrium, in which prices mirror economic reality without human distortion?

Economists’ longstanding debate about market efficiency sheds light on these questions. In the 1970s, Eugene Fama argued in his efficient-market hypothesis that asset prices reflect all available information, and it is therefore impossible for an investor to outperform the markets consistently. This thesis shaped modern finance, only to be countered a decade later by Robert Shiller, who argued that stock prices are far more volatile than would be expected if investors were making decisions based on strictly rational thinking. He proposed instead that human irrationality drives market bubbles, crashes and overall inefficiency. Despite their opposing views, Messrs Fama and Shiller were jointly awarded the Nobel Prize in 2013.

Our perspective aligns with Mr Shiller’s: Market participants’ irrational behaviour can cause market inefficiency. Yet market inefficiency isn’t solely the product of market participants’ acting irrational at times; different circumstances can compel even rational investors toward actions that collectively generate inefficiencies. Each player in the financial market is constrained by unique economic circumstances, and these economics drive even the smartest players to act in a way that isn’t necessarily efficient for the underlying asset or the market as a whole.

Example: A natural-gas producer hedging a production output may have a significantly lower optimal trading price than a utility safeguarding its end users’ contracts. Unless these two participants – which often bring significant volumes of the commodity to the market – trade at exactly the same time, their market actions can drive asset prices far from their fundamental value.

Hedge funds and other speculative entities may intervene, seeking to correct and benefit from the inefficiencies. Their actions, however, are also bound by economic constraints, such as limited capital or risk parameters. When they hit these limits, the speculative entities may be forced to unwind their positions, amplifying the price swing they had been working to dampen. As a result, while trying to solve one mispricing, they may introduce a set of new mispricings, perpetuating and even amplifying the cycle of price inefficiencies.

We saw this phenomenon with GameStop and other meme stocks – which gained popularity through social media – when risk limits drove short sellers to buy back the stocks they were shorting, driving those equities ever further from their fundamental value.

These inefficiencies aren’t only products of extreme market conditions but are recurring phenomena, even in stable economic periods. Nearly a century ago economist Nicholas Kaldor documented wild price swings for corn and hog markets. Today, speculative traders, including quantitative algorithms, frequently exit their positions while solving for a market price inefficiency. In fact, these market actions are often crucial parameters in their strategies to ensure a consistent volatility of their returns. Their actions can be individually logical and profitable over the long term but collectively disrupt the market’s march toward an efficient equilibrium.

AI could reduce dramatically or even eliminate behavioural irrationality, the inefficiencies born of pure human bias and emotion, not derived from economic constraints. Quantitative systems already outperform human traders in most scenarios when market conditions aren’t at their extremes. Unfettered by human biases, AI has the potential to uncover complex market patterns and relationships beyond human ability.

On the other hand, structural irrationality – the inefficiencies born from the inherent constraints and economic imperatives of each market participant, including AI – could persist or even grow. Even the most sophisticated AI algorithm must operate within the confines of risk parameters and capital limitations. For that reason, it can bring a new set of inefficiencies during its market participation.

We saw a glimpse of this during the flash crash of 2010, when algorithms reacted to other algorithms’ market actions that were triggered by their economic constraints.

AI models have exhibited incredible growth in intelligence in recent years. The current generation of AI – Google Gemini Ultra – has scored 90 per cent on a benchmark assessment called the Massive Multitask Language Understanding test. This is an astonishing leap from the AI models that were commonly scoring around 60 per cent two years ago, and it’s already in line with or higher than scores achieved by human experts. Other measurements of AI performance show it is rapidly reaching or surpassing human levels. AI will likely outperform human portfolio managers – even humans using AI tools – in due time. Yet this doesn’t guarantee overall market efficiency.

The increasing intelligence of individual market participants doesn’t necessarily translate to collective market wisdom. Instead, as the Red Queen told Alice, each participant may have to move faster and faster merely to stay in the same place.

As we enter this new era, we must grapple with the reality that the financial markets may continue to reflect their human creators in their irrationality – a paradox of intelligent inefficiencies.

Alen Brynjolfsson is CIO of Tiara Capital, a commodities trading fund using machine learning strategies. Erik Brynjolfsson is a professor at Stanford and co-chairman of Workhelix, a company that assesses machine-learning opportunities.

The Wall Street Journal