Adaptive Market Hypothesis: Yes Virginia, there is alpha in the market

I thought I'd expand on the Adaptive Market Hypothesis (AMH) I referred to in a previous post asking if technical analysis works.  AMH is a new, competing hypothesis to the Efficient Market Hypothesis (EMH).  EMH, I think, is best summarized in an old joke:

Two finance professors walking down the street. One spots a $100 bill lying on the pavement and points this out to his colleague, who says, "That cannot be a $100 bill or someone would have already picked it up." And so they continue walking.

The AMH version of the joke might add on the following line:

A quantum physics student picks up the $100 bill, invests it in a quantitative trading algorithm he's been working on and turns it into $1 Million in 10 years.

The jist of the argument is this:  EMH says excess markets profits are not to be found; AMH says they are.  Andrew Lo coined the term Adaptive Market Hypothesis in his paper just two years ago in 2004, but the basis for it extends back through many studies that attacked the EMH.  Adaptive markets were being described before Lo as well. 

According to Lo, EMH can be traced back to Samuelson, a nobel prize winning key player in the book Fortune's Formula and the Kelly Formula debate.  EMHers believe that market participants are rational, and all excess profits have been squeezed out, and thus prices are random and unpredictable.  The biggest extension to Samuelson's work was the concept of risk-adjusted returns:  you can actually have excess returns, but not without added risk.

The Efficient Market Hypothesis has survived so long because it is virtually impenetrable to argument or proof.  Point out an investor like Warren Buffett who is making excess profits and an EMH believer will likely retort with "he's taking excess risk" or "put a thousand monkeys in a room with typewriters and eventually one of them will type the Declaration of Independence."  It's mainly a thought exercise for those who choose to bang their heads against that particular brick wall.

Arguments against the EMH have emerged, however, from Behavioral Finance (which Lo summarizes in his paper), or from studies that show excess risk-adjusted market returns can be found (that were also not data-mined), such as described in the series "Can technical analysis beat the market?"  One of the key, overlooked arguments against the Efficient Market Hypothesis in my opinion is the Grossman-Stiglitz Paradox, which states:

If a market were informationally efficient, i.e., all relevant information is reflected in market prices, then no single agent would have sufficient incentive to acquire the information on which prices are based.

In other words, if a market is efficient, there is no incentive for it to remain so.  A corollary to this which I have long believed, but have never seen described anywhere can be stated as follows:

Markets become as efficient with respect to a given measure as the most marginally profitable player attempts to make them.

As an example, options on stocks become as efficient as the most marginally profitable arbitraguer makes them.  Why the most marginally profitable?  Because highly profitable arbitrageurs with an edge will exploit an opportunity for all it's worth, backed by their own growing bankroll or other people's money until it just marginally pays to do so.  In the process, they may make it unprofitable for less efficient players.  A quasi-equilibrium is reached whereby the most marginally efficient player survives and ekes out excess returns until market dynamics change.  Note that if there are still more profitable players, the equilibrium is likely in flux.  Another way of looking at this is:

The people who work so hard to make the market as efficient as it is must get paid excess risk-adjusted return to do so - they don't work for free.

The above scenario is exactly what happens in biological evolution, which is what the Adaptive Market Hypothesis is all about.  In fact many of Lo's theories grew out of work with Doyne Farmer from the Santa Fe Intute, who also happen to research Complex/Evolutionary Adaptive Systems like CASTrader.

Research into the AMH is obviously very young, but there are several implications of AMH that can be summarized as follows:

• Yes, you can make excess profits in the financial markets.  Opportunities for arbitrage exist (Grossman-Stiglitz Paradox), and may appear and disappear.  Rather than a trend toward ever increasing efficiency as EMH implies, efficiency ebbs and flows with market euphoria, panics and crashes.
• For awhile, anyway. Risk/reward relations are unlikely to be stable over time.  Opportunities appear and disappear.  Risk premiums are shaped by the participants in an evolutionary manner based on their past experiences.  (People who have never experienced a bear market may demand less risk premium)
• Thus, investment strategies wax and wane, too.  Methods that worked before may fade, only to surface again when market conditions dictate. A graph from Lo's paper (at left) shows the serial correlation of the S&P Composite from 1871 to 2003.  Think of the blue as a measure of efficiency over time: how predictable are future prices based on past prices.  If the market were efficient, you would see no blue, and if the market is more efficient than it has been in the past, it's not by much!
• Innovation is the key to excess returns.  An investor must adapt to the changing conditions to keep their excess returns flowing.
• Changing conditions can make you extinct in the markets.  Think dinosaurs, think LTCM

One thing that I think Lo has completely overlooked, which surprises me, is the role of cooperation in biological evolution, complex adaptive systems, and by extension, the Adaptive Market Hypothesis.  If "survival of the fittest" is the dark side of evolution, then cooperation is the bright side.  Biological species and agents in complex adaptive systems show time and time again the advantage of banding together achieves.  Species and agents often cooperate in ways which assures their long-term survival.  Financial markets are constantly adopting new innovations.  Weather derivatives and ETFs are new, recent ways to invest that change the landscape.  Traders of weather derivatives can cooperate with investors who buy weather derivatives to hedge all sorts of weather related risk.  Like the flower offers the bee pollen in return for helping it reproduce, weather derivative buyers can unknowingly make it profitable for traders to create a liquid market. Cooperation creates new opportunities for capturing alpha for diverse species.

Another point that Lo talks about, but really doesn't drive home as an implication of AMH is how dominant agents can implode suddenly, being too "adapted" for their own good.  Dinosaurs grow too big for their own good and when the environment changes, they go extinct.  Hedge funds like LTCM can become highly leveraged, making a lot of money until they implode.  Ditto for Amaranth.  In my opinion, the Kelly Formula is the "secret sauce" that can optimize the chance for survival.  Get greedier than indicated by the Kelly Formula, and you'll eventually blow up; bet less than the Kelly Formula, and you'll be outcompeted by people who do.  The Kelly Criterion is the equilibrium survival state of the AMH.

Further reading:

Andre Lo's original paper (very readable) and a follow-up paper  Update: a whole bunch of other papers by Lo

A CXO Advisory summary of the paper

Blog posts:

Econophysics

Perfectly Reasonable Deviations

Mahalanlobis

Media:

Financial Planning

Business Week

Academic research in support of AMH:

The adaptive markets hypothesis: evidence from the foreign exchange market

Other:

Pensions at Work

Comments

[Mike]

Nice post. Robert Hagstrom's book 'Investing: The Last Liberal Art' did an excellent job of conveying these ideas (especially regarding the evolution of strategies). It is a beautiful and immensely readable introduction to these topics published in 2000.

[Alan J]

Thanks, Mike. I've read Hagstrom's Buffett book, but had no idea he was writing about this. It's on my reading list.

[viagra online]

Traders of weather derivatives can cooperate with investors who buy weather derivatives to hedge all sorts of weather related risk. Like the flower offers the bee pollen in return for helping it reproduce, weather derivative buyers can unknowingly make it profitable for traders to create a liquid market. Cooperation creates new opportunities for capturing alpha for diverse species.

[generic viagra]

Traders of weather derivatives can cooperate with investors who buy weather derivatives to hedge all sorts of weather related risk. Like the flower offers the bee pollen in return for helping it reproduce, weather derivative buyers can unknowingly make it profitable for traders to create a liquid market. Cooperation creates new opportunities for capturing alpha for diverse species.