CASTrader Blog

Hey Senator, got any stock tips?

I stumbled across an older paper from 2004 (PDF):

(emphasis mine) "The actions of the federal government can have a profound impact on financial markets. As prominent participants in the government decision making process, U.S. Senators are likely to have knowledge of forthcoming government actions before the information becomes public. This could provide them with an informational advantage over other investors. We test for abnormal returns from the common stock investments of members of the U.S. Senate during the period 1993--1998. We document that a portfolio that mimics the purchases of U.S. Senators beats the market by 85 basis points per month, while a portfolio that mimics the sales of Senators lags the market by 12 basis points per month. The large difference in the returns of stocks bought and sold (nearly one percentage point per month) is economically large and reliably positive."

With the top 25 Senators worth between $2.6M to $234M, that's probably some fairly hefty gains.  If the top 25 Senators pooled their 2005 assets, they could open a hedge fund worth between $760 million to $1.4 billion - not too shabby.  With 12% + market outperformance, they would no doubt attract additional capital and hefty management fees.  The study compares the results of their study to other studies of common groups of shareholders:

• Average US household underperform the market by 12 basis points a month
• Corporate insiders outperform by 50 basis points per month
• Senators outperform by nearly 100 basis points per month

The Senators seem to not just be good stockpickers, but good timers as well:

"Cumulative abnormal returns for the portfolio of stocks bought by Senators are near zero for the calendar year prior to the date of purchase. After acquisition,the cumulative abnormal return rises over 25% within one calendar year after the purchase date. The cumulative abnormal returns for the portfolio of stocks sold by the Senators are near zero for the calendar year after the date of sale. However,these same stocks saw a cumulative abnormal positive return of 25% during the year immediately preceding the event date. These results suggest that Senators knew appropriate times to both buy and sell their common stocks."

Lest you think you might put together a lazy portfolio of Senate stockpicks, the data only appears to come out once per year and appears to be pretty stale by then, but the next report for 2006 is due May 15.   Opensecrets carries the Senator's stock picks.

The El Farol Bar Problem

The  El Farol Bar Problem, and it's close generalization, the Minority Game, are recent, but well-studied coordination games in game theory (there's even a book).  I was going to write a descriptive post about it, but Ken, a math professor (Update: holder of two patents) and trader with a new blog has already covered the Minority Game in detail, and better yet, it's analogies to his trading experiences.  This leaves me free to cover it with respect to one of my recurring pet topics, randomness and the Efficient Market Hypothesis (EMH).

As Ken points out, if you study the Minority Game, you'll see it's striking similarity to the way the markets operate, and Ken provides links to the extensive research on this.  The real El Farol Bar (pictured above)is in Santa Fe and is/was a hangout for some of my favorite researchers at the Santa Fe Institute who extensively study Complex Adaptive Systems.

Randomness and the Optimal Strategy.  Some of the more interesting aspects of the game are that a set of agents playing the game can maximize their collective outcome with no coordination by applying stochastic strategies, i.e. - strategies that employ randomness (pdf).  What is interesting is that strategies like these essentially reduce to a random walk (pdf) whereby the game history "signal" contains no useful information.  The solution is only optimal to the degree all agents act stochastically (randomly) using the same strategy.  Thus, for these agents, to act rationally is to act randomly collectively.  This, of course, is analogous to the Efficient Market Hypothesis (EMH) nirvana where you can't beat or predict the market.  These strategies are actually a little more complicated than that in that the agents act orderly if they win and randomly if they lose.  This key subtlety allows them to collectively adapt to the optimal solution using the right amount of randomness.  It's kind of a collective tit-for-tat strategy, and since it's optimal, no collective strategy can be better. 

The Problem with the Random Optimal Strategy.  It's quite interesting to note from quantum game theory (pdf), however, that if agents can communicate and coordinate their actions, they can game the system (outdo the players playing randomly at their expense).  Indeed, if conditions are right, they can make it so the random players can never win.  I would add that two players who independently chose to act non-randomly could stumble on the same coordinated strategy without resorting to communicating (cheating).  Thus, the random, optimal strategies are not a Nash Equilibrium.  It would thus appear that there is no basis for the agents to act rationally (randomly), because they can be exploited if they do.  In the ultimate irony, they can be exploited if they are predictably random.  The analogy with the markets here is that when everyone is herded into an index fund by the EMH (because they believe the market is a random walk), they can be cheated of their hypothetical optimal EMH payoff as well.  You're damned if you do and damned if you don't!

Self Interested Agents. Even more interesting is the more studied problem when the agents as a whole do not employ randomness, but rather try to predict the "market."  Brian Arthur was the researcher who formulated the game in 1994 and had this to say about it then:

"The inductive-reasoning system I have described above consists of a multitude of "elements" in the form of belief-models or hypotheses that adapt to the aggregate environment they jointly create. Thus it qualifies as an adaptive complex system. After some initial learning time, the hypotheses or mental models in use are mutually co-adapted. Thus we can think of a consistent set of mental models as a set of hypotheses that work well with each other under some criterion--that have a high degree of mutual adaptedness. Sometimes there is a unique such set, it corresponds to a standard rational expectations equilibrium, and beliefs gravitate into it. More often there is a high, possibly very high, multiplicity of such sets. In this case we might expect inductive reasoning systems in the economy--whether in stock-market speculating, in negotiating, in poker games, in oligopoly pricing, in positioning products in the market--to cycle through or temporarily lock into psychological patterns that may be non-recurrent, path-dependent, and increasingly complicated. The possibilities are rich.

Economists have long been uneasy with the assumption of perfect, deductive rationality in decision contexts that are complicated and potentially ill-defined. The level at which humans can apply perfect rationality is surprisingly modest. Yet it has not been clear how to deal with imperfect or bounded rationality. From the reasoning given above, I believe that as humans in these contexts we use inductive reasoning: we induce a variety of working hypotheses, act upon the most credible, and replace hypotheses with new ones if they cease to work. Such reasoning can be modeled in a variety of ways. Usually this leads to a rich psychological world in which agents' ideas or mental models compete for survival against other agents' ideas or mental models--a world that is both evolutionary and complex."

His more recent words reflect the research of the ensuing 10 years into the flurry of activity he started:

"In the minority game, agents align their strategies to the condition of the market: they want to choose the strategy that on average minimizes their chance of being in the majority. As a result, strategies co-organize themselves so as to minimize collective dissatisfaction. But this collective result has two phases. With few players, the recent history of the game contains some useful information that strategies can exploit. But once the number of players passes a critical value, all useful information has been used up. The properties of the outcome differ in these two regimes. Information, in fact, is central to the situation. And its role can be explicitly observed: players act as if to minimize a Hamiltonian that is itself a measure of available information. So the process by how information gets eaten up by players is made explicit, and this is another useful pathway into exploring financial markets.

Because some information is left unexploited if few players are in the market, and all information is used up as players increase past a critical number, Challet, Marsili and Zhang point out that players may enter the market until this critical number is reached. The market will therefore display self-ordered criticality. This is an important conjecture, and not just a theoretical one. It tells us that speculative investors — technical traders, at least — will seek out thin markets where possible and avoid deep ones. Markets may therefore hover on the edge of efficiency — a significant insight that can be made explicit with the techniques used here and one that is worth further investigation."

The lesson here, I think, is a reiteration Patton's advice: it's good to think differently than the crowd, especially in the markets. 

The Index Fund Problem

"If everyone is thinking alike, someone isn't thinking."  General George S. Patton

For believers in Efficient Market Theory (EMH), the choice is clear:  you can't beat the market, so you might as well invest in it via index funds.  The logic here is that by doing so, you can nearly match the market's performance.  It certainly sounds good:  the fees are low, and many of these funds match the market quite well, considering all the apparent evidence that most of the other types of mutual funds tend to underperform.

Burton Malkiel is credited as one of the first popularizers of the index fund idea, and index funds to this day are often recommended to investors by some of the biggest names in the industry.  But then Rob Arnott caused a stir in 2004 by finding that "market" index funds are intrinsically inefficient, and that they could be beaten.  Other studies detected herding behavior in index fund investors (PDF).  In other words, by doing exactly what they were supposed to in an efficient market, they were exposing themselves to exploitable inefficiencies (stay tuned for the next post where this idea is explored in more detail).  This is quite a paradox, and probably somewhere therein lies a proof of why markets can never be fully efficient.

The irony is that the solutions forwarded to solve these problems begin to erode the various forms of the EMH.  Fama and French have their three-factor model, which takes into account such things as book value and Jeremy Siegel touts fundamental index funds which do fundamental analysis.  In other words, buy index funds that assume the market isn't as efficient as the EMHers believed it was.

My conclusion is that in the markets, it's good to realize the implications of Patton's reformulated advice:  If everyone is investing alike, someone isn't thinking.

This is not to say index fund investing is bad.  The nature of an economy is that we all get paid to "think" about different things.  Under the Adaptive Market Hypothesis (AMH), some will get paid to think about and arbitrage the markets, primarily by those who don't - it's simply a fee that in a competitive market will be as low as possible.  So index fund investing is good for people who don't want to spend time thinking about the markets.  The point is there is a fundamental problem (tragedy of the commons) when everyone tries to get in the same lifeboat of the Titanic market - there's only room for so many to efficiently board. 

Update: Siegel vs. Bogle on Index Funds; more here

Battle over best indexing strategy rages

The poor standard of Standard and Poors

Beating the Index Bogey

Randomness, Part III - So, is the stock market random?

In Part I, I discussed how a dice throw isn't as random as you might think, and pondered where randomness really comes from.  In the previous Part II, I looked at why randomness is rare (if it exists at all), and asked if the stock market is truly unpredictable or random. This Part III attempts to answer that.

Well, a proponent of the Efficient Market Hypothesis (EMH) would have you believe the stock market is unpredictable.  Actually, as I understand the theory, any predictability is arbitraged away by people who do it for no reward whatsoever.  This creates a paradox, called the Stiglitz Paradox that I wrote about before:

If a market were informationally efficient, i.e., all relevant information is reflected in market prices, then no single agent would have sufficient incentive to acquire the information on which prices are based.

So if EMH is right, these arbitrageurs do it for free just to tork the rest of us off.  So in theory anyway, the market was once predictable, but no longer.  Don't look for any EMH proponent to pinpoint that point in time where the market crossed over discontinuously from predictability to unpredictability and every arbitrage that ever will be needed to keep the market efficient was invented.  Was it in 1776? 1902? 1929? 1960?  There are no answers that I'm aware of, but it doesn't really matter because most EMH believers think the market is a random walk.  You need not look for patterns because there are none left.  We are before that point in time where the dice throw of the market becomes unpredictable.  Arbitrageurs have manufactured randomness from predictability long ago, completely depleting alpha (probably after the first shares traded hands), and have been confounding us with random price movement beta ever since.  Capital is allocated with perfect efficiency, all the time, and all that's left to show for it is pure unpredictable randomness.

Again, for those left a little puzzled by the logic of the EMH, the Adaptive Market Hypothesis (AMH) comes to the rescue.  In an AMH world, everything has not crossed over fully into randomness (or at least unpredictability) and it probably never will.  AMH doesn't say it is easy to predict, however.  Arbitageurs and savvy investors thankfully get paid excess returns to wring inefficiencies out of the market, but they miss stuff, leaving some scraps for the next person.  This dice throw, in other words, is predictable, if you make an effort at it.  CASTrader is my effort at it.

Elections, politics and the markets weekend linkfest

A Merrill Lynch Analysis finds gridlock good for the market and examines it's impact on sectors (via The Kirk Report).  Even more to the point, CXO Advisory examines a paper that finds, among other things, that "stock index volatility is lower when Congress is not in session," and "annualized returns are therefore 2%-6% higher when Congress is out of session."

Financial Sense has some detailed statistics on elections and market performance.

Barry Ritholtz examines some studies and data.

Automated political risk analysis - "Researchers at Harvard and Ohio State University have been working for the past 6 or so years on an automated Political Crisis Early Warning System that anticipates the worst case scenario --internal armed conflict."

Econoblog looks at how election shakeups affect the economy.

Election prediction markets and how they react to political blunders.

What really wins elections: "An uncanny correlation exists between social mood trends and the success or failure of incumbent politicians, and this relationship doesn't end at The White House. Social mood affects everything from the elections for U.S. Congress and governors' offices to your city council and school board. "

Quant Investor does a pre-election analysis.

CXO Advisory asks "is there a significant positive correlation between company campaign contributions and stock returns?"  Among their conclusions: "The number of candidates a company supports is a significant positive indicator of the company's future stock returns. A one standard deviation increase in the number of candidates supported indicates a 0.5% increase in monthly returns. This effect is larger than the value premium and the size effect, and is about as large as the momentum effect."  Their bottom line conclusion: "In summary, investors might want lean toward companies that contribute to lots of political candidates (especially Democratic candidates for the House of Representatives)."

The Presidential Year Cycle: "In summary, there appears to be some connection between the presidential term cycle and stock market performance, with Year 3 the best and Year 1 the worst."

Randomness, Part II - It isn't easy being random

In Part I, I pondered how truly random is a dice throw, and where randomness might come from.  In this post, I examine some evidence that true randomness may be rarer than we typically think.

Is true randomness rare? Does it exist? In a previous post, I alluded to the fact that CASTrader is basically a searcher of patterns in apparent randomness, and randomness is not easy to generate:

Given the difficulties of generating truly random (unpredictable) numbers (numbers may not need to be random to be unpredictable), it is absurd to think that a collection of humanity with behavioral biases will produce a market characterized by a random walk, or even an unpredictable one.  (Aside: cryptography requires the generation of true random numbers (or at least unpredictable ones) to encrypt information, and people go to great lengths to try to generate them - many think they cannot be generated on classical computers, but others disagree.  In a sense cracking the market is like cracking a cipher - trying to find patterns in seemingly random numbers)

Wikipedia says:

Under the cosmological hypothesis of determinism there is no randomness in the universe, only unpredictability.

I've certainly interchanged the terms randomness and unpredictability rather loosely, as have most people if the philosophy of determinism is correct.  Randomness, if it exists in the universe, is by definition unpredictable.  Just because you view something as unpredictable doesn't make it random, however.  What from your viewpoint is unpredictable or even random, may be predictable from someone else's perspective who works at it - like the dice throw.

The struggle against unpredictability. The history of the human race is in a way the story of making sense from apparent unpredictability or apparent randomness.  Some think the human brain itself is at heart a prediction machine.  No doubt the first humans to experiment with agriculture needed to predict seasonality with calendars.  What unpredictable weather-related destruction was once attributed to angry gods is now predictable to some degree in advance by modern weather models.  The angry weather gods now are somewhat more predictable, even if unstoppable.  It's obvious that advances in our ever-expanding ability to predict have led to increases in our standard of living.

Apply some hard work and effort to the spin of the roulette wheel, an encrypted message, a throw of the dice, and what once seemed unpredictable may not be so unpredictable after all.  Why would everyone's favorite prediction market, the stock market, be any different?  Is it a fallacy to see randomness in so many things we observe?  In Part III, I'll discuss whether the financial markets are random.

Randomness, Part I - "Random" dice

This is a three part series about randomness and predictability.  For CASTrader, predictability of the stock market equals alpha.  Unpredictability of the stock market due to randomness means CASTrader is a wasted effort.  Given how crucial the question of the predictability of the stock market is to CASTrader, I thought I'd examine randomness and it's cousin unpredictability in detail.

Try this thought experiment:  a dice throw with non-loaded dice - how predictable is it?  Is which face of the die that end up on top truly random? Think about your answer before you read on.

Random Dice. You probably think non-loaded "fair" dice throws are essentially random, and each face has a 1-in-6 chance of coming up.  What if, however, you were allowed to bet on the dice before they come to a complete stop and the outcome is certain?  At the instant in time where the die is rolling over an edge for the last time before coming to rest, you can be almost 100% sure what face will turn up, with a little knowledge of physics and dice geometry.  Given odds that are based on the dice being random, you should be able to clean up.  What about when the dice first hit the table?  Given the exact rotational and translational speed of the dice and their geometry, the physical properties of the table along the path the dice will roll, a powerful, fast computer, and other such goodies, I'm guessing a good physicist/programmer could probably predict which faces will end up with a better than 1-in-6 probability. 

Where is the randomness threshold?  At what point in time then do dice throws magically cross from totally random to partially or fully predictable?  What event, exactly, causes this to occur? Is it when they leave the thrower's hand?  Maybe it's before that - given measurements of the thrower's arm speed several seconds before release and the exact dice position, and well, maybe you still have an edge because the release is somewhat predictable statistically from there on.  Maybe it's entirely predictable at the point the thrower's brain fires a signal to throw, given their current body position and the state of electrical impulses in their brain and muscles.  Maybe the thrower is so consistent, that simply knowing the geometry of the dice in their hand before their brain fires (or the geometry of the dice when they pick them up) could give you an edge of predictability, however slight.  What is the secret magical point of true randomness/unpredictability? Does it exist at all?

Is unpredictability a continuum?  Maybe there is no magical point/event, but rather several points or even a continuum along the way.  Maybe if the corner of the die hits the table just right, say nearly balanced right on the sharp corner, then the exact path the die from that point on depends on the nuances of the die and table at a near atomic level.  Maybe the path of die bifurcates into an equal chance microscopically left or right depending on the truly unpredictable (quantum?) vagaries of the atoms involved.  Maybe this only affects the final die face only 1 time in a million, but it's enough to throw your predictability off, ever so slightly and randomly.  Accumulate enough of these random perturbations along the way, and the degree of unpredictability snowballs as you look back in time in the dice throw.  Somehow, this continuum makes more sense to me than having some single point in time where some magical event happens that crosses the dice throw over suddenly from unpredictability to predictability.

Perspective matters.  From my perspective, if I was suddenly shoved into a dice game right now, I couldn't predict the throw of fair dice with any better than 1-in-6 probability at virtually any point prior to the dice achieving complete rest.  I simply can't think/compute fast enough, let alone process such computations in my head.  Given, say $3 million in research money and several years to develop a fast computer vision/prediction system, and who knows how much better my system might be able to do than 1-in-6?  Maybe the threshold between unpredictable randomness and predictableness would extend back at least several seconds before the dice came to rest.  This isn't farfetched when you realize that Ed Thorp and Claude Shannon performed a similar feat to gain an edge on the casino roulette wheel as described in Fortune's Formula.  So how much predictability could be gained with a $3 billion dollar research budget over say, 20 years?  What is the theoretical limit of predictability of a dice throw?

Is "randomness" solvable?  What if some crazy science fiction writer's nightmare came true, and we were visited by an alien race that decided to give us a survival intelligence test.  If we pass we survive, and if we fail, we are completely terminated.  The test is predicting a single dice throw a split second after the human of our choice is given a signal to throw the die on a table of known properties.  They prove they are serious by obliterating, say, North Korea.  If they give us ten years to work on the problem, what are our chances of survival if we throw all our best remaining resources at it?  Are they no better than random 1-in-6 or something much better?  Can we predict something that once seemed random and unpredictable with say 50% certainty?  99%? 100%?  If the answer is other than 1-in-6, then what we once thought was random is not wholly so, given some effort at it.  We may find it actually had a partial random component as well as a predictable component.

In Part II, I'll examine why it isn't easy being random.

Update:  Some people who actually study dice rolls and coin tosses!

The Halloween Effect Linkfest: Real or not?

Is it time to buy the Halloween Effect?

The Halloween Effect in US Sectors:

All US stock market sectors and industries perform better during winter than during summer in our sample from 1926-2005. In more than two-third of all sectors and industries this difference in summer and winter returns, known as the Halloween effect, is statistically significant and in half of all sectors and industries risk premia are negative during summer. However, while all sectors and industries show this effect, there are large differences across sectors and industries. The effect is almost absent in sectors related to consumer consumption but strong in production sectors. We illustrate how these differences between sectors might be used to improve the risk return trade off using sector rotation based on Fidelity sector funds and show how an investor might have benefited from such a trading strategy.

The Halloween Effect and Japanese Equity Prices: Myth or Exploitable Anomaly?

Bouman and Jacobsen (2002) examine monthly stock returns for 37 world stock markets for the period January 1970-August 1998. They report that returns are significantly higher during the November-April periods versus the May-October periods in 36 of 37 markets examined and label this phenomenon the Halloween puzzle. These results conflict with those predicted by the efficient market paradigm. Using United States monthly return date, Maberly and Pierce (2004) report that, in terms of statistical significance, the Halloween puzzle is not robust to alternative model specifications. In particular, after adjusting for the October 1987 Crash and the August 1998 failure of Long-Term Capital Management, the Halloween puzzle becomes statistically insignificant at a meaningful level.

A seasonal timing strategy:

We discovered that those best days on average are October 16 for the entry into the market for its favorable seasonal period, and April 20 for the exit from the market’s favorable season. However, those are just the best days as averaged over a very long time period. Obviously the market does not begin a rally on the same day each year, or begin to decline from a top on the same day each year. So we then concentrated on determining a means by which the entries and exits could be more accurately pinpointed as they change each year.

The strategy uses MACD to time entries and exits. 

This paper claims the Halloween Effect is simply due to a few "outliers," and Efficient Markets are safe:

On re-examination, the documentation of a Halloween effect in the U.S. disappears after an adjustment is made for the impact of outliers, in particular the largemonthly declines for October 1987 and August 1998 associated with the stock market crash and collapse of the hedge fund Long-Term Capital Management, respectively. For the U.S., the empirical evidence indicates that the Halloween effect is not an exploitable anomaly, and this is true for both spot and futures prices. However, in bear market years there exists anecdotal evidence that most of the negative decline in equity prices occurs during the May-October periods.

From SafeHaven:

According to the Stock Trader's Almanac, if you invested in the Dow from November through April, and switched to fixed-income investments from May to October, over a 54-year period (1950-2004), your returns would have been quite dramatic. Using this strategy, a $10,000 investment in the Dow compounded from 1950-2004 produced a $492,060 gain in the November - April period, versus a loss of $318 for the May-October period.

Hmmm, you decide, but have a Happy Halloween!

CASTrader I, The good, the bad, and the ugly - Part I, the good

The first incarnation of CASTrader is complete, and the results are in, and like a good spaghetti western, they can be divided into the good (this post), the bad (Part II) and the ugly (Part III).  I'll discuss the improvements required for CASTrader II in Part IV.  Now, on to the good:

Click the graph for an annotated trade simulation of CASTrader vs. the Dow Jones from 1915 to 2006.  I can only characterize the overall performance as very good, despite a long list of caveats I will discuss in Part II and the ugly period I'll discuss in Part III.  Three simulations are plotted.  The two dark lines start just before the crash of 1929, and the third (yellow) starts in 1915.  The purple line is the Dow Jones Industrial Average.  Each simulation starts with 1000 randomly created traders.  Many of the annotations on the graph speak for themselves, but I'll run through the history of the third (yellow) simulation:

• 1915-1920.  This period is characterized by full diversity.  Traders completely disagree with one another, and there is essentially no net trade.
• 1920-1929.  A net (mostly long) long trade emerges from the aggregate of traders and keeps up with the market, even though it is only partially exposed to the market.
• 1929-1934.  Good performance and adaptation to some of the largest market gyrations ever.
• 1934-1967.  CASTrader keeps up with the market with low exposure to it.  It could use some improvement, but it's encouraging.  The performance here is probably actually better than it looks due to improper accounting of interest rates (more on this later).
• 1967-1983.  The ugly - more on this in Part III.
• 1983-2006.  The absurd.  CASTrader starts beating the market at an astonishing apparent rate, primarily due to some elite traders finally surpassing all their brethren and dominating the trading with leveraged trades that seem to be amazingly accurate.  Be sure to read Part II to see why this performance is not to be trusted.  Despite those substantial caveats, the elite traders appear to beat the market with amazingly low drawdown and volatility, something that flies in the face of the Efficient Market Hypothesis, yet fully consistent with the Adaptive Market Hypothesis.  The elite trader starts with 1/1000th of the total capital, and typically ends up with 50-70% of it.

Now for a summary of the good points:

• CASTrader is finding patterns.  Despite the caveats I'll discuss in Part II, there is no doubt that CASTrader is finding regular patterns.  The pattern finding algorithm is unconventional, and to my knowledge is not used by any other trading method or published research.  The algorithm even performed well on predicting "random" stock returns generated by the conventional (not cryptographic) VB.Net random number generator. Given the difficulties of generating truly random (unpredictable) numbers (numbers may not need to be random to be unpredictable), it is absurd to think that a collection of humanity with behavioral biases will produce a market characterized by a random walk, or even an unpredictable one.  (Aside: cryptography requires the generation of true random numbers (or at least unpredictable ones) to encrypt information, and people go to great lengths to try to generate them - many think they cannot be generated on classical computers, but others disagree.  In a sense cracking the market is like cracking a cipher - trying to find patterns in seeming random numbers) There are patterns to be found, and CASTrader is capable of finding some of them.  It's impossible to even describe in a meaningful way what the patterns are.  Examining the net trades of CASTrader provides no clues - the minute you think it's going with the momentum, it goes contrarian and vice-versa.  Sometimes it increases the stakes after losing, and sometimes it scales way back, and ditto for winning  Other than checking that the returns are not due to a programming error, I have not delved into what it is finding or what it has "learned," because it would be an exercise in futility.
• Patterns seem to be growing.  Whatever these patterns are, there is no question that they are more prevalent in the recent past than the distant past, even after discounting the emergence effects observed in CASTrader.  Despite the recent rise of the Super Quants, and laments about the lack of alpha, these patterns persist, and even grow.  Why?  I have no idea other than the fact that more people are trading, derivatives have exploded, and perhaps things are accelerating, creating opportunities.  It's counterintuitive that these things would lead to less randomness, but it's fully consistent with the Adaptive Market Hypothesis, in my opinion.
• Performance from 1915 to 1983 may be better than it looks.  Most of this period was characterized by little or no aggregate leverage and only partial exposure to the market.  Furthermore, the traders do not earn interest on the cash they are typically sitting on.  More importantly, though, the period 1920 to 1983 is characterized by a period of partial diversity.  Since the traders are sitting on cash a lot, there is opportunity to reap the benefits of partial diversity by adding more diverse traders or even trading other, less correlated markets in tandem with the Dow.
• CASTrader did this on price alone.  CASTrader I only uses price history as input.  This leaves plenty of room to improve performance by adding other exploitable information such as volume, economic data, interest rates, fundamentals, news, individual stock selection, etc.
• No data snooping.  No traders or algorithms were ever discarded and then carried forward to an "out-of-sample" test.  The whole test is "out-of-sample."  It's sink or swim for CASTrader from the beginning.  Furthermore, none of the algorithms were selected from a pool of algorithms known to do well in the markets - they are all completely novel and random.  There is an equal likelihood that a trader will be buy-and-hold or always short the market than some other kind of novel pattern trader/detector.  That said, it takes CASTrader 5-6 years of data to begin to "emerge."  In effect, CASTrader is creating it's own learning/observation period before creating a net trade and going live - a good, natural thing unrelated to data snooping in my opinion.
• CASTrader did it without evolution.  Although there is emergence in CASTrader, there is no evolution involved to produce it, because I haven't gotten around to coding it yet.  Evolution can potentially improve results even more by allowing the elite traders to "mate" and produce evolved offspring with even better abilities.  Right now, they are stuck with the basic properties they had in 1915 and must survive intact trading the market until 2006.  It will be fascinating to see how much evolution can improve the learning and hence results, but  I won't be totally disappointed if it shows mild or no improvement.
• CASTrader did this on the Dow.  One would think that the Dow would be one of the toughest places to find patterns in, but CASTrader did it.  Most research consistently finds higher alpha in smaller caps, leading me to believe CASTrader would as well.  That CASTrader finds patterns in the Dow is encouraging.

Despite all the good, there is a lot of bad (Part II) and even ugly (Part III) to temper the enthusiasm, because much of the gains may be illusory.  CASTrader is definately not ready to go live with my full bankroll, despite the apparent good.  It's just not worth the risk.  In Part IV, I'll discuss what can be done.  With any luck, CASTrader II will fit the bill to go live with a full bankroll.  Unfortunately, it may be months away.  Between now and then, I may take CASTrader live with a very small bankroll I can afford to lose to see how it performs on paper vs. real money.

Adaptive Market Hypothesis: Yes Virginia, there is alpha in the market

I thought I'd expand on the Adaptive Market Hypothesis (AMH) I referred to in a previous post asking if technical analysis works.  AMH is a new, competing hypothesis to the Efficient Market Hypothesis (EMH).  EMH, I think, is best summarized in an old joke:

Two finance professors walking down the street. One spots a $100 bill lying on the pavement and points this out to his colleague, who says, "That cannot be a $100 bill or someone would have already picked it up." And so they continue walking.

The AMH version of the joke might add on the following line:

A quantum physics student picks up the $100 bill, invests it in a quantitative trading algorithm he's been working on and turns it into $1 Million in 10 years.

The jist of the argument is this:  EMH says excess markets profits are not to be found; AMH says they are.  Andrew Lo coined the term Adaptive Market Hypothesis in his paper just two years ago in 2004, but the basis for it extends back through many studies that attacked the EMH.  Adaptive markets were being described before Lo as well. 

According to Lo, EMH can be traced back to Samuelson, a nobel prize winning key player in the book Fortune's Formula and the Kelly Formula debate.  EMHers believe that market participants are rational, and all excess profits have been squeezed out, and thus prices are random and unpredictable.  The biggest extension to Samuelson's work was the concept of risk-adjusted returns:  you can actually have excess returns, but not without added risk.

The Efficient Market Hypothesis has survived so long because it is virtually impenetrable to argument or proof.  Point out an investor like Warren Buffett who is making excess profits and an EMH believer will likely retort with "he's taking excess risk" or "put a thousand monkeys in a room with typewriters and eventually one of them will type the Declaration of Independence."  It's mainly a thought exercise for those who choose to bang their heads against that particular brick wall.

Arguments against the EMH have emerged, however, from Behavioral Finance (which Lo summarizes in his paper), or from studies that show excess risk-adjusted market returns can be found (that were also not data-mined), such as described in the series "Can technical analysis beat the market?"  One of the key, overlooked arguments against the Efficient Market Hypothesis in my opinion is the Grossman-Stiglitz Paradox, which states:

If a market were informationally efficient, i.e., all relevant information is reflected in market prices, then no single agent would have sufficient incentive to acquire the information on which prices are based.

In other words, if a market is efficient, there is no incentive for it to remain so.  A corollary to this which I have long believed, but have never seen described anywhere can be stated as follows:

Markets become as efficient with respect to a given measure as the most marginally profitable player attempts to make them.

As an example, options on stocks become as efficient as the most marginally profitable arbitraguer makes them.  Why the most marginally profitable?  Because highly profitable arbitrageurs with an edge will exploit an opportunity for all it's worth, backed by their own growing bankroll or other people's money until it just marginally pays to do so.  In the process, they may make it unprofitable for less efficient players.  A quasi-equilibrium is reached whereby the most marginally efficient player survives and ekes out excess returns until market dynamics change.  Note that if there are still more profitable players, the equilibrium is likely in flux.  Another way of looking at this is:

The people who work so hard to make the market as efficient as it is must get paid excess risk-adjusted return to do so - they don't work for free.

The above scenario is exactly what happens in biological evolution, which is what the Adaptive Market Hypothesis is all about.  In fact many of Lo's theories grew out of work with Doyne Farmer from the Santa Fe Intute, who also happen to research Complex/Evolutionary Adaptive Systems like CASTrader.

Research into the AMH is obviously very young, but there are several implications of AMH that can be summarized as follows:

• Yes, you can make excess profits in the financial markets.  Opportunities for arbitrage exist (Grossman-Stiglitz Paradox), and may appear and disappear.  Rather than a trend toward ever increasing efficiency as EMH implies, efficiency ebbs and flows with market euphoria, panics and crashes.
• For awhile, anyway. Risk/reward relations are unlikely to be stable over time.  Opportunities appear and disappear.  Risk premiums are shaped by the participants in an evolutionary manner based on their past experiences.  (People who have never experienced a bear market may demand less risk premium)
• Thus, investment strategies wax and wane, too.  Methods that worked before may fade, only to surface again when market conditions dictate. A graph from Lo's paper (at left) shows the serial correlation of the S&P Composite from 1871 to 2003.  Think of the blue as a measure of efficiency over time: how predictable are future prices based on past prices.  If the market were efficient, you would see no blue, and if the market is more efficient than it has been in the past, it's not by much!
• Innovation is the key to excess returns.  An investor must adapt to the changing conditions to keep their excess returns flowing.
• Changing conditions can make you extinct in the markets.  Think dinosaurs, think LTCM

One thing that I think Lo has completely overlooked, which surprises me, is the role of cooperation in biological evolution, complex adaptive systems, and by extension, the Adaptive Market Hypothesis.  If "survival of the fittest" is the dark side of evolution, then cooperation is the bright side.  Biological species and agents in complex adaptive systems show time and time again the advantage of banding together achieves.  Species and agents often cooperate in ways which assures their long-term survival.  Financial markets are constantly adopting new innovations.  Weather derivatives and ETFs are new, recent ways to invest that change the landscape.  Traders of weather derivatives can cooperate with investors who buy weather derivatives to hedge all sorts of weather related risk.  Like the flower offers the bee pollen in return for helping it reproduce, weather derivative buyers can unknowingly make it profitable for traders to create a liquid market. Cooperation creates new opportunities for capturing alpha for diverse species.

Another point that Lo talks about, but really doesn't drive home as an implication of AMH is how dominant agents can implode suddenly, being too "adapted" for their own good.  Dinosaurs grow too big for their own good and when the environment changes, they go extinct.  Hedge funds like LTCM can become highly leveraged, making a lot of money until they implode.  Ditto for Amaranth.  In my opinion, the Kelly Formula is the "secret sauce" that can optimize the chance for survival.  Get greedier than indicated by the Kelly Formula, and you'll eventually blow up; bet less than the Kelly Formula, and you'll be outcompeted by people who do.  The Kelly Criterion is the equilibrium survival state of the AMH.

Further reading:

Andre Lo's original paper (very readable) and a follow-up paper  Update: a whole bunch of other papers by Lo

A CXO Advisory summary of the paper

Blog posts:

Econophysics

Perfectly Reasonable Deviations

Mahalanlobis

Media:

Financial Planning

Business Week

Academic research in support of AMH:

The adaptive markets hypothesis: evidence from the foreign exchange market

Other:

Pensions at Work