CASTrader Blog

The Halloween Effect Linkfest: Real or not?

Is it time to buy the Halloween Effect?

The Halloween Effect in US Sectors:

All US stock market sectors and industries perform better during winter than during summer in our sample from 1926-2005. In more than two-third of all sectors and industries this difference in summer and winter returns, known as the Halloween effect, is statistically significant and in half of all sectors and industries risk premia are negative during summer. However, while all sectors and industries show this effect, there are large differences across sectors and industries. The effect is almost absent in sectors related to consumer consumption but strong in production sectors. We illustrate how these differences between sectors might be used to improve the risk return trade off using sector rotation based on Fidelity sector funds and show how an investor might have benefited from such a trading strategy.

The Halloween Effect and Japanese Equity Prices: Myth or Exploitable Anomaly?

Bouman and Jacobsen (2002) examine monthly stock returns for 37 world stock markets for the period January 1970-August 1998. They report that returns are significantly higher during the November-April periods versus the May-October periods in 36 of 37 markets examined and label this phenomenon the Halloween puzzle. These results conflict with those predicted by the efficient market paradigm. Using United States monthly return date, Maberly and Pierce (2004) report that, in terms of statistical significance, the Halloween puzzle is not robust to alternative model specifications. In particular, after adjusting for the October 1987 Crash and the August 1998 failure of Long-Term Capital Management, the Halloween puzzle becomes statistically insignificant at a meaningful level.

A seasonal timing strategy:

We discovered that those best days on average are October 16 for the entry into the market for its favorable seasonal period, and April 20 for the exit from the market’s favorable season. However, those are just the best days as averaged over a very long time period. Obviously the market does not begin a rally on the same day each year, or begin to decline from a top on the same day each year. So we then concentrated on determining a means by which the entries and exits could be more accurately pinpointed as they change each year.

The strategy uses MACD to time entries and exits. 

This paper claims the Halloween Effect is simply due to a few "outliers," and Efficient Markets are safe:

On re-examination, the documentation of a Halloween effect in the U.S. disappears after an adjustment is made for the impact of outliers, in particular the largemonthly declines for October 1987 and August 1998 associated with the stock market crash and collapse of the hedge fund Long-Term Capital Management, respectively. For the U.S., the empirical evidence indicates that the Halloween effect is not an exploitable anomaly, and this is true for both spot and futures prices. However, in bear market years there exists anecdotal evidence that most of the negative decline in equity prices occurs during the May-October periods.

From SafeHaven:

According to the Stock Trader's Almanac, if you invested in the Dow from November through April, and switched to fixed-income investments from May to October, over a 54-year period (1950-2004), your returns would have been quite dramatic. Using this strategy, a $10,000 investment in the Dow compounded from 1950-2004 produced a $492,060 gain in the November - April period, versus a loss of $318 for the May-October period.

Hmmm, you decide, but have a Happy Halloween!

Dark Market Design, Part II: Market type and structure

In Part I, we examined an overview of the design of the CASTrader II dark market and it's components.  In this Part II, we will look at the type and structure of the market itself and various options for it.  It's not necessary that the dark market need follow the typical Dutch double auction or Continuous Double Auction (CDA) format, as there are many other types studied in the field of auction theory. 

Existing design.  CASTrader I used a very simple dark market that could be extended to CASTrader II and modified to account for transaction costs.  Very simply, the CASTrader I dark market polled each trader about how much of a security they would like to buy or sell at a given price, matched buys and sells until a net trade emerged.  The net buy or sell trade would then be assumed to be executed at the current real market price.  This method has several advantages:

• Simplicity - I doubt any market is easier to code.
• Minimal transaction costs - Although CASTrader I did not take into account transaction costs, it could be modified to include them.  Commission and spread costs would only apply to the net trade and be spread over all traders, minimizing them. The bid-ask spread is said to be the penalty for a market order.  Market orders would thus be cheaper.
• No internal liquidity issues - Traders effectively have all the liquidity they want via the net trade clearing mechanism (assuming real market liquidity exists).

The existing design has the following drawbacks, however:

• Requires a reference price - When traders are polled, there must be a reference price they are polled with.  This price must be the price of the real market.  Large, real bid-ask spreads will complicate this somewhat.  Also, when the market is closed, there is no reference price, so the 24/7 dark market doesn't mean much.
• Transaction costs - however minimal, they still exist.  Moreover, they are an unknown quantity to the trader at the time of polling because they depend on the net trade.  Transaction costs, especially unknown ones complicate trading strategies and rule out others. 
• Inefficiency - Unless the design is made more complex, this design is inefficient in that each trader is activated and polled every time the price changes.  If the trader is running on a different computer than the market, this can really slow things down.  Efficiency can be somewhat improved at the expense of the programming simplicity advantage.

In the final analysis, the best design of the dark market is the one that gives CASTrader the best advantage over the real market.  For reasons I won't belabor, I think the commission-free market I outlined in Part I is the better choice in this respect, although I certainly can't prove it.  It will come at a cost of more programming complexity, but once complete, I think it will prove extremely useful and extendable.  The future plans of CASTrader call for markets of securities that have no real-world analog and hence no "reference" price, thus some alternative to the old design will have to be developed.  The old design can always be revisited later as an alternative and/or submarket to the commission-free market.

Other Designs.  Optimark was patented by IBM as an alternative and is discussed here.  It appears to be designed to give large institutional traders a way to trade without moving the market.  I suspect with the rise of algorithmic trading as a method to do the same thing within the existing market framework, they are having some challenges selling it.  The Vickrey Auction has some applications in other domains, but appears to be a poor choice for a securities market or CASTrader.  This paper indicates that double auctions are more communication efficient than some other auction types in a grid computing environment.  By extension, I'd guess it's generally efficient in a non-grid environment as well.

Even very different forms of markets could be examined.  The recent rise of prediction markets has resulted in testing out various new forms of markets.  Some of these new types of markets don't need market makers at all as CDAs typically do in thin markets.  These include Robin Hanson's Market Scoring Rules (update: David Pennock has an excellent article on Implementing Hanson's Market Maker) and David Pennock's Dynamic Pari-mutuel Market (DPM) of which Pennock writes:

To my knowledge, a DPM is the only known mechanism for hedging and speculating that exhibits all three of the following properties: (1) guaranteed liquidity, (2) no risk for the market institution, and (3) continuous incorporation of information. A standard pari-mutuel fails (3). A CDA fails (1). A CDAwMM, the bookmaker mechanism, and an MSR all fail (2). Even though technically an MSR exposes its patron to risk (i.e., a variable future payoff), the patron's maximum loss is bounded, so the distinction between a DPM and an MSR in terms of these three properties is more technical than practical.

...

A DPM also has some drawbacks. The payoff for a wager depends both on the price at the time of the trade, and on the final payoff per share at the market's close. This contrasts with the CDA variants, where the payoff vector across possible future outcomes is fixed at the time of the trade.

Hanson's combinatorial market (via the Now Economy) offers some interesting possibilities as well, such as betting on mutually exclusive outcomes. I plan to add near frictionless prediction markets to CASTrader at some point, giving yet-to-be-designed traders new ways to trade.  For example, a atrader may hedge a stock bet with bets on the pedicted outcome of GDP.  These alternative market forms may be useful for that down the road design.

Choice for CASTrader.  For now, however, it appears that the tried and true continuous double auction (CDA) is as good of choice for a market as I will find for CASTrader's immediate needs.  The main drawback of the CDA is liquidity, which I will address via market makers as discussed in Part III. There are some efficient algorithms for CDAs that may be of some use.

Limit Order Book.  Traders will be able to place limit orders on the CDA dark market, and unlike some stock exchanges like the NYSE, the order book will be fully open and fully electronic, so traders can see the liquidity available.  There is no reason to use a specialist system or make the order book closed, and I believe this design may give CASTrader another potential edge over the real markets.  That said, CASTrader will not feature any traders who intentionally exploit the order book information to trade until some later date.

Dark Market Design, Part I: Overview

As mentioned in the analysis of the performance of the first incarnation of CASTrader, one of the key components of CASTrader II will be a more sophisticated dark market.  This three part series will look at the basic design of the dark market and it's external components.  At left is a diagram of how the dark market will interact with other components of CASTrader. 

Traders. The "Traders" block basically represents what has already been implemented - a market pattern finder.  The traders will receive some upgrades, but the basic coding is done.  The upgraded traders will interface directly with the dark market, rather than a real-world market.

Dark Market.  The dark market design itself isn't really all that interesting, except for what basic form it will take and how it will operate.  Should I pattern it after the stock market or something else?  I'll cover that in Part II.  As far as how it will operate, the CASTrader dark market will run continuously 24/7 and be commission-free, unlike some of the real markets it will interface to.  These features provide several advantages:

• Easier coding - no need for traders to know when markets are open or closed.
• Continuous information integration - at some point, traders in CASTrader will trade on the news, and the ability to do that 24/7 may provide an advantage to CASTrader.
• Enable more sophisticated strategies - in a 24/7 market, other, more sophisticated trading strategies can be eventually implemented.
• Strategies that are marginal in a market with commissions may be feasible in the CASTrader dark market, enabling greater partial diversity.

Right now, the market will be fairly basic, but eventually the market will have an open order book and traders that take advantage of it. 

Arbitrageurs.  Arbitrageurs will connect the dark market to the real market, engaging in riskless arbitrage between the two markets.  Whenever the dark market gets out of whack vs. the real market, the arbitrageurs will bring them back in line for a guaranteed profit.  Eventually, the arbitrageurs will be automated agents, but for now, the arbitrageur will be me with the help of some alerting software.  Arbitrageurs will effectively present the net trade (or sum total of knowledge) of CASTrader to the real market via an aggregated trade.  They enable the benefits of partial diversity and commission-free trading while still hopefully extracting money from the real markets.  They will only do this when they can cover real-world commission costs.  Eventually, I'll experiment with automated arbitrageurs that take some calculated risks via statistical arbitrage.

Market Makers.  Finally, market makers will provide the dark market with liquidity, just as they do in real markets.  Market makers will be automated algorithms.  The easiest market maker to add is the Shannon Trader.  I'll discuss the Shannon Trader and market makers further in Part III.

Continuous Improvement.  All of the agent components (traders, arbitrageurs, and market makers) will be subject to continuous improvement.  My basic philosophy is this:  if a new agent can be created that is profitable, it is going to extract money from one of two sources - other CASTrader agents, or the real market.  In either case, it's a good thing.  In the former case, it simply makes CASTrader less susceptible to lose money to the real market.  The new agent profits at the expense of older agents and keeps the profits rather than letting the real market take them.  Thus, development will be focused on three areas rather than just market pattern finding.

Implementation details.  Any one of the components will be designed to run on separate computers, or at least in separate processes.  This will allow an easy way to distribute the processing load among various networked computers as CASTrader grows.  Even the dark market could be broken up across computers by partitioning the set of securities traded among several component markets.  In Part II, I'll discuss what form these markets will take.

The Shannon Method

Welcome, readers from Trading Psychology and the Motley Fool, and thanks to Dr. Steenbarger and "bladow," a poster on the Mechanical Investing board at the Fool for recommending this blog.  I was in Montana at the time most of you arrived, but I extend a belated welcome to all of you.  Also, thanks to Rod at Perfectly Reasonable Deviations for the words of encouragement on this blog.

Claude Shannon is widely recognized as the father of the information age and is featured prominently in the book Fortune's Formula, which I previously reviewed.  In a previous post, I discussed briefly a relatively unknown, but simple and elegant portfolio management method attributable to Claude Shannon.  Oaktree Research sums up the method nicely (as does Fortune's Formula in the section called "Shannon's Demon"):

In the 1960s, Shannon gave a lecture in a hall packed with students and teachers alike in MIT, on the topic of maximizing the growth rate of wealth. He detailed a method on how you can grow your portfolio by rebalancing your fund between a stock and cash, while this stock stays in a random ranging market. (He used a geometric Wiener example). Essentially, you buy more when stock price is low, using the cash at hand, or sell more when stock price is high, with allocation of 50-50% value at each interval.

Considering how simplistic it is, it's unusual in that it's a guaranteed market timing model.  If you follow it, you are guaranteed to be selling some of your position at or near market tops and buying at or near market bottoms, depending on how often you adjust the 50-50 split.  It's also somewhat unusual in that volatility is your friend, not your enemy.  According to Fortune's Formula, when asked if he used the method to manage money, Shannon replied that "the transaction costs would kill you."  Transaction costs now of course are much less than in Shannon's day.

Shannon's method is nothing more than a simplified case of the more widely known method of rebalancing, which to my knowledge traceable back to Harry Markowitz's Efficient Frontier, developed in the early 1950s.  Shannon's method could even be considered a special case of the Kelly Criterion, where a fixed percentage of your bankroll is "bet" at any given time.  Yet Kelly's work in the 1950s was influenced by Shannon's earlier work in the 1940's.

In any event, the Shannon method of 50-50 allocation can be shown to be optimal, and it is much simpler in application than either the Efficient Frontier or even the Kelly Criterion.  Although a casual reading of Poundstone's book would give an impression that it works better than it does, it is well worth investigating.  Well before Poundstone's book, and before even hearing that Shannon investigated the same thing, I investigated the method independently.  The question of how well it works is best illustrated in pictures.  At left is what would have happened if you had applied the method for 10 years to JDSU stock starting December 1993.  Note that there are no transaction costs assumed, but there is also no interest paid on the 50% cash stake either.

At left is the method applied to JPM for 10 years beginning 1984.  So does the Shannon method work?  The answer probably depends on how you measure performance, as well as what stocks you choose to apply the method to, and even what other investment alternatives you have.   Almost invariably, after running the Shannon method on a stock for some time, you will discover some point in that stock's history where the Shannon portfolio briefly matches or surpasses buy and hold due to a selloff in the stock.  For example, this happened with JPM in February 1988 and again in November of 1990.  Oddly, this point often uncannily marks a major bottom in the stock.

You will also discover that there will be long periods where Shannon will underperform on a total return basis.  This is true even if transaction costs are free and some moderate interest rate is paid on the 50% cash position.  What these graphs don't tell you, however, is that the Internal Rate of Return (IRR) of the Shannon method nearly always beats buy and hold, often by substantial amounts.  This is because the Shannon method harvests volatility, typically investing cash in a stock when it drops and then quickly selling it for a small profit when it rises.  For some small and volatile stocks, the Shannon method generates as much as a 35% IRR vs. 10% for buy and hold.  Interestingly, my research indicated that although the Shannon method could be applied at very small price changes, it works nearly as well at say 3-4% price changes in the stock, even if there are no transaction costs.  Thus you can minimize transaction costs and still get most of the benefit of the Shannon method.

A high IRR doesn't do you much good though unless you can reinvest the proceeds in something equally good.  I mentioned in the previous post that I use a method similar to Shannon's method to manage my investments now.  What I do is select several diverse (and hopefully uncorrelated stocks) and apply the Shannon method.  The trick is that they all share the same virtual pool of cash.  So I pretend each stock investment has an equal amount of cash reserve, even though my total portfolio has only say 20% cash.  The theory of this is that I am putting some of that IRR of one stock to work in another stock when one goes up and the other goes down. 

To the extent stock bounce around randomly (and uncorrelated), I profit from volatility.  If all the stocks drop substantial amounts, the shared cash pool would dry up and I either stop the Shannon method or go on margin if I think it's justified.  Based on my experience and my stock selections, this method returns 3-4% over and above buying and holding the stocks individually (after transaction costs).  With a 401K account, the method can be tax efficient too.

Assuming the new CASTrader works, it will handle portfolio management rather than my twist on the Shannon method.  The Shannon method will live on in CASTrader, however, as a sort of market maker on the dark market I am building in.  The Shannon Trader, as I will call it, will always have a bid/ask just below/above the current price on the frictionless CASTrader dark market.  The more volatile the dark market is, the more the Shannon Trader will profit, and the more partial diversity, price stability and dark liquidity it will provide the other traders.

Update: Additional info on Shannon's Demon at Nuclear Phynance.

Data Sources Linkfest

One thing CASTrader will eventually consume in abundance is data.  In preparation, I thought I'd start a category to catalog sources of data.  I don't vouch for any of these, they're just on my list for investigation.

Downloading Economic and Financial Data from the Internet.  This paper shows how to use Rmetrics to download quote data from Yahoo! quotes and econometric data from economagic.com, which supposedly has 100,000 economic time series avaliable.  Wow, that would keep a number-cruncher busy for awhile.  According to their website, Rmetrics is:

the premier open source solution for financial market analysis and valuation of financial instruments. With hundreds of functions build on modern and powerful methods Rmetrics combines explorative data analysis and statistical modeling with object oriented rapid prototyping. Rmetrics is embedded in R, both building an environment which creates especially for students and researchers in the third world a first class system for applications in statistics and finance. Rmetrics allows you to study all the source code, so you can find out precisely which variation of the algorithm or method has been implemented. This let's you understand what the source code does. Thus Rmetrics can be considered as a unique platform ideally suited for teaching financial engineering and computational finance.

Rmetrics runs in "R", which according to their website:

R provides a wide variety of statistical (linear and nonlinear modelling, classical statistical tests, time-series analysis, classification, clustering, ...) and graphical techniques, and is highly extensible. The S language is often the vehicle of choice for research in statistical methodology, and R provides an Open Source route to participation in that activity.

Estimating world GDP, from 1,000,000 years ago, no less.

St. Louis Fed Economic Data

Comprehensive economics data sources on the web - University of Michigan

Quite a bit of free high frequency data is available from Dukascopy, a Swiss brokerage house

Resources for economists on the internet

SMTM Perl/Tk Global stock ticker  See also beancounter, rquantlib, and quantian

Economics data and information links - University of Omaha

Global Financial Data

Pinnacle Data Products

Martin Capital Current Economic Data

The Chart Store

Behavioral Finance Linkfest

Proof that losing money really is scary:

The fear of losing money can be similar to the fear of physical pain, according to a study of brain scan images.

The finding could potentially shed light on why people who make high risk financial decisions, such as stock market players, sometimes develop anxiety disorders, says Mauricio Delgado at Rutgers University in Newark, New Jersey, US.

More on his study here.

The cost of anchoring:

During the 2004 Berkshire Hathaway annual meeting, Warren Buffett said the following about anchoring:"When I bought something at X and it went up to X and 1/8th, I sometimes stopped buying, perhaps hoping it would come back down. We’ve missed billions when I’ve gotten anchored.

Randomness and the 'Ludic' fallacy:

One of the fallacies that the Prof. Epps character committed was what Nassim Nicholas Taleb has called the "Ludic Fallacy" (to my knowledge, he first coined and defined this term in his forward to Aaron Brown's The Poker Face of Wall Street). As I understand it, the Ludic Fallacy refers to the unfortunate habit of many quantitative professionals (statistics and economics professors, Wall Street traders, management consultants, government technocrats, etc.) to over-simplify the nature of randomness and chance by making mistaken analogies to games ('ludic' -- of or relating to games or play -- comes from the Latin ludus -- game or play) of chance.

The market's patterns are not our own:

The results also suggest that market returns are patterned in a manner that is precisely the opposite from the pattern of human expectations. If we see something rise during the past day and also during the past five days, it is human nature to look for the trend to be our friend and continue into the next several days. Conversely, if the market is down today and has been down over the past week, we naturally consider the market weak and expect further price softness.

But what happens is precisely the opposite.

DKR to launch behavioral finance fund

Machine learning and machines in the market linkfest

A Markov Chain Analysis of Blackjack Strategy.

Naive Bayes classifier vs. logistic regression

Machine learning videos

Portable alpha - rise of the machine:

Automated algorithm-based investment strategies have been gaining ground with institutional and high-net-worth investors over the past few months. But do they pose a real threat to traditional fund managers? Gareth Gore reports

What if you built a machine to predict hit movies?

Four years ago, when McCready was working with a similar version of the program at a firm in Barcelona, he ran thirty just-released albums, chosen at random, through his system. One stood out. The computer said that nine of the fourteen songs on the album had clear hit potential—which was unheard of. Nobody in his group knew much about the artist or had even listened to the record before, but the numbers said the album was going to be big, and McCready and his crew were of the belief that numbers do not lie. “Right around that time, a local newspaper came by and asked us what we were doing,” McCready said. “We explained the hit-prediction thing, and that we were really turned on to a record by this artist called Norah Jones.” The record was “Come Away with Me.” It went on to sell twenty million copies and win eight Grammy awards.

Soldier of fortunes:

I scarcely dare ask, but is the computer in the building? "Yes," he says. "It's a big one." He leads me through a locked door into what I can best describe as a large, noisy cupboard. Banks of superpowered servers are churning through calculations of unimaginable complexity. Schroeder and I can hardly hear each other above the whirring of the fans trying to keep them cool. On average, this machine buys and sells $10m every hour. On the floor is a row of plugs - just ordinary plugs. I keep far away from them.

The Pope issues a warning on artificial intelligence:

Pope Benedict told scientists on Saturday that by believing only in "artificial intelligence" and technology they risked the fate of the mythical Icarus, whose home-made wings melted when he flew too close to the sun.

"Contemporary life gives pride of place to an artificial intelligence ever more enslaved to experimental tecnhiques, thereby forgetting that all science should safeguard mankind and promote his tendency to authentic goodness," the Pope said.

Emergence is coupled to scope, not level

Is the Increased Speed of Direct Market Data Feeds Worth the Price?

So for a simple, albeit not light-speed fast, aggregated data feed, you now need ticker plants, feed handlers, normalizers, stream processors and time-series databases. Oy. And by the way, none of this is cheap. Including buying the data and transporting it (which I didn't even mention), you are looking at each feed costing upwards of six figures. Multiply that by a dozen and, along with the expanding number of markets, suddenly you are talking real money.

Trading Signal Linkfest

http://healthmap.org/  (via Kurzweil)

Dorfman dividend appeal screen

Trade Ideas Stock Screens

Worldometers - world statistics updated in real time.

A Warren Buffett screen.

The Taylor Effect in autocorrelation of stock returns.

Why corporations are holding more cash.  (they need more to cover uncertainties these days)  Via Information Arbitrage - be sure to read his discussion.  A little more here.

Has there been a fundamental shift in the nature of inflation?

How good is the Baltic Dry Index as a proxy for economic activity?

CASTrader I, The good, the bad, and the ugly - Part IV, Don't Worry, be Happy

This is the last of the series of posts on the first incarnation of CASTrader.  In Part I, Part II and Part III, I looked at the good, the bad, and the ugly points of CASTrader I, respectively.  In this post, I promised I would look at how to improve CASTrader.  It's good to have a critical eye for anything like this, but I think I was worrying about some things more than I needed to, at least for the moment.

The Ugly.  First, I'll tackle the ugly - the 75% drawdown in capital suffered during the simulated 16 years between the heat of Vietnam through the days of disco, and the early Reagan years.  Basically, I'm not going to sweat about this - yet.  Even if a similar period repeats starting the day I take CASTrader live, a period like this would just be a slow bleed of capital.  This means I've got time to solve this problem in a meaningful way.  I've concluded CASTrader needs more information than price alone to perform well during a period like this.  Although I plan to add every scrap of tradeable information anyone ever thought of eventually,  I think the key piece of information needed most right now is interest rates, so they will be added to CASTrader II.  It's important for traders (or any investor) to evaluate the tradeoff between "risk-free" return and speculation on securities.  CASTrader is no different.  I don't know if this will solve the problem, but I've got time to find something else if it doesn't. 

The leverage problem.  This is the problem where CASTrader wants to use up to 7x leverage because the Kelly Formula is telling it to.  Actually, I throttled the run at 7% maximum leverage, because it wanted even more leverage to post even more absurdly astonishing (and unrealistic) results.  I'll carefully let CASTrader build up to whatever leverage it (and I) can handle (within reason), and use put/call options (probably deep in-the-money) to give it that leverage and limit my risk of a Black Swan event.  I can put a hard floor on my downside (to what I paid for the calls or puts) with no cap on upside that way.  If CASTrader can perform anything near what it does in the simulation, the added friction of options (vs. futures) won't matter. 

Transaction costs.  Transaction costs are mainly a slow-bleed type problem before 1983 in the simulation, so like the "ugly" problem, I have time to solve this.  That said, CASTrader II will to factor in real transaction costs, which will result in less trades and less profits, particularly in a pre-1983 style market.  Again, if CASTrader can perform anywhere near like it did post-1983 for the near term, improving performance after transaction costs is a back-burner problem.  That said, I have an ace up my sleeve - a dark market.  CASTrader II will have one, and it will be better than the rudimentary one in CASTrader I, and if I'm correct, it will minimize transaction costs.  There will still be transaction costs that limit CASTrader's ability to trade, but they will be less and less of a factor the more diverse algorithms I add to CASTrader (more partial diversity).

The paper to real world problem.  The easiest way to solve this is to start small and ramp up.  The first time CASTrader goes live, it will be with maybe 5% of my investable capital.  As I begin to trust it's performance, and assuming it's good, I'll add to that.  If I don't, it's back to the drawing board for CASTrader III.  CASTrader II will slowly get a feel for the real world.  It will have the opportunity to learn and adapt to whatever is different between paper and real world without the opportunity to lose what I don't want to lose.  A few other ideas I'll implement are:

• Add whatever sources of real intraday data to the simulation I can so it is as realistic as possible.
• Increase the trader base from 1000 to perhaps 1,000,000+.  Each of the current runs of CASTrader takes about 2 hours for the 90 years of data.  This means 83 days of my computer running 24/7 to do the 1000 extra simulations.  It's time to drag out the old junk computer(s) and put them to work, and probably buy a faster one.  It's a good thing it will be winter and the heat will at least go to good use.  Since the traders will likely have wealth distributed according to some power law, each additional batch of traders will increase the chances of finding even better traders than the elites of the simulations described previously.  At a minimum, I'll have a somewhat more diverse set, because each run produces a slightly different result. At the risk of datasnooping, I'll kill off any traders that haven't made some minimum return after the 90 years, simply to allow CASTrader to go live with a manageable number.  At a minimum, if they can't pay for their meager computation expense along the way, they'll die (cruel and Darwinian, but effective - I promise they won't even feel it).    By the time I go live with 100% bankroll, I may have simulated some 20,000,000+ traders (4.5 years of laptop computation time!  I'll definately need more computers, assuming the little buggers can pay for them).
• For every trader that makes the cut, I'll apply a pretty extreme discount factor such as the square root or even the log of their bankroll as a "weight" for how much capital they receive when they go live.  This will "punish" the elites and "reward" the also-rans.  This is counterintuitive, but it will add a little magic of partial diversity back in. It is simply a risk/reward tradeoff that trades the risk an elite will do very poorly going live vs. the opportunity cost that it does just as well or better.  Elites who can do well in the real world will quickly emerge and dominate again.  This also gives CASTrader a chance to get used to the cruel, cruel real world before losing a lot of money in it.

Ther are still no plans for genetic reproduction of traders, the "magic" of many complex adaptive systems.  How cruel am I, making them sweat, pay, even die for the opportunity to make me money and not even letting them have sex with each other?  Frankly, if the simulations are any indication, they don't need to have sex - yet.  They seem to do well without it, and much to my surprise, their useful lifetime is apparently at least 90 years.  That said, I'll happily let them have sex and hopefully hear the ka-ching of little evolved trader feet running around - when I get around to the weeks/months of coding required to allow it.  The rest of the above, plus some other magic ingredients I want to implement represent months of work.  Perhaps by the first of the new year CASTrader II will be ready to go live.  I don't want to delay any longer than that.

CASTrader I, The good, the bad, and the ugly - Part III, the ugly

Part I and Part II discussed the good and bad points of the first incarnation of CASTrader, and this post will discuss the truly ugly.  The truly ugly period is from 1967 to about 1983 (16 years out of 90) where CASTrader was truly baffled by the market (see graph at left).  It seems CASTrader is finding patterns in the Dow Jones data for other periods (especially 1983-2006) which may or may not be tradeable, but the 1967-1983 period is unique and seems to confound CASTrader.  The aggregate collection of traders gets the market wrong on a frequent basis, even though in aggregate, their exposure to the market is limited.  This means they are calling the market even more wrong than it appears.Over this period, CASTrader loses 75% of it's capital.

Despite the performance in other periods, it is simply unacceptable to utilize a system that underperforms the market that bad for a full 16 years, especially during a period where the market is horrible anyway.  It's unclear what is so unusual about this period, since CASTrader performed remarkably well during the Great Depression and the last five year's worth of sideways market movement.

A casual perusal of the simulations during the simulations shows that it may be partially the elite trader's fault.  Although the elite traders are not yet dominant like in the 1990-2006 period, they typically have a large relative bankroll (10%) and are influential.  They typically get slowly killed during this period and bleed away their dominance at a higher rate than the aggregate of traders (although the aggregate loses money too).

Whatever worked so good for the elites in the 1920-1967 period and later in the 1983-2006 period is simply absent from 1967-1983.  Moreover, it doesn't just not work, it bleeds them.

Again, allowing the traders to earn interest on their relatively high cash stockpiles would help, especially with the high interest rates during parts of this period, but this may be offset or even worsened after factoring in trading costs.

Improving CASTrader's ability to adapt to a period like this may be the toughest challenge of all.  I'll discuss some possible options in Part IV.