CASTrader Blog

Quant "Bloodbath" - The lessons learned

There's quite a bit of commentary out about the "10,000 year" bloodbath the quants have been experiencing for the past few weeks.  I'll summarize some of it, then add my own thoughts, and then getting back to the point of this blog, summarize what it means for CASTrader.

What blogs I read think.  First, Abnormal Returns consistently has a roundup of the current market pulse and regularly expands on a current theme of the day, such as the bloodbath in more detail.  Zero Beta previously warned about the dangers of the hedge fund "black box" quant models, and in an aftermath post, points the finger at leverage, finding a source that indicates these funds typically run 3 to 10 times leverage.  Paul Kedrosky thinks fat tails and "10,000 year" events are the wrong way to look at things.  Kedrosky believes there is a mode of the markets that conforms to the statistical distribution model, but also one that is distributionless.   He makes the analogy of regime change in the markets being like schools of fish responding to a predator.  The Epicurean Dealmaker makes a similar analogy to Kedrosky's distributionless mode by describing the markets as being similar to and avalanche or failure of grains in a sandpile.  David Merkel at The Aleph Blog thinks the carnage is not over.  1440 Wall Street locates a defiant, "damn the torpedoes, full speed ahead" quant.

Updates: Is the fingerprint of Barclay's evident? 

Factor models gone bad?

Commentary from Lehman on what happened. (PDF) (via World Beta).  More analysis at DealBreaker.

Update II: Rick Bookstaber, author of Demon of our own Design, has analysis. (via Felix Salmon)

What I think.  Andrew Lo's Adaptive Market Hypothesis (AMH) gives an explanation for all of this, and confirms many of the above analyses, but uses a different analogy.  An excerpt from a post on the AMH I wrote last October:

Yes, you can make excess profits in the financial markets.  Opportunities for arbitrage exist (Grossman-Stiglitz Paradox), and may appear and disappear.  Rather than a trend toward ever increasing efficiency as EMH implies, efficiency ebbs and flows with market euphoria, panics and crashes.

For awhile, anyway. Risk/reward relations are unlikely to be stable over time.  Opportunities appear and disappear.  Risk premiums are shaped by the participants in an evolutionary manner based on their past experiences.  (People who have never experienced a bear market may demand less risk premium)

Thus, investment strategies wax and wane, too.  Methods that worked before may fade, only to surface again when market conditions dictate.

Innovation is the key to excess returns.  An investor must adapt to the changing conditions to keep their excess returns flowing.

Changing conditions can make you extinct in the markets.  Think dinosaurs, think LTCM.

...

Another point that Lo talks about, but really doesn't drive home as an implication of AMH is how dominant agents can implode suddenly, being too "adapted" for their own good.  Dinosaurs grow too big for their own good and when the environment changes, they go extinct.  Hedge funds like LTCM can become highly leveraged, making a lot of money until they implode.  Ditto for Amaranth.

On related note, I've become much more convinced by this episode that predator-prey relationships exist in the markets and that backtests are dangerous.  An excerpt from that post:

Conventional backtesting systems that I am aware of do not attempt to measure how your system alters the market.  Prey-type backtests are based on the assumption of infinite liquidity, which isn't a bad assumption for small traders.  In fact, a perfect backtest could not possibly be performed on a predator system because you cannot possibly know apriori the exact consequences of your actions.

The "45-year lookback models" that don't account for liquidity and other key ingredients of the AMH are of limited value, especially when running billions.  There's only so much alpha for a given model or strategy (and your fellow trader's models) to exploit.  If you're trading like everyone else - you're prey.

What it means for CASTrader.  It's pretty simple: adapt or die.  As I also wrote in the AMH post:

In my opinion, the Kelly Formula is the "secret sauce" that can optimize the chance for survival.  Get greedier than indicated by the Kelly Formula, and you'll eventually blow up; bet less than the Kelly Formula, and you'll be outcompeted by people who do.  The Kelly Criterion is the equilibrium survival state of the AMH.

Kelly can only be applied when you know the odds and your edge, and it must be applied with great care, otherwise it can be hazardous to your wealth.  In an adaptive marketplace, the odds and your edge change - they are by no means static.

The Murky World of the Monetary and Banking System

Money, on the face of it, seems pretty simple, but it's kind of like the sausage factory.  Our modern monetary systems around most of the free world, as I understand it, are fractional reserve banking systems.  This means our monetary system is debt-based, which has some apparently startling implications to some (from Wikipedia, emphasis mine):

Robert H. Hemphill, credit manager of the Federal Reserve in Atlanta stated: “If all the bank loans were paid, no one would have a bank deposit and there would not be a dollar of coin or currency in circulation. This is a staggering thought. Someone has to borrow every dollar we have in circulation, cash or credit. If the banks create ample synthetic money we are prosperous; if not, we starve. When one gets a complete grasp of the picture the tragic absurdity of our hopeless position is almost incredible, but there it is. It (the banking problem) is the most important subject intelligent persons can investigate and reflect upon. It is so important that our present civilization may collapse unless it becomes widely understood and the defects remedied very soon.”

I'd like to be at the hypothetical table when Cramer and this guy got together for a few beers after the Bear Stearns conference call.  What does synthetic money in the above quote mean?  Well, if you thought your government created all the money, you'd be wrong.  They actually only create a fraction of it such as in the form of coinage and paper money, among other things.  Private banks create much of it via loans.  The central banks have the delicate task of controlling how much money is created (or destroyed) via open market operations.  If you think about it, unless you're Tony Soprano, nearly all of the money you handle is non-physical, and nearly all this money is created by regulated private banks. 

Seigniorage. Creating money out of thin air is, as you can imagine, quite profitable and is called seigniorage when a monarchy, bank or government does it.  If you do it yourself, say by color xeroxing $20 bills on a color copier, it's called counterfeiting, and you'd be in big trouble, bub.  If you deposit money in a bank (most likely by check or electronic payment), a fraction of that money is kept on-hand by the bank, and the rest is loaned out to say, finance a sub-prime mortgage.  In theory, money now exists in your account and as a loan.  That loan money is used to pay builders who deposit the profits and the process continues ad-infinitum.  Of course, it's more complicated than that with the Federal Reserve (which is actually quasi-private) involved as well as banking regulations, but you get the idea.  Conspiracy theories about all this abound ever since the Federal Reserve Act of 1913 basically created the current system.

Seigniorage and Inflation.  If the money supply inflates, seigniorage obviously does too.  A cynical view would say that those who create money can kick back and relax during inflation, while the rest of us have to work hard to keep ahead of it or stay even.  Somewhere I read that a dollar today is worth about $.04 in 1913 dollars. Hiding money under a mattress just doesn't pay.  In 1971, we adopted Chartalism, as the rest of the world did or had previously done.

Currency.  Money is emotional and political, of course, and there are people who want to abolish the current system, making the State the only authority to create money (seigniorage reform) and profit from it.  There are those at the other end of the spectrum who want to give nearly anyone the authority to be a bank (free banking).   No matter who within an economy profits from seigniorage, the nation or entity that controls the currency, controls the profits from it.  If yours happens to be a reserve currency, you control the seigniorage profits from creating the reserve currency, among other benefits.  There are people who don't like that (not one bit), and I never really understood why the Euro was created until I discovered this.  To the extent the Euro replaces the Dollar as a reserve currency, subsidies will flow to Europe rather than the US, and US banks I would think would see less profits.  It's pure speculation, but to the extent the subprime mess increases that trend, it will have a greater impact.  (Update: a WSJ list of fallout).

In a future post, I'll try to decipher what all this means for CASTrader. 

Beyond the Continuous Double Auction - Part III, A new kind of market

This is Part III of a series searching for an alternative to the Continuous Double Auction (CDA) market to serve as the dark market for CASTrader.  Part I discussed the problems associated with Continuous Double Auctions.  Part II (at Midas Oracle) looked at some existing alternatives to Continuous double Auctions, and this final part examines a new type of market.

A new kind of market.  It occurred to me that most of the existing and planned CASTrader trading strategies can have their market orders expressed as continuous mathematical functions.  Any trading strategy utilizing the Kelly Formula, for instance, can express their order to buy or sell as a continuous function (the Kelly Formula itself), because the Kelly Formula calculates how much a Kelly adherent should invest based on the odds (price) offered - the better the odds, the more it buys.  Likewise, the Shannon Method is a continuous portfolio adjusting strategy that buys/sells with price drops/rises.  In fact, given a frictionless market, the Shannon Method can serve as a market maker, in a continuous manner much like Hanson's market maker does, even in the thinnest of markets.  The difference is, the Shannon Method can be profitable because it is a volatility harvester. Even more sophisticated strategies such as Cover's Universal Portfolio can be expressed as continuous functions (except Cover's invention applies to a whole market of stocks).  Hanson's market maker, with it's continuous price function could also be utilized if the market needs additional subsidized liquidity, with the caveat that it need not be the only market maker.  In this type of market, the line between trader and market maker is blurry. 

Market Clearing. When you aggregate a bunch of continuous orders (price functions) together, clearing the market becomes a process of solving the functions for the exact price where buy and sell volume match each other, just like a call auction market.  A single order or batch of orders coming in would typically be matched and partially filled by every other trader with a standing continuous order.  At that exact price, the market is cleared and stable until new orders come in and the market is cleared again.

Continuous Clearing (or not). This type of market can be cleared continuously like a CDA if desired.  Other than the constraint that new orders be held in a queue while the last batch is being solved for a clearing price, the market can process new orders continuously, even sequentially.  However, if you prefer to mix in the liquidity and volatility advantages of a call auction market, the market can be cleared on a discrete basis on any time interval you like.

Sophisticated Order Types.  Sophisticated strategies become much easier to code in such a market, which means less time coding strategies and more time inventing or adapting them.  For instance, VWAP algorithms don't look very easy to code (PDF) in a CDA.  By allowing traders to spread their orders over time and/or volume, as well as price in this new type of market, a VWAP algorithm becomes just another order type, and trivial to implement.  In fact, many sophisticated strategies I am aware of can become just another order type, meaning they are available to all traders.  The "market" becomes primarily a solving machine, with fair treatment to all comers.  Front-running other trader's orders becomes difficult, if not impossible, and all traders are on a more even playing field with regard to trade execution.

Multiple Markets. To the extent different markets are linked together by traders and arbitrageurs, the solving (clearing) process becomes a more involved number-crunching optimization problem, since equilibrium prices must be solved on all linked markets simultaneously.  To the extent you have markets circularly linked by arbitageurs and traders, price solving gets significantly more computation intensive, but is perfectly suited to something like a grid.

Friction.  One thing that appears to be crucial to this type of market is the absence of friction (specifically fixed friction per transaction) in the form of commissions, fees, taxes and other inducements to trade less.   Any kind of fixed friction will devolve the market to just another CDA by rendering the solving algorithm into an order matching algorithm as traders switch to discrete orders to overcome friction.   Friction handled as a fixed percentage of a transaction could be accomodated, since it penalizes all trades equally and doesn't force traders to use discrete orders.  In the CASTrader dark market, there will be no artificial friction, and no fixed friction of any kind - the only friction this market will have will be the natural price movement due to a new order.

Optimal Trading.  Say a trader believes strongly a given security will be rangebound for some time, but can't predict price movement within the range.  The trader could set up a continuous order to be maximum short at the top of the range, maximum long at the bottom, and having a smoothly varying position inbetween.  The trader has a good tool for harvesting volatility via a single order, and a way of optimizing execution price vs. the opportunity cost of unfilled limit orders at guessed turning points.   

Conclusion.  There are some distinct disadvantages to this type of market to balance what appear to be substantial advantages.  First, describing a continuous market order isn't necessarily a human-friendly task.  My own discretionary trading with the system will likely involve a learning curve.  Second, typical trades will be broken up into many microtrades, which is a nightmare for anything but an automated accounting system.  The cross-market solver will be a technical challenge to get working efficiently, but will at least be much more interesting to code than the order-matching algorithm of a CDA.  Third, this is unknown territory, kind of like CASTrader, and that's exactly why I'm choosing to adopt this new market design.

Beyond the Continuous Double Auction - Part I, What's wrong with CDAs

It's come time to code the dark market for CASTrader II, so I've been thinking about market structures.  The more I think about the Continuous Double Auction (CDA) market, the more I think there's something better, even though it's been the basis for virtually every securities market around since replacing the call market in 1871 to 1885.  Specifically, I don't like the bid-ask spread, the simplicity of order types, and the complexity of automating trading with it.  Because of these deficiencies, I think CDAs are inherently inefficient, albeit big, modern CDAs with decimal pricing and electronic arbitrage are a lot more efficient than they were even a few years ago.  There is a sense of fairness to consider as well:  some sophisticated trading strategies rely on location near the exchange to mint money (via scalping) on a millisecond timeframe, fueling low-latency arms races among the powerful players - few little guys stand a chance against such bad boys of Wall Street and elsewhere.  Average Joe investors don't stand much chance against such crooked shenanigans such as front-running mutual funds, either.  Insider information is not required to front-run orders, however.  Even little guys can front-run orders showing up in the market, which has led to another arms race of algorithmic trading where the big mutual and pension funds try to disguise their intentions as much as possible.  But even a cagey little guy better make sure he's not being front-run behind his back. 

The biggest knock against CDA markets, in my opinion, is the relative complexity involved to come up with a good automated execution strategy.  In the CDA markets, you are either a price-taker (market orders) or price-maker (limit orders).  As a price-taker, you have to be conscious of liquidity available and be careful not to trade too large.  Even the smallest of traders in the thinnest of markets can move prices to their detriment.  You can be a price-maker to avoid all this, but then you have to hope you set your limit order at just the right price, or you have opportunity costs.  If you don't execute well, you lose.  Writing computer code to execute well in a CDA isn't exactly easy.  Thus, a lot of the traders currently existing and envisioned for CASTrader need a substantial amount of code to interface with a CDA.  The problem is that a market filled with traders that don't execute well is an inefficient market. If CASTrader algorithms don't execute well, they'll likely leak money to the real market.  I'll spend more time writing code to try to make an internal dark CDA market competitively efficient than I will trying to beat the real markets.  For all these reasons, I'm scrapping my previous decision to use CDA.

Part II examines some existing alternatives to Continuous Double Auctions.  (Part II is posted at Midas Oracle to give people who think about market designs more than I do a chance to rip this post series to shreds).

Order for Free

I'm constantly referring to real-world models as inspiration for designing CASTrader II.  For instance, there is a bank that will pay/charge interest on deposits/loans (interest was something I identified as a key problem in CASTrader I).  So, each trading algorithm has a cash account from which it can write "checks" to buy securities.  Also, there is a module that provides novation to ensure transactions between traders are reliable and trustworthy.  When it comes time to fund the various traders in the transition to real trading, I will have to decide how much I trust their backtests and capitalize them accordingly.  The problem becomes: how do I decide how to fund a trader, and once I do, how do I get money back out to fund new traders, computers or simply my own needs? 

Whose money is it anyway?  One solution to the money out problem is to simply liquidate the trader's positions and kill it off a little or completely.  Another is through taxes (discussed later) that provide a constant stream of money back.  I've decided on what I feel is a much better way: each trader will itself have shares traded on the internal dark market.  Suddenly, not only can traders (and I) invest in real-world securities, they can also invest in themselves.  To the extent they are any good at it, they should allocate capital a lot more efficiently among trading algorithms than I could ever dream of.  To bootstrap the system, I will own 100% of the shares in each trader intially funded.  I will have the option of either liquidating the assets underlying the shares, or selling the shares, and a trader can also issue and sell more shares if it wants.  It's interesting to note that to the degree CASTrader is a better trader than I am, it could eventually develop an independent pool of cash/investments that it "owns" and I can't touch (on paper anyway).

IPOs.  The above system can also help solve my problem of how to allocate cash to a given trading algorithm based on backtests.  I can just sell new trading algorithms at an IPO, and let the traders decide how much they are worth based on backtest data.  With any luck, CASTrader will become a good handicapper, and I'll receive a stark assessment as to the value of my skills as a trading algorithm programmer.  If I disagree, I'm always free to invest my own cash in the new traders.  The genetic traders that mate can themselves produce IPOs of their own offspring - pairs that produce good offspring will thus have monetary incentive to mate, as well as a source of income that isn't trading.

Taxes.  After my rant about taxes, I've decided taxes are one real-world system I am not porting to CASTrader, or at least to the dark market traders.  Unfortunately, the arbitrageurs that connect the dark market to the real world will be assessed real-world taxes on each trade.  Thus for arbitrage to occur, the friction of taxes and commissions must be overcome, yet trading on the dark market will be commission and tax free.  This contrast will further serve to increase the diversity ratio, which is a good thing, at least at first.   With any luck, the system as a whole will adapt to real-world taxes and try to minimize them with respect to profits.

Order for Free.  The thing about Complex Adaptive Systems is that with luck, a lot of the above will come without significant effort on my part, because of what Stuart Kauffman called "order for free."  The traders will adapt, with a little guidance and help, into assessing the value of not only real-world securities, but also themselves and new algorithms.  I relegate myself to a position of just another trader from micromanager of capital allocation.  In a sense, I compete with CASTrader itself.  Where I see real value, I can capture it with discretionary trades, making the whole system smarter in the process.  Eventually, I hope CASTrader will cross a threshold and become the one to school me in the art of trading.

Complex Event Processing - Part I, Overview

I stumbled upon a relatively new technology while researching OpenTick and cooperative agent frameworks called complex event processing (CEP), which is closely related, if not identical to event stream processing (ESP).  Basically, as I understand it, CEP is a technology for deriving patterns from a stream or "cloud" of events.   It sounds like what every trader (at least the automated traders) do all the time - processing a stream of ticks, bars, news or other data into trades.  Indeed, use of CEP is growing in algorithmic trading and seems a well-suited model for the next generation of automated trading platforms.

An excellent paper describes the design requirements of these systems (PDF).  At the heart of one type of CEP/ESP seems to generally be a real-time event processing server that interprets event processing language (EPL) client requests.  One could even call it continuous computing, although that seems to be developing a slightly different meaning.  High volume and low latency are the typical key design goals.  A client can describe an event pattern to be notified of whenever it occurs.  EPL is modelled on SQL, the standard language of databases.  The great thing about SQL is that you can retrieve data from a database in just about any form you want quickly.  I assume EPL is much the same way.  It's certainly very powerful in concept.

For a "new" technology, there seem to be blogs galore covering it: Marco, Apama, David Luckham, Sudeep Goyal, Todd Biske, tibco and David Soul, among others.  There are also several ESP/CEP engines out there:

Esper (open source): "Esper is a 100% Java component for CEP and ESP applications.  Esper enables rapid development of applications that process large volumes of incoming messages or events. Esper filters and analyzes events in various ways, and responds to conditions of interest in real-time."

Stream (open source): "In applications such as network monitoring, telecommunications data management, clickstream monitoring, manufacturing, sensor networks, and others, data takes the form of continuous data streams rather than finite stored data sets, and clients require long-running continuous queries as opposed to one-time queries. Traditional database systems and data processing algorithms are ill-equipped to handle complex and numerous continuous queries over data streams, and many aspects of data management and processing need to be reconsidered in their presence. In the STREAM project, we are reinvestigating data management and query processing in the presence of multiple, continuous, rapid, time-varying data streams. We are attacking problems ranging from basic theory results to algorithms to implementing a comprehensive prototype data stream management system."

Borealis (open source): "Borealis is a distributed stream processing engine that is being developed at Brandeis University, Brown University, and MIT. Borealis builds on our previous efforts in the area of stream processing: Aurora and Medusa."

There are certainly commercial engines too, like Coral8, and Apama.  In Part II, I'll discuss the implications of all this for CASTrader II.

Cooperative agent programming frameworks

A great thing about blogging is you get ideas pointed out to you in your comments section.  Two readers pointed out some interesting cooperative agent programming infrastructures I was not aware of before:

Cybele: "...CybelePro provides its users with a robust high-performance infrastructure for rapid development and deployment of large-scale, high performance agent-based systems." There is an open-source version: "OpenCybele adopts a service-layered architecture promoting plug-n-play capability of agent services, such as error handling, thread-management, event handling, communication, timer, and migration.  The agent services are categorized into three types of layers–-basic, fundamental, and supplemental--based on their relevance to a typical agent runtime environment."

Then there's new Microsoft technologies called Concurrency and Coordination Runtime, and Decentralized Software Services (PDF): "Imagine an application model composed of ubiquitous state-driven services. Each service has a little bulletin board of richly-typed key/value pairs and can trigger events whenever a value is changed. Services can be instantiated anywhere where there is a lightweight service host (such as a PC, microprocessor, browser, or smartphone) and can register themselves as visible to other services, either by kind or identity. Services easily detect, address, and send messages to each other, requesting changes to each other’s state values and registering for another’s change notifications. Running services can migrate from machine to machine. Service state is completely versioned and serializable and can be backed up, restored, and rolled forward and back in time."

The Internet Communications Engine (ICE): "The Internet Communications Engine (Ice) is a modern alternative to object middleware such as CORBA™ or COM/DCOM/COM+, with support for C++, C#, Java, Python, Ruby, PHP, and Visual Basic. Ice is easy to learn, yet provides a powerful network infrastructure for demanding technical applications. Ice shines where technologies such as SOAP or XML-RPC are too slow, or do not provide sufficient scalability or security."

Cougaar: "Cougaar is an open-source, Java-based architecture that supports distributed, agent-based applications."

Update:  More ideas from Marc Jacobs:

PeerChannel: "the term PeerChannel is commonly used to refer to the P2P capabilities of Windows Communication Foundation. Regardless of the way you refer to it, the PeerChannel in Windows Communication Foundation hides virtually all of the complexity traditionally associated with developing a P2P application and is, in my view, a game changer in the P2P application development arena.

The PeerChannel Mesh  The PeerChannel mesh is designed for message flooding. However, PeerChannel contains mechanisms that can propagate a message to part of the mesh, rather than to the entire mesh. For this reason, it is more accurate to say that a PeerChannel mesh is designed for multiparty messaging."

Digipede: "The Digipede Network™ is a distributed computing solution that delivers dramatically improved performance for real-world business applications. Built entirely on the .Microsoft NET platform, it is radically easier to buy, install, learn, and use than other solutions.

The Digipede Network allows you to combine the power of your Microsoft Windows desktops and servers to improve the scalability and speed of your most compute-intensive, transaction-intensive, and data-intensive applications. This powerful software tool parcels out your most complex computing jobs across your network, dynamically allocating the computing power of both dedicated and idle resources."

Update II:

Alchemi: "Alchemi is an open source software framework that allows you to painlessly aggregate the computing power of networked machines into a virtual supercomputer (desktop grid) and to develop applications to run on the grid.

It has been designed with the primary goal of being easy to use without sacrificing power and flexibility.

Alchemi includes:

• The runtime machinery (Windows executables) to construct computational grids.
• A .NET API and tools to develop .NET grid applications and grid-enable legacy applications."

Yahoo Real-Time Quotes, VB.NET and CASTrader

CASTrader II's communications infrastructure/network is done.  It works pretty well, with what looks like adequate throughput.  Things I like about the method chosen (simple text files) are:

• Robustness: Program states can be remembered even when machines crash.  Also, the lockup of one machine probably won't bring the whole system to a halt (assuming it's not the file server).
• Simplicity: Nothing could be simpler from a programming perspective, imo.  Data can be automatically captured for posterity (backtesting) as part of the design.
• Easy to Debug:  Simply open a text file and read it.

Things I don't like:

• Too simple: For now, the system is a little too simple for everything.  For instance, each communication "channel" is a separate file, and for high traffic channels, these files will fill up fast - they need to be designed to act as a queue, where old communications are purged (or archived) from time to time.  This is easy to fix.
• Channel overhead.  I am envisioning lots of communication channels between the various programs that will make up CASTrader.  Thus, there will be a lot of file tracking overhead.  Since channels must be polled for new data by each channel listener, there may be a lot of wasted CPU cycles.  If this ends up being a problem, I can fix it with a more complicated design.

I went ahead and created the first program to publish a channel on the network - a Yahoo real-time quote getter - a real time ticker, in other words.  This will be a crucial channel used by the CASTrader dark market arbitrageurs.  At first, the arbitrageurs will pop-up a window, proposing a real-market trade to bring the dark market in-line with the real market.  Once I'm comfortable, I'll probably look into automated trading, probably via the Interactive Brokers API.  The arbitageurs will have to at that point verify the Yahoo data and account for various problems with the feed that can occur.

Coding the real-time ticker was trivial.  Yahoo has a vast array of data available in addition to their real time quotes.  I believe you have to have a subscription to get real time data from Yahoo.  Now all I need is an arbitrageur and a dark market signal for it to trade off of, and CASTrader II will be trading the markets.  The arbitrageur is easy, the dark market is the hard part.

Update:  Ooops!  I didn't realize that a Yahoo real time quote download as coded using the above method does not actually give real-time quotes, probably due to cookies, as Yahoo must not realize I have an account.  Fortunately the fix is even easier than the above, albeit at the overhead of running Internet Explorer:

Public Class Form1
    Private Sub WebBrowser1_DocumentCompleted(ByVal sender As System.Object, ByVal e As System.Windows.Forms.WebBrowserDocumentCompletedEventArgs) Handles WebBrowser1.DocumentCompleted
        Dim element As System.Windows.Forms.HtmlElement
        For Each element In Me.WebBrowser1.Document.GetElementsByTagName("PRE")
            MsgBox(element.InnerText)
        Next
    End Sub
    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
        Me.WebBrowser1.Navigate("http://finance.yahoo.com/d/quotes.csv?s=MSFT+IBM+GM+HSY+JNJ&f=snl1d1t1ohgdr")
    End Sub
End Class

Update II:  Yahoo is good, but Opentick is better.

CASTrader II Communications Infrastructure

It's past time to get some more programming done on CASTrader.  CASTrader II will be a distributed computing system running lots of little programs that do one thing well and communicate continuously with each other.  One program might run a dark market, one might be an arbitrageur, and one might be a community of genetic traders.  One of my first tasks is to set an infrastructure so these programs can communicate.  Basically, I want to set up a Distributed Command Pattern in .NET.  There's what looks like a good, if dated book on P2P with .NET, with downloadable source code.  Since I'm using .NET as a programming platform, I have a few options, but none of them are particularly appealing:

• .NET Remoting.  This is great in concept, but apparently poor in real-world application.  It's a great idea:  the ability for one program to be able to utilize an object from another program.  There are some good guidelines for using it.  You can even use it for cycle sharing applications.  Unfortunately, some of the most useful (Client Activated Objects and events) features for CASTrader aren't scalable even across a LAN.  I could use these features on each local computer, then devise some protocol for each computer to talk to each other, but I'd rather just use one method.  Remoting in a LAN environment essentially is limited to message passing.  Worse, it can't be used for two-way without additional code, since events aren't recommended, but I could make it work, I think.
• Windows Communication Foundation (WCF).  This is the latest and greatest from Microsoft and looks like it fixes the above, unfortunately, it's still in Beta as far as I can tell.  It's free now, but Microsoft will require me to spend more money to use it once it's officially released.  Sorry, Bill, I'm really getting tired of that, and I'm not going to do it on principle just because the previous technology does not work all that great.
• Old School Solution.  I could just use the old trusty internet technologies of TCP and the connectionless UDP to do what I need to do, basically reinventing the above technologies.  UDP is the lesser known and used technology, and certainly has some issues, and .NET needlessly complicates that in some cases.  It does, however, have a multicast feature that would be useful for transmitting things like prices.
• Old, Old School Solution.  As I mentioned before, I love Unix simplicity, and the simplest solution is inspired by Unix: have programs communicate via simple text files, where each file is a sequential channel of information any of the programs can read or write to concurrently.  I can certainly do this since I anticipate CASTrader to extend no further than a LAN for the foreseeable future.  It seems at least as scalable as the other solutions to me, but I don't know what problems are lurking under the surface when I get a bunch of programs reading and writing the same files concurrently (like corrupt files or performance issues).  My intuition says that there are few with this old technology, even on Windows - they've been ironed out and optimized.  Unfortunately, each program must poll each "channel" repeatedly for new information.  However, with disk-caching, the solution basically runs in memory and can easily be more secure and potentially fast as the other methods.  Better yet, it's serverless (except for the file server), and I need not code any higher level communications protocol or message passer.  It's fairly trivial code and has numerous other advantages I haven't listed.  The only question is - will it work in the real world?

I choose to gamble on the last option for now, since it's so trivial to write and even load test.  The whole thing will be wrapped in an abstract class library that will mask the internals should I need to change it to one of the other methods without changing code in every single program.

Is datasnooping an issue for CASTrader II? Part I

What is datasnooping? In the trial run performance of CASTrader, I was quite pleased that it was able to detect price patterns in the market with no datasnooping (aka data dredging or even data mining, although "data mining" commonly references "good" data analysis).  Current academic research about possible trading patterns in the market now typically go to great lengths to prevent datasnooping, because otherwise, they are highly suspect results.  You may ask "If I find a pattern in the market that is highly profitable, shouldn't I exploit it?"  Not necessarily - how many other patterns did you discard to find that one?  Simply put, if it's too many, you may be guilty of datasnooping. 

What happens when you datasnoop?  You fool yourself - you are fooling yourself by finding fleeting patterns that will not last.  Those profits, however real in the past, are illusory going forward.  At best, you might match the market.  At worst, you could underperform it.  The worst problem in my opinion, though, is when traders attempt to pick the best trading system on a so-called risk-adjusted basis.  Depending on the way you measure performance, you could be convincing yourself you have a high-return system with some added volatility, when all you really have is mediocre returns with high volatility.

Datasnooping Reality Checks. Academics have come up with "reality checks" for datasnooping, which certainly help build confidence in sorting out datasnooped effects from real ones.  In my opinion, however, these reality checks are exactly that - a check, not a guarantee.  In other words, I don't think a pass/fail on the reality check necessarily means a system will pass/fail going forward.

CASTrader and datasnooping.  The first incarnation of CASTrader was developed without datasnooping.  The pattern finding algorithms were new and novel (and none were discarded or biased), and incorporated no pre-conceived notions of what patterns should look like - they were not based on anyone else's work on technical analysis that could have been datasnooped.  Furthermore, traders in CASTrader are created randomly and with equal starting capital, and thrown at the data in a "sink or swim" fashion.  In other words, I'm testing all hypotheses collectively from the get-go and discarding none.  This is about as far from datasnooping as one can possibly get in my opinion.  I'm fully confident the patterns CASTrader I found are not the result of datasnooping.  Of course, I haven't exactly applied CASTrader to the market yet, and therein lies the rub.

CASTrader II will be subject to the datasnooping effect, and I'll address that in Part II.