CASTrader Blog

OpenTick, and clean financial data

Looking around for sources of intraday data to test CASTrader, I stumbled on OpenTick.  I was quite surprised to find that they offer this data for free - 10 Terabytes worth.  Plus, they offer an open source API.  They also offer streaming quotes from a variety of sources.  They plan to offer this data free forever, as part of developing other products.  All you pay is the exchange fees, which is obviously where they get their data from.  As this appeared to be the greatest thing since sliced, buttered, jammed bread that floats in the air rather than falls on the floor,  I signed up on the spot.   In a matter of a few minutes I downloaded and tweaked their sample application and had realtime, Level II tickstream of the QQQQ blasting data into a console window, all via VB.NET, without even making the top 10 CPU-consuming processes on a 90% idle machine.

CASTrader II doesn't really need this Ferrari of a system right now, and the Model T of a solution would do just fine, but I suppose if you're just running to the store for groceries, you may as well go in style.  Plus, I needed the historical data.  The Yahoo solution will still be useful as a backup/check in the event of outages, and even to provide the fundamental data at some point.

There was chatter on various quant message boards I found questioning the integrity of OpenTick's data along the "you get what you pay for line of thinking," but I'm not too worried, as CASTrader won't be dabbling in any high frequency finance with the outside world for awhile.  Second, CASTrader has a built-in data filter in it's design - the arbitrageurs.  CASTrader will have two price data streams - the internal dark market and the real world, with the arbitrageurs filtering between them.  The internal dark market price stream will be pure, clean and error free, and and the arbitrageurs will protect it from the outside world by dealing with any noise.  The great thing about a Complex Adaptive System is that most of this work is done free by the system - arbitrageurs that don't deal with noise well die out, while those who do thrive and evolve into better ones.  I just have to make sure there is a diversity of arbitrageurs.

Thus, a clean, high frequency dataset can be produced for future backtesting of next generation of traders by simply recording the dark market price stream, and playing it back for them.

Data Sources Linkfest

One thing CASTrader will eventually consume in abundance is data.  In preparation, I thought I'd start a category to catalog sources of data.  I don't vouch for any of these, they're just on my list for investigation.

Downloading Economic and Financial Data from the Internet.  This paper shows how to use Rmetrics to download quote data from Yahoo! quotes and econometric data from economagic.com, which supposedly has 100,000 economic time series avaliable.  Wow, that would keep a number-cruncher busy for awhile.  According to their website, Rmetrics is:

the premier open source solution for financial market analysis and valuation of financial instruments. With hundreds of functions build on modern and powerful methods Rmetrics combines explorative data analysis and statistical modeling with object oriented rapid prototyping. Rmetrics is embedded in R, both building an environment which creates especially for students and researchers in the third world a first class system for applications in statistics and finance. Rmetrics allows you to study all the source code, so you can find out precisely which variation of the algorithm or method has been implemented. This let's you understand what the source code does. Thus Rmetrics can be considered as a unique platform ideally suited for teaching financial engineering and computational finance.

Rmetrics runs in "R", which according to their website:

R provides a wide variety of statistical (linear and nonlinear modelling, classical statistical tests, time-series analysis, classification, clustering, ...) and graphical techniques, and is highly extensible. The S language is often the vehicle of choice for research in statistical methodology, and R provides an Open Source route to participation in that activity.

Estimating world GDP, from 1,000,000 years ago, no less.

St. Louis Fed Economic Data

Comprehensive economics data sources on the web - University of Michigan

Quite a bit of free high frequency data is available from Dukascopy, a Swiss brokerage house

Resources for economists on the internet

SMTM Perl/Tk Global stock ticker  See also beancounter, rquantlib, and quantian

Economics data and information links - University of Omaha

Global Financial Data

Pinnacle Data Products

Martin Capital Current Economic Data

The Chart Store