Complex Event Processing - Part I, Overview

I stumbled upon a relatively new technology while researching OpenTick and cooperative agent frameworks called complex event processing (CEP), which is closely related, if not identical to event stream processing (ESP).  Basically, as I understand it, CEP is a technology for deriving patterns from a stream or "cloud" of events.   It sounds like what every trader (at least the automated traders) do all the time - processing a stream of ticks, bars, news or other data into trades.  Indeed, use of CEP is growing in algorithmic trading and seems a well-suited model for the next generation of automated trading platforms.

An excellent paper describes the design requirements of these systems (PDF).  At the heart of one type of CEP/ESP seems to generally be a real-time event processing server that interprets event processing language (EPL) client requests.  One could even call it continuous computing, although that seems to be developing a slightly different meaning.  High volume and low latency are the typical key design goals.  A client can describe an event pattern to be notified of whenever it occurs.  EPL is modelled on SQL, the standard language of databases.  The great thing about SQL is that you can retrieve data from a database in just about any form you want quickly.  I assume EPL is much the same way.  It's certainly very powerful in concept.

For a "new" technology, there seem to be blogs galore covering it: Marco, Apama, David Luckham, Sudeep Goyal, Todd Biske, tibco and David Soul, among others.  There are also several ESP/CEP engines out there:

Esper (open source): "Esper is a 100% Java component for CEP and ESP applications.  Esper enables rapid development of applications that process large volumes of incoming messages or events. Esper filters and analyzes events in various ways, and responds to conditions of interest in real-time."

Stream (open source): "In applications such as network monitoring, telecommunications data management, clickstream monitoring, manufacturing, sensor networks, and others, data takes the form of continuous data streams rather than finite stored data sets, and clients require long-running continuous queries as opposed to one-time queries. Traditional database systems and data processing algorithms are ill-equipped to handle complex and numerous continuous queries over data streams, and many aspects of data management and processing need to be reconsidered in their presence. In the STREAM project, we are reinvestigating data management and query processing in the presence of multiple, continuous, rapid, time-varying data streams. We are attacking problems ranging from basic theory results to algorithms to implementing a comprehensive prototype data stream management system."

Borealis (open source): "Borealis is a distributed stream processing engine that is being developed at Brandeis University, Brown University, and MIT. Borealis builds on our previous efforts in the area of stream processing: Aurora and Medusa."

There are certainly commercial engines too, like Coral8, and Apama.  In Part II, I'll discuss the implications of all this for CASTrader II.

Comments

[Tim Bass]

FYI: Not all event processing languages are SQL based, and only a subset are based on SQL.