Orinoco – Microsoft’s CEP Engine

Microsoft officially announced Orinoco at TechEd 2009, which is Microsoft’s Complex Event Processing (CEP) engine. Richard Seroter did a great post after the announcement. CEP is an exciting technology space and great to see MS join the party. With the current trend of investment vehicles moving away from low volume complex derivative products towards more high volume flow products that investors can more easily understand and risk, the importance of building high through-put, low latency trading applications can only get higher in my opinion. Orinoco will likely be a great tool for algorithmic and program trading applications, which accounts for ~49% of NYSE trading currently. That’s not to mention a slew of other applications both inside and outside of the financial services space.

The sweet spot for CEP’s are around the processing and analysis of very large numbers of real-time events, potentially 100’s of thousands per second. What’s an event? well it could be a whole slew of things, for example: real-time ticking market data for a trading system, click-stream events from a web site, intelligence information, RFD tag data, real time analysis of instrumentation data from distributed systems, there are plenty of interesting scenarios where CEP can be leveraged.

CEP platforms are kind of like inverse databases – instead of issuing queries over data, the data (events) is passed over the queries. Queries can result in new data streams (events) being generated (projected) which in turn can be passed over other queries. Events can be correlated/joined with each other events. Queries in Orinoco are defined using LINQ, which will potentially lower the bar for adoption in .NET applications, a key point here, especially the application of CEP in trading systems is that the real value / IP is in CEP queries themselfs, this is where the magic is defined. Some of the query capabilities that will exist in Orinoco includes:

  • Projection: computations can be performed over individual events as they are transformed
  • Aggregation: e.g. SUM and COUNT
  • Windowing: CEP’s use the notion of windows which you can think of as an in-memory database table of events or a sub set of events, windows can be used for temporal operations, for example, you could use a window to capture the average price over a 10 second sliding window
  • Joining: correlation of values from different event stream
  • Selection/filtering: SQL like select statement over an event stream
  • Not exist: detection of a missing event
  • Multicast: send the events to more than one output adapter
  • TOP-K: ranking of events
  • Extensibility: finally, queries can be extended using custom .NET logic
  • Ordering: the engine can be used to perform the heavy lifting around the re-ordering of data that arrives out of order

Of course you need to get your event data into and out of Orinoco, for this it has an adapter framework to enable it to be hooked up with with event sources and sinks (input and output adapters). It seems out of the box these will be pretty limited, but from what I can tell they won’t be terribly onerous to write. As soon as I get my hands on the binaries I’ll find out!!

Advertisements

~ by kevinsmi on July 7, 2009.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: