Another year has passed in the world of Complex Event Processing and everything looks great, or does it?

The CEP Market

I have followed the CEP business for a while now. In 2001, the early days of CEP, when we started with ruleCore there were basically nothing called CEP. Then suddenly I started to hear this talk about Event Stream Processing and after a while it changed to Complex Event Processing - CEP- About the same time marketing departments (basically the founders plus friends) of a handful of startups went nuts. Bold claims were published almost daily and "we are faster than you" were the typical marketing message for several years.

This year have been rather quiet from a marketing perspective and the most balanced year this far. Very few bold claims of being first, being fastest or having coolest features. Mostly rather boring press releases about new versions and a some talk about new partnerships and customer stories.

Maybe the CEP companies started to use professional PR firms? Probably a good idea. But a bit boring, I miss the performance claim war and the personal touch of marketing from startups in their early days.

Also the marketing machines of IBM, Tibco, Progress and Oracle now have an event processing solution to deal with. It seems that event processing is hiding somewhere inside these giants below layers of SOA, BPM and other more mainstream technologies. Only to be seen occasionally. But I’m expecting to see more from these giants during 2009, I think they are just warming up and trying to understand what to do with this new thing called event processing. Maybe we will even see some announcements from Microsoft? (who will they buy to get up to speed?)

It seems that finance is the dominant market these vendors go after. Others, like Coral8, have started to broaden their message and talk more about applying event processing to business intelligence problems. Especially Coral8 seems to be heading in the right direction. It will be nice to see how their new strategy works, if they don’t get acquired my someone first.

There are also marketing activity from other vendors attempting to get the attention of telco, CRM, fraud detection and logistics users. It seems that everyone is still trying to figure out where to go and exactly what the best marketing message looks like.

I think 2009 will be a defining year for CEP and most vendors will discover where event processing solutions are most appreciated by the users. During 2009 I would expect most vendors to follow the money and go for sectors where they can quickly increase their revenue. In 2009 it’s time to start making money in the CEP business.

The SQL Based Approach

We have during 2008 seen number of products getting more advanced and powerful all based on the same idea that streams should be queried using same type of queries which are used to query data at rest.

Most of these products have some kind of textual language which look similar to SQL. A couple have graphical variations on the theme in the connect-the-boxes-with-arrows style. These vendors have added a lot of features to their own SQL-derived language in order to make it work with streams of data. Most languages are powerful and can be used to create complex systems with some ease. These SQL dialects can be found in products from Coral8, Esper, BEA WebLogic Event Server, StreamBase and Aleri.

Personally I see that these SQL based products are great tools for creating data driven algorithms and performing computations on all kinds on data feeds. It’s no surprise that these vendors are targeting the financial markets. It seems that all vendors in this camp have had some success in attracting some customers from the financial sector.

It will be interesting to see what happens if (when?) the major RDBMS vendors announce the ability to register SQL queries in their databases and run them continuously. When this happens, the CEP vendors with SQL based products might have a hard time in making a point for a separate product for processing data streams. Their products will just look too similar to the newly added streaming SQL (or whatever the RDBMS vendors will call that new feature).

My guess is that we will see announcements from at least one major database vendor in 2009 with features for processing streaming data and continuously executing queries. It might not be a bold guess that Oracle will be the first one implementing something like this and then followed by the regular open source copy cats. When this happens I would not be surprised if some of these CEP vendors re-invent themselves as solution providers for the financial industry. After all, they have a great tool for solving problems in that domain.

Rules

A number of vendors have chosen a rule based approach to complex event processing. The ruleCore CEP Server, IBM Business Events and TIBCO BusinessEvents, Progress Apama and Agent Logic all use some form of rules.

In this category we have as many approaches to event processing as they are vendors. Depending on the background of the vendor you can see different aspects highlighted in the various products.

RuleCore provides a foundation for adding reactivity to event-driven SOA, with location aware event processing as its speciality. True to its heritage Progress Apama talks mostly about algo trading, smart order routing but are finding its way into other areas like Telco and RFID. Agent Logic seems to focuse on the GUI (Can we see CIA/NSA operatives busy with creating new rules each day?) in order to give the spooks a great tool to listen in on various telco trunks (I made this up, but if I were NSA/CIA that’s what I use it for).

TIBCO and IBM feels a lot like the traditional message broker tools where you control the flow of events and take decisions depending on state. If you have been working with traditional message brokering tools you might wonder what the excitement is all about as this sounds like old tools with new name, now they are called events instead of messages.

Aleri have a combination of a SQL based language and a special purpose stream processing language called SPLASH, which I will continue to watch with interest.

At ruleCore we have spent the whole year in improving the product by adding support for location aware rules. Basically native support for GPS events. So now we can easily create rules which track vehicles and detect deviations from expected behavior or security violations. Geofencing is now really easy to do using Google Maps.  We also improved the context aware rule evaluation features and have now a really nice declarative rule evaluation model for tracking business entities.

Others

Apart from the vendors I already mentioned there are also a small number of other vendors that I have not heard much about this year. Not sure if they are silent or just not on my radar screen due to lack of time. If wish I had more time to follow all the interesting companies doing event processing out there. There are Event Zero, Pion, Senactive, WestGlobal, RTM and maybe others which I can’t remember right now. A quick look on their web pages indicate that there are at least some activity going on and lots of ambition to bring event processing to the masses.

Customers?

So, how many customers are there out there using any of these CEP products?

I hate to sound pessimistic. But from a distance, I don’t see that many customers actually using CEP. I’m sure many of the vendors have a number of customers which might keep them busy for a while. But I think we could still fit all CEP users into a single room. Not sure if the small number of customers is a problem. There are not many CEP vendors to begin with, so even a small group of customers can keep us running.

If you explain the CEP concept to a groups of developers I’m sure many of them go "hey, that’s something we’re been doing for a long time". So I think there are many developers which are "doing CEP" but not actually using a tool specially made for event processing. The normal Java, EJB, .Net tools seems to work for now. All these will eventually start to look at "real" CEP tools.

Community

In 2008 we saw a lot of activity in the Event Processing Technical Society. It’s mission is "To promote understanding and advancement of Event Processing technologies, to assist in the development of Standards to ensure long-term growth, and to provide a cooperative and inclusive environment for communication and learning". It was founded by 29 companies which more or less define the whole event processing world as we see it today. During 2009 there will be lots of interesting activity in a number of workgroups. Interoperability is the one I’m going to watch closely and hopefully have time to take part in.

The Future

A couple of years ago, about 1998, a company called iSpheres developed a CEP product. Things were going fine and then management came and shared their infinite wisdom and called for a vertical marketing approach. Energy trading was the silver bullet! Energy in the form of electricity can’t be stored to it has to be traded in real-time. So all forces was focused on solutions for trading electricity. Then came Enron…

Back to 2008 and the era of financial meltdown. Look at the current crop of CEP companies focusing on algorithmic trading and other solutions for the financial industry. Do I even need to point out the obvious here?

Some of the CEP companies will probably have a hard time during 2009. I suspect that at least a couple will go belly up, others will be acquired to avoid that fate and many others will have to fight for venture capital.

If I put on the customer/user hat for a while I still can’t see a convincing message from many of the major CEP vendors. Most users that I talk to just don’t get it either. It is nor clear to them why they should invest time and money in these products.

An informal survey that I have made shows clearly what I have suspected; There is simply no demand on a larger scale for event processing! Most users just conclude that the current generation of tools like their favorite SQL database, application servers and programming environments can do the job. I’m not saying that there are anything wrong with the current CEP tools. It’s just that very few feels that they have a need for this kind of tools.

The major challenge for the event processing vendors in 2009 will be to convince the users that they need a completely new technology to solve streaming problems. And yes, most users would benefit from stopping development of their own streaming solutions and buy one instead. Surely most vendors will win customer during 2009, but at a high cost and only after many sales calls and visits.

But I’m still optimistic. Event processing is like a heavy train with a undersized engine. It will take a while to accelerate but when it have gained speed it’s hard to stop. (Here I thought of inserting a joke about our state controlled railroads, but it didn’t seem fair to kick at someone already laying down).

The products of many CEP vendors have shown great improvements in 2008 and they start to look really good. So there’s nothing wrong with the technology.

Let’s see if the customers will find the world of CEP during 2009…

(On an unrelated note I would like to add a bold prediction that during 2009 we will see something like the 2000 dot-com bubble burst. There are just too many companies with very hard to understand business ideas. They all seem to have in common that everything are given away free and revenue should be made by some other clever and complex scheme like advertising, services, or whatever. Sites like Google and Facebook will be the first ones to hit by sudden problems and will drag most of the social-something and web2.0-something companies with them creating the second IT crisis. Everyone with a business idea (that is, how to make money) that I can’t understand in 10 seconds will go down.)

 

 

 

 I got some good comments on my last post on event identity from my readers, some of which can be found in the comment section of my previous post.

I must been a bit tired when writing that last post as I complicated things when I started to think about event IDs. But thanks to my helpful readers, I’m now back on track again after a couple of nights of sleep and lots of xmas food high on carbs and fat…

A good way to think about event id seems to be this:

The event is a notification about an observation,

the observation observes a real-world event,

which can be observed by multiple observers,

thus multiple event objects can contain information about the same real world event…

 

With this in mind the event identity is nothing more than the identity of the observation.

The problem I started to think about in the last post is still a real problem but it happens so to speak at the next level.

What if we want our event processing rules to see only one copy of each real-world event? This would simplify the creation of rules in a good way. This kind of event fusion capability is something that would be a great feature in the event processing platform you are using…

 

Let’s look at an example…

 

To continue with our vehicle tracking example (we are doing lots of location aware stuff with ruleCore right now so that’s why I’m stuck in using vehicles in my examples all the time. But the problem is the same independent of what you are tracking)

A common thing one does with trucks, is to wait for them. So a typical reaction rule in ruleCore would be to detect the arrival of a truck, so you can do something else while waiting for them.

Optimally, the rule should be simple to write: 

"Trigger rule when vehicle enters zone A".

In a complex environment things are not that easy. We could detect the fact that a vehicle has entered zone A using a number of methods:

1. Somebody scans the vehicle or its cargo using RFID scanner which is known to be located inside zone A.
2. Security notes that a vehicle has passed the gates.
3. The driver calls in and says he’s arriving in a few minutes.
4. We note that the drivers cell phone is inside zone A, using the cell network’s positioning mechanism.
5. And the obvious one - The GPS of the truck sends a location update from inside zone A.

Just in this simple example we have five types of events which could be used to determine that a vehicle is inside zone A. To make life easier for the rule desgner, we would like to provide these five events as an unified ZoneEntry(ZoneId, VehicleID) event instead.

There are probably a number of solutions for this. Currently in ruleCore we use rule hierarchies to solve this. We have domain specific rules at lower level feeding rules at higher level with these high level events.

But I really would like to find a generic solution for this one.

I think the military rooted sensor information fusion world have lots of good ideas on this one.. Stay tuned…