Many event processing tools and engines contains some kind of processing element which takes events as input and produces events as output.
Currently* there seems to be to two concepts which are popular among the CEP tool vendors. There’s the SQL based approach, and the rule based approach. Both normally share the powerful mechanism of taking events as input and producing events as output.
The processing elements, SQL queries or rules, can as a result of this easily be organized in hierarchies. Organizing things in hierarchies is certainly nothing new. But I would like to show how powerful hierarchies are when applied within event processing.
I’m a bit surprised that there have not been more talk about hierarchical event processing in the CEP community. It’s a concept which is worth explaining in detail to CEP users.
Recently we have created a geofencing application using ruleCore, or rather the new geospatial ruleCore. It’s intended for fleet management, mobile asset management, telematics and all kind of location based solutions which need a quick alert capability to immediately get notification about unexpected vehicle behaviour.
We used rules organized in a hierarchy to implement all the geofence logic:
This hierarchy of rules allowed us rapidly to create the initial types of geofences found in the requirements list. It gives us a nice structure which can easily be extended when we add new types of geofences, like the off-route (Alert when a vehicle is not following its designated route) geofence we added recently.
It also allows us to add new rules in a modular fashion to monitor a specific property of a geofence. For example, there are rules which monitor some of the geofences and create alerts they are not triggered during a specified period. Great for arrival and departure alerts.
At the bottom level we feed the rule hierarchy with the GPS positions. The Level 1 rules trigger when the vehicle is within a zone. Level 2 in turn has rules which trigger when a vehicle is considered to entered a zone, which is not the same as being inside the zone.
In this way the events go from simple low level position update events to rather high level events at the upper level.
So for your next (or first) event processing project, spend some time in the design process and try to see if your rules are naturally hierarchical. If organized in this way you have a flexible and powerful abstraction which also tends to be easy to understand and explain to new crew members.
Most of the time the levels can also be give descriptive names as each level tends to handle rules at a certain abstraction level. For example, monitoring a web site and its infrastructure could use rules on four levels:
and so on.
Each level can emit alert events to the outside world in addition to the level above. This would allow for example the network manager to only listen to events from level 0 in the above example.
* The current set of vendors would obviously want you to believe that the current tools are mature and there are nothing new coming just around the corner, that is: buy now, don’t wait. But the event processing market is a very young one. The concepts are new. We are lightyear away for anything called mainstream and don’t even think of finding standards. In the years to come I’m sure there will be new and interesting concepts (which will not carry the legacy of SQL or rules) and implementations created for specially for event processing. The current tools are inspired by existing research and uses extensions to the SQL concept and the concept of rules. If we were to start over with a clean sheet I’m sure there would be more exiting tools than the current generation.
Some people seem to be quite surprised to find that TIBCO BusinessEvents, classed as a Complex Event Processing product, includes a “rule engine”. Especially those that look at the CEP tools market and just see the stream-based CEP products doing continuous queries. Nonethless, there are good reasons why a production rule engine is ideal not just for business decisions, but also event filtering and correlation.
First, lets classify “rules” and “production rules”. Production rules are a type of rule that represent an “IF THEN” statement, but typically applied not to single data variables, but to classes of data (and events). Many commercial rule engines, including BusinessEvents, use developments of the Rete algorithm devised in the 1970s as an efficient pattern matching algorithm for dealing with multiple patterns and multiple data elements. The rule engines that use this algorithm typically provide “forward chaining” in that any rule’s action can update data tested in some other rule, and the rule engine can automatically fire the new rule too. The “interesting” aspect of this is that the rule engine determines when rules fire, not the programmer ordering the rules in some way. For this reason, production rules in rule engines are considered “declarative” - their execution is independent of the order they are defined, which makes design-time scalability easier. Needless to say, the Rete algorithm means that the cost of adding rules to a system is not linear - twice the rules take far less than twice the space and far less than twice the execution time.
The fact that production rules are defined in terms of classes adds another dimension in flexibility. These “declarations” in BusinessEvents, or RuleVariables in the OMG PRR standard, add runtime scalability - the same rule can execute for 1 or 1000 instances of a class, without any additional configuration.
So there are 3 basic parts of a (production) rule (used in a Rete-type rule engine): declaration/RuleVariables, condition, and action:
For event processing purposes, the production rule format and inferencing rule engine has a number of advantages:
So production rules enable the definition of arbitrarily complex and numerous event patterns. When combined with an event-aware rule engine, production rules provide an Event Condition Action (ECA) rule mechanism combining the features of business rule engines with event-driven behavior. The rules can represent basic raw event processing, complex (abstract) event processing, and event-based decision processing - using rulesets to differentiate these as appropriate! They can be combined with mechanisms like state models and query languages. They are a proven technology, and they work well.
Other References:
Earlier posts covered the value of state modeling to CEP, the differences between rules and decisions and decisions and CEP [*1], and CEP vs BREs and BRMSs.
Notes:
[1] The difference between event-processing rules, event-driven rules, and rules for business decisions has been much-blogged-about. See these blogs by JT (twice!), Opher and Marco - the latter 2 pretty much covering the other, albeit ECA-based, rule-based CEP systems. In summary, of course, event-processing rules, event-driven rules, and rules for business decisions can all overlap, depending on the application.
One of the questions that sometimes comes up in our meetings with prospective clients is "Why should I buy your whizz-bang CEP platform to do <insert use case here> and then build on top of it when I can go get a shrink-wrap app that does most what I want straight off?". Of course, this is a very easy question to answer: "If you can get an app that does what you want now, and you're confident it will meet your requirements as they evolve going forward, play nice with your existing systems, and the price is right, then why are you wasting time talking to us?". Marc Adler said as much recently with his comments on Risk Management systems, which were right on the money.
Of course, this answer is really an open incitement to the audience to start a discussion about what they really need - now and in the future - and often (more often than not in my experience) it is soon apparent that there is no app out there which will do all, or even most, of what they need - and extending a shrink wrap app to do that is often more work and pain, if it's possible at all, than building it themselves.
So here the salesperson will introduce the core value proposition of an open CEP platform; the ability to support existing and future requirements, to meet the underpinning performance requirements, the necessary resilience, to provide the connectivity, and - vitally - the tooling to allow the same vendor-supported and maintained technology to be applied to the client's constantly evolving needs, with rapid time-to-market. In short, the salesperson emphasizes the benefits of a more "strategic", platform-based approach to solving the client's short and longer-term requirements and - if the salesperson has done his homework and read his audience right - that one-size-fits-all shrink wrap approach is often consigned to the trash.
The nice thing about this approach is that there's no sleight of hand here; every word of the pitch is true.
Let's look at the use of Apama within Foreign Exchange (FX). We've blogged previously about how Apama is being used extensively for real-time FX rate Aggregation and algorithmic execution; just last week our latest client to go live - Standard Chartered - talked publically about their deployment of Apama for their FX spot dealing. We've built a Solution Accelerator to offer relevant but customizable connectivity, algorithms, monitoring and dealing GUIs out of the box. We've even won awards (ok, enough already!). However, although we're (clearly) not shy about trumpeting the power of our offering in this area, the more telling aspect of our deployments - and the aspect of relevance for this article - is not what clients do with our Aggregator; it's what they do *next*.
I was visiting one of our large Tier 1 banking clients in New York a couple of weeks back. They've been using Apama for FX Aggregation for nearly a year now. I walked through the door on to the trading floor and first thing I saw was 3 screens with - unfamiliar - Apama Dashboards on them. "That looks nice" said I (somewhat gormlessly). "Yes, it's our new auto-hedger - it uses the aggregated book feed from your Solution Accelerator and lays off our risk positions over your ECN connections according to a new algo we've built". All developed and running in Apama, extending our Solution Accelerator with new algos and screens - and all unbeknown to me!
I was somewhat more in the loop concerning the work of our friends at one of the world's largest Banks, who have been using Apama for FX Aggregation for well over a year, connecting to multiple liquidity pools to improve their execution. That bank has have recently launched a new FX pricing engine - built with Apama - connecting to their Apama FX Aggregator for real-time pricing, running analytics to derive per-client spreads for market making (similar to how I describe [here]). An auto-hedger is in the works - again extending the initial FX Aggregation deployment and based on the Apama CEP platform. And at least 3 other FX clients have either rolled out or are in the process of rolling out market making solutions as extensions of Apama FX Aggregation deployments (and those are the ones I actually know about!).
Moving beyond auto-hedging and market making, we have a growing number of examples of the benefits of a more strategic "platform" approach, such as Apama clients like ING in Europe taking FX feeds into their Equities Algo systems (also built on Apama) for cross-border spreads (locking in an FX rate as the third leg of the trade), prop shops engaged in currency forward / spot arbitrage, and clients like Louis Capital FX extending their Apama-based FX agency hedging their FX options trading. (As an aside, one the hottest topics for us right now in FX is the area of real-time Forward FX price streaming to large corporates - once the province of the very biggest banks - utilizing real-time volatility and depth of book analytics ...)
The power of the more strategic approach enabled by CEP platforms like Apama, armed with the out-of-the-box connectivity, performance, resilience and tooling, to meet client's current and future needs, is nowhere more ably demonstrated by our experiences in FX.
An industry colleague mentioned to me over the Summer that they had attended a “Cloud Computing” talk where it was mentioned that cloud computing would overtake Complex Event Processing. I didn’t get a good response as to what “overtake” meant in this context - that all CEP will be done in cloud computing land? - that cloud computing will overtake CEP in the hype charts (assuming CEP is there anyway)? - that clouds of events will be processed near-source in-the-cloud is the inevitable future for event processing?
To some cynics, cloud computing will be seen purely as a means of delivering virtual server farms, primarily for SaaS (Software As A Service) vendors and their customers, who in turn for the most part are simply part of a renamed ASP (Application Service Provider) market. Amazon’s EC2 seems to be a pretty good example of cloud computing today, and it’s name (”Elastic Compute Cloud”) pretty much describes its function - the virtual mainframe resource for everyone else.
“Computing resources for data / event processing” are a somewhat orthogonal problem to the actual software algorithms used in CEP and event-based decisioning. There is surely no theoretical reason why you could not deploy a CEP application onto EC2 / into a cloud, although latency from event source to the cloud and back again could be an issue, as well as the need for reliable high performance shared data stores. On the latter point, I’m not sure how easy or performant TIBCO BusinessEvents‘ data grid would be on EC2, whose current uses seem almost entirely to be web-service and SOA-based. On the other hand, “the cloud” is possibly a good solution for global-scale event sources, for example as would be found in a large-scale distributed social networking CEP application.
So is there any connection between the terms “event-cloud processing”, as done by CEP, and “cloud computing”? Not particularly, as today the sources of events are not the “same cloud” as the cloud computing centres. On the other hand, I would not be at all surprised to find that the cloud computing architectures relied on event processing techniques to manage users, SLAs and pricing for their systems.
Index
Past alphabet soups have compared CEP to (2) BI, (3) BAM, (4) BPM, (5) BPEL, (6) SOA, (7) SQL, (8) MDM, (9) ETL. Future TLAs are TBA.
One of the difficult things when it comes to marketing CEP in general and specifically the ruleCore CEP Server is terminology.
The ruleCore CEP Server uses as its main processing concept something we call “rules” or “active rules”. The problem with that is that my rules are not your rules. The odds for misunderstanding me when I talk about rules are astronomical. Rules mean different things to everyone.
But we still call our the thing inside ruleCore for rules, because technically it is an active rule engine. People just hear the rule part and assume that we have a variation of the traditional rule systems based on RETE och similar algorithms. Nothing could be farther away from the truth. And we really can’t blame the customers either, it’s totally our fault as a vendor if customers don’t understand what we are selling. But we don’t have a better name for them so we are kinda stuck with ‘rules’
On the positive side, at least some people, understand what we are up to. There are basically two main kinds of rules in my world:
The first ones I would like to call (inspired by James Taylor) Decision Rules,
The second category is something I would like to call Activation Rules.
Both rules are important for a complete solution. The Activation Rule tells you when there is a decision to make and the Decision Rule makes or helps you to make the decision.
The two types of rules solves two different problems:
By using Activation Rules you can start your decision process much more quickly. A real-world example could be to alert an operational fleet manager that a truck is not following its designated route. Without an Activation Rule the fleet manger would manually have to check periodically if all trucks are on track. A Decision Rule would then be used by the fleet manager to decide how to manage the exception.
So my problem is currently how we should communicate this fact to potential buyers. Maybe we should just rename our rules to something else. Or perhaps create a higher level abstraction which we use when talking about the system. The rules could just be an implementation detail.
One particular idea for a marketing message would be to talk only about business event tracking. That’s basically what ruleCore is good at anyway. The rules are just how it is done. For example we could talk about tracking vehicles with GPS or tracking a zone for the presence or absense of a allowed number of containers (also with GPS). Combined with ERP data we can track trucks, business processes and other things as they move around. Might be a good idea, have to think more about this.
Although we contributed one session (on PRR and decisions) to the RuleML conference co-hosted with BR Forum this year, I missed some of the other but nontheless significant talks. I see the keynotes are now officially published, so below are some reviews of the presentations from the web:
The other presentations are up too. I won’t go through all of them, but in particular note Mark Linehan’s talk on an ontology for time (and thence an ideal basis for a business-level CEP language), and Harold Boley and Ben Craig’s personal agents (using distributed rule systems like BusinessEvents allows). Mark Linehan won the best paper award, justifiably, for his work on mapping SBVR rules (i.e. business policy constraints) to UML classes and OCL (i.e. object constraints) - which will be interesting to compare with Microsoft’s research work in this area.
The 2nd day of the BPM ThinkTank Europe event included a round table session on “process governance”. I am not as familiar as I should be with TIBCO BPM’s governance provisions, but on the Complex Event Processing / Event-Driven Decisions part we support Business Rule Management in TIBCO BusinessEvents 3.0 through a managed, extensible, state-based / rule-driven workflow mechanism. This role-based “process” manages the development / deployment lifecycle for decisions (a.k.a. decision tables) modeled / tested in BusinessEvents’ Decision Manager application. As such, this level of control, beyond simple SCM-based change control functions, is generally more than traditional BRMS / BPMS tools provide (and more than other ESP / CEP tools, of course).
The round table of course was looking at the wider issue of process governance, which is a subset of business governance, which is directly related to business models and controls. The discussion covered management control (or lack thereof) of processes and process improvement, through to possible “best practices”. One of the latter might be corporate adoption of aspects of the BMM or Business Motivation Model, defining the links between business processes, policies and goals. These links can then be used to define the processes around governing change in organizations.
Disclosure - TIBCO is on the Revision committee for OMG BMM 1.1.
The 24th Annual FIA Futures & Options Expo took place in Chicago this week in and we used that venue to announce an Apama partnership with CQG. A quite significant player in the Futures trading market, CQG offers real time quotation, charting and technical analysis tools. Via this new partnership, CQG users can now automatically execute trades that have been signaled from within CQG's technical analysis models, leveraging algorithms that are offered via the Apama CEP platform.
A key element in the partnership is the deployment model. Apama is being hosted in the CQG infrastructure such that their users do not have to install Apama locally in order to access the benefits of the integrated offering. They can select and parameterize the execution algorithms via Apama dashboards and gain very low latency execution through CQG’s co-located facilities at major global exchanges.
Deals
like this signal the growing interest in Apama. Part of the appeal is the flexibility of the underlying Apama CEP platform (and its supporting tools). That flexibility encompasses different asset
classes, the different application use cases that are needed to support those
asset classes and, as is evidenced here, different deployment models.
Shown below is a CQG screen, with the Apama dashboard component highlighted.
Mostly of interest to existing Syndera users: TIBCO has released Syndera 3.3.0, which is now available from the TIBCO download site.
This release mostly consists of a “TIBCOization” of the real-time dashboard rather than new functionality per se. Or, if you prefer, Syndera has been “TIBbed”, “TIBCO’d”, or “TIBCOnified”. That means internal, license and packaging changes only.
More later.