CLIPS: A Tool for Building Expert Systems

Ok… I know how this sounds, but… have you considered changing the name? I, too, like Drools a lot, but seriously, the looks when I tell a bunch of stuffy MBAs that we’re going to manage the complex rules using a tool called “Drools”…

Try pronouncing it "D-rules".

Okay, I'll give in: Nearly any good academic research library should have and be able to find the paper:

"A Real-Time System-Adapted Anomaly Detector", Information Sciences, volume 115, pages 221-259.

The core math is some measure preserving transformations of the data -- get to move data around without changing probability distribution -- for any distribution. The usual place see measure preserving is in ergodic theory. Recall, my work is distribution free which means need make no assumptions at all about probability distributions. In the context of computer and network performance data, distribution-free is nearly essential. After moving the data around, get to do some counting that leads to calculating false alarm rate.

So the paper is some applied probability, that is, does some probability calculations in the service of some, call it, mathematical statistics.

The simplest view is that it's all just nearest neighbors detection -- if get a point too far from the old points, then raise an alarm. Since I wrote the paper, that idea has become common. But apparently what has not become common is how to adjust and calculate false alarm rate, and in practice that is really important. Indeed, for any detection, my work can report the lowest false alarm rate for which the (real time) observation would still be a detection -- so get to have not just a detection but, intuitively, a seriousness measure.

"Joe, this is a detection, and it will remain a detection with a false alarm rate of one in a billion. Put down your pizza, and let's investigate that." So, right, could display a strip chart with such data.

For calculating false alarm rate there is a simple approach that gives the right answer, but the simple approach makes an impossible assumption of independence. Well, the right answer is still true; it's just that an actual proof of false alarm rate, where don't assume independence, is a bit tricky mathematically.

This math is a close cousin of not independence but a weaker assumption exchangeability where often can get the same results as if had independence.

My math may be something of a rationalization of part of resampling as pursued by P. Diaconis and B. Efron at Stanford.

In an intuitive sense, we almost have independence. Hmm -- approximate independence is not a big field. There is a paper by M. Talagrand, A New Look at Independence that may be able to shed some light on my paper, find another way to do the calculations and show that the same calculations hold approximately, maybe with some bounds, in other situations. Talagrand is one heck of a good mathematician, a student of Choquet, a member of Bourbaki.

It would be nice to have the most powerful statistical hypothesis tests as from the classic result of Neyman-Pearson; alas, in the context we don't have enough data to use Neyman-Pearson. But I did come up with a weaker but still useful sense in which my techniques do yield the most powerful test. In practice, the test will likely be seen as nicely powerful, that is, even when false alarm rate has been selected to something small, the detection rate is still, likely in practice, about the highest can hope for for that selected false alarm rate.

There is an issue of how to make the computations fast -- so, need something out of computational geometry. I worked up a technique that should be reasonably fast. The solid state drives of today should do wonders for this technique.

The computational geometry and the most powerful test derivation are not in the paper.

There's more that can be done.

Might also use the work for other anomaly detection problems. One nice point is, although might be using lots of variables, don't encounter "over fitting".

What we were doing with expert systems was mostly just thresholds on one variable at a time. Well, in some cases, we know so much about the variable that that is okay. But there can be thousands of variables, collected by HP, Microsoft, etc., and we can't know just from personal knowledge or experience just what thresholds make good sense. And we have nothing solid on false alarm rate or how to adjust it -- yes, we could get an approximation by going back through the data. And, then we have still less on detection rate. If want a server farm that is reliable and secure, about the best you can hope for, then maybe my work would be one of the important tools and means.

This stuff about "multi-dimensional" is serious and important. E.g., suppose we have three threshold detectors. Then whether we wanted to or not, we just assumed that the 3D region of normal, healthy, not an anomaly or sick, performance is a 3D box. The box does not promise to fit reality well. Make the box too small, and get too many false alarms. Make the box bigger and can reduce the false alarm rate, but since we still have a box that does not fit reality well we are stuck with detection rate too low and a "poor" detector.

Well, can think a little and intuitively convince yourself that the region of normal (healthy) performance could be fractal, e.g., the Mandelbrot set, and still the math should work.

To me, for monitoring server farms and networks, my little paper totally blew out of the water with the doors blown off nearly everything we were doing with expert systems.

Let us all know what utility you find in the paper!

Gary was also one of the first and primary authors of CLIPS.

If I recall correctly, a summer intern in the AI section under the Mission Planning Directorate, whose name I have forgotten, developed a prototype or a proof-of-concept of CLIPS. At the time, 1984-ish, the AI group at JSC had many expensive Symbolics and Lisp Machine computers running ART. The intern's project showed that a pattern matching rules based engine could be developed in a conventional programing language for distribution to traditional cost effective hardware.

The challenge is scale, if you aren't using a very flat and controlled (database like) rule system - so more like a knowledge store than a reasoning system - then you run into big problems of complexity real fast. But even worse is the brittleness. Everything is fine, until it suddenly isn't and then you discover you can't touch anything without everything blowing up.

On the other hand, if you have a system of case statements or if thens that is a bit opaque (or very opaque) refactoring them to use an expert system is a very good idea.

And, if you have a homebrew knowledge store then using a rule system is also a good idea - but now-a-days you might want to try building a knowledge graph instead, mostly because that's v.trendy.