Live data from Hacker News

Outlier Detection in SQL

periscopedata.com

31–36 of 36 posts

Re: Outlier Detection in SQL

#31
post #26

I've recently been doing a bunch of stuff with sports stats, which involves lots of GIS data. This sort of thing comes up a lot - trying to find a player's 'territory' based on coordinates of their actions in a game, without including outlying events that cause you to overestimate the area. There's a concept in animal behaviour called a 'home range' which is more or less the same thing - GPS attached to tigers in the…

I hope it works out but they need to do a lot of work on it between these CTPs and RTM. They've made a critical mistake in the way they've marketed and documented it.

* They call every use a "data scientist". This is fun and assuredly deserved for R experts but is going to scare a hell of a lot of potential users away.

* The installation notes give you a lot of steps but zero insight into why you're doing them. Install these two packages; okay but what exactly are they doing? We need to know this kind of thing. R people might know...

* There's very little troubleshooting documentation either on the installation/initial setup (and there are a LOT of bugs in getting it going; trust me I spent most of last week digging through them) but also once it's operational how it's going to be tracked and managed within the context of everything else that's going on in the SQL instance. Like the memory pools, wait states, resource governor pools, etc. We kind of need to know this stuff. Otherwise we see a server that gets reported as slow, we know it uses R, but we don't know where to look to determine whether R is causing it or not and what we can do about it.

* AFAIK it's single-threaded. Considering most places have super-downsized their CPUs and are going for massively threaded servers these days (which is arguable given Microsoft's 2012+ per-core licensing model; but I'm talking about servers in very big enterprises that have unlimited licensing agreements), that's not going to end well. It's very possible this will be fixed before RTM though.

* And finally, because they've bought revolution and rebranded it as Microsoft R a few days ago, and now it's split into SQL and non-SQL products, there's going to be a lot of confusion and a long wait for appropriate training materials to catch up for all of us non-R users.

* I haven't been impressed with the current tutorial materials. As a non-R user we're going to need a lot more to really understand how to use it. I'm sure there are fantastic R resources available but if you want to understand it with a Microsoft SQL Server background then it's a slightly different story...

Re: Outlier Detection in SQL

#32

Standard deviations assume normal distributions... but ultimately when you look for some two standard deviations from the mean, you're just look for an event that only occurs 95% of the time. A technique that works regardless of distribution is percentiles. Postgesql supports these now as well.

I wish they went into more detail within the context of what we're discussing. I've looked at the wiki pages and my eyes glazed over.

Trying to pick out "bad" stats from the gigabytes of information I gather from the servers I manage is always a top-of-the-list thing that I want to do.

But when we're talking about 95%; well I mean it's fine for showing on a graph but how can you run alerting for something like that?

Re: Outlier Detection in SQL

#33

These are some cool SQL tricks! I like it. The big caveat with the standard deviation technique is that it assumes a normal distribution. Many datasets are not actually distributed normally (power-law, Poisson, beta, etc, etc) and so the technique won't work. It's a much harder problem to 'generically' detect outliers without knowledge of the underlying distribution. I don't have any idea how to do it (though a forme…

This is a very good point (to me as someone without any stats knowledge).

I think when the article begins it should first provide code that can validate that the values you're going to give the function will fit within a normal distribution that makes the outlier detection worthwhile. Is that possible?

Re: Outlier Detection in SQL

#34
post #26

I've recently been doing a bunch of stuff with sports stats, which involves lots of GIS data. This sort of thing comes up a lot - trying to find a player's 'territory' based on coordinates of their actions in a game, without including outlying events that cause you to overestimate the area. There's a concept in animal behaviour called a 'home range' which is more or less the same thing - GPS attached to tigers in the…

Interesting - I just helped on a similar project that showed football (soccer) player zones of control using voronoi diagrams for a CES demo: http://www.geekwire.com/2016/sensoria-microsoft-soccer-colla...

We used a custom PowerBI visualization (d3 + TypeScript). For dev purposes I just averaged data with some simple SQL queries, but the real thing used (I believe) 10k datapoints fore each player over the course of the match.

Re: Outlier Detection in SQL

#35
post #31
post #26

I've recently been doing a bunch of stuff with sports stats, which involves lots of GIS data. This sort of thing comes up a lot - trying to find a player's 'territory' based on coordinates of their actions in a game, without including outlying events that cause you to overestimate the area. There's a concept in animal behaviour called a 'home range' which is more or less the same thing - GPS attached to tigers in the…

I hope it works out but they need to do a lot of work on it between these CTPs and RTM. They've made a critical mistake in the way they've marketed and documented it. * They call every use a "data scientist". This is fun and assuredly deserved for R experts but is going to scare a hell of a lot of potential users away. * The installation notes give you a lot of steps but zero insight into why you're doing them. Insta…

Can you easily fit things like glm across a view or large table with the SQL Server R addon?

Re: Outlier Detection in SQL

#36
post #31
post #26

I've recently been doing a bunch of stuff with sports stats, which involves lots of GIS data. This sort of thing comes up a lot - trying to find a player's 'territory' based on coordinates of their actions in a game, without including outlying events that cause you to overestimate the area. There's a concept in animal behaviour called a 'home range' which is more or less the same thing - GPS attached to tigers in the…

I hope it works out but they need to do a lot of work on it between these CTPs and RTM. They've made a critical mistake in the way they've marketed and documented it. * They call every use a "data scientist". This is fun and assuredly deserved for R experts but is going to scare a hell of a lot of potential users away. * The installation notes give you a lot of steps but zero insight into why you're doing them. Insta…

[deleted]
Post reply on HN