Using R to detect fraud at 1M transactions per second [video] • Hacker News Board

Re: Using R to detect fraud at 1M transactions per second [video]

#61

Post by Jweb_Guru » Sat, Oct 01, 2016, 6:29 PM UTC

blahi wrote: post #55

Jweb_Guru wrote: post #53

Earlier quoted context omitted.

That may be, but it's not clear to me from the article. What's the feature I'm missing that makes the SQL Server offering so much more compelling?

Out of core algorithms, attatchment of data straight into R for a couple of very important things.

> Out of core algorithms.

Does this just mean third-party modules? If so, doesn't http://www.joeconway.com/plr/doc/plr-module-funcs.html suffice?

> attatchment of data straight into R for a couple of very important things.

Isn't this doable through http://www.joeconway.com/plr/doc/plr-global-data.html?

Apologies if these are dumb questions, as I'm not very familiar with R.

Re: Using R to detect fraud at 1M transactions per second [video]

#62

Post by blahi » Sun, Oct 02, 2016, 6:55 AM UTC

Jweb_Guru wrote: post #61

blahi wrote: post #55

Earlier quoted context omitted.

Out of core algorithms, attatchment of data straight into R for a couple of very important things.

> Out of core algorithms. Does this just mean third-party modules? If so, doesn't http://www.joeconway.com/plr/doc/plr-module-funcs.html suffice? > attatchment of data straight into R for a couple of very important things. Isn't this doable through http://www.joeconway.com/plr/doc/plr-global-data.html ? Apologies if these are dumb questions, as I'm not very familiar with R.

No. They have their own highly optimized algorithms. They also have their their own distributed data structures.

I am not sure about that 2nd link. Seems like it's just UDFs written in R.

Re: Using R to detect fraud at 1M transactions per second [video]

#63

Post by Jweb_Guru » Sun, Oct 02, 2016, 5:02 PM UTC

blahi wrote: post #62

Jweb_Guru wrote: post #61

Earlier quoted context omitted.

> Out of core algorithms. Does this just mean third-party modules? If so, doesn't http://www.joeconway.com/plr/doc/plr-module-funcs.html suffice? > attatchment of data straight into R for a couple of very important things. Isn't this doable through http://www.joeconway.com/plr/doc/plr-global-data.html ? Apologies if these are dumb questions, as I'm not very familiar with R.

No. They have their own highly optimized algorithms. They also have their their own distributed data structures. I am not sure about that 2nd link. Seems like it's just UDFs written in R.

Ah, okay. It wasn't clear to me that Microsoft had written its own algorithms. That does seem very useful, then (though presumably those algorithms and data structures could be used outside of the SQL Server environment, I presume that Microsoft is using them to encourage people to use SQL Server rather than another solution).

I believe the second link is referring to being able to initialize and share data between functions within the R runtime (rather than having to transfer back and forth between Postgres and the runtime). Is that not what you were referring to?

Re: Using R to detect fraud at 1M transactions per second [video]

#64

Post by blahi » Sun, Oct 02, 2016, 5:39 PM UTC

Jweb_Guru wrote: post #63

blahi wrote: post #62

Earlier quoted context omitted.

No. They have their own highly optimized algorithms. They also have their their own distributed data structures. I am not sure about that 2nd link. Seems like it's just UDFs written in R.

Ah, okay. It wasn't clear to me that Microsoft had written its own algorithms. That does seem very useful, then (though presumably those algorithms and data structures could be used outside of the SQL Server environment, I presume that Microsoft is using them to encourage people to use SQL Server rather than another solution). I believe the second link is referring to being able to initialize and share data between f…

>being able to initialize and share data between functions within the R runtime

That's right.

>rather than having to transfer back and forth between Postgres and the runtime

But that's not. There's a difference between being able to use data outside of the database (from the R runtime) in my UDFs (executed in Postgres) on one hand and being able to attach 2TBs of data straight from an SQL table in the R runtime on the other. I don't even care that much about the algorithms. Moving the data is the bottleneck most of the time. And Microsoft is actually late to the party (but better than never). Oracle, Netezza, Vertica and Hana have been able to do it for quite a while now.

You are spot on about being able to use the algorithms outside of SQL Server. You can use them on Teradata or Hadoop or rent your own VMs on Azure to use them or you can buy standalone licenses too.

Re: Using R to detect fraud at 1M transactions per second [video]

#65

Post by Jweb_Guru » Sun, Oct 02, 2016, 7:29 PM UTC

blahi wrote: post #64

Jweb_Guru wrote: post #63

Earlier quoted context omitted.

Ah, okay. It wasn't clear to me that Microsoft had written its own algorithms. That does seem very useful, then (though presumably those algorithms and data structures could be used outside of the SQL Server environment, I presume that Microsoft is using them to encourage people to use SQL Server rather than another solution). I believe the second link is referring to being able to initialize and share data between f…

>being able to initialize and share data between functions within the R runtime That's right. >rather than having to transfer back and forth between Postgres and the runtime But that's not. There's a difference between being able to use data outside of the database (from the R runtime) in my UDFs (executed in Postgres) on one hand and being able to attach 2TBs of data straight from an SQL table in the R runtime on th…

So by "attach 2 TB of data straight from an SQL table into the R runtime" you mean that Microsoft taught R to interact directly with SQL Server's storage engine? If so, I agree, data movement is almost always the bottleneck for large data sets, and I don't think PL/R can do that (though I am not sure if that's a necessity due to the way Postgres's language plugins work, or something that could be done with enough effort).

However, if all you mean is that SQL Server can transfer the data a tuple at a time to R on the same server (in memory), I believe that PL/R and Postgres interact like that already (again, maybe I'm wrong). And I don't know how much extra overhead that provides over talking directly to the storage engine, anyway.

Re: Using R to detect fraud at 1M transactions per second [video]

#66

Post by blahi » Sun, Oct 02, 2016, 9:12 PM UTC

Jweb_Guru wrote: post #65

blahi wrote: post #64

Earlier quoted context omitted.

>being able to initialize and share data between functions within the R runtime That's right. >rather than having to transfer back and forth between Postgres and the runtime But that's not. There's a difference between being able to use data outside of the database (from the R runtime) in my UDFs (executed in Postgres) on one hand and being able to attach 2TBs of data straight from an SQL table in the R runtime on th…

So by "attach 2 TB of data straight from an SQL table into the R runtime" you mean that Microsoft taught R to interact directly with SQL Server's storage engine? If so, I agree, data movement is almost always the bottleneck for large data sets, and I don't think PL/R can do that (though I am not sure if that's a necessity due to the way Postgres's language plugins work, or something that could be done with enough eff…

>Microsoft taught R to interact directly with SQL Server's storage engine

They have created 2 new services for SQL Server 2016 - BxlServer and SQL Satellite which facilitate the communication and data exchange. They obviously have additional speedups for the proprietary runtime (that was one of the main selling points of the company they acquired - fast data access to several RDBMS), but it's plenty fast for regular R too.

https://msdn.microsoft.com/en-us/library/mt709082.aspx

Re: Using R to detect fraud at 1M transactions per second [video]

#67

Post by dandermotj » Sun, Oct 02, 2016, 9:41 PM UTC

vegabook wrote: post #48

dandermotj wrote: post #47

Earlier quoted context omitted.

You're not wrong and absolutely totally wrong at the same time. R is the furthest thing from C you could find in paradigm, syntax and performance, but yes much of the underlying code is C or Fortran. But really you're missing the point. R's purpose is interactive, exploratory and scientific computing and that's what it is incredibly good at. It wasn't intended for high performance computing, but there are ways of get…

So well put. But what is Rho? Intrigued...

https://github.com/rho-devel/rho

Re: Using R to detect fraud at 1M transactions per second [video]

#68

Post by 0x001E84EE » Mon, Oct 03, 2016, 7:11 PM UTC

apohn wrote: post #24

vasaulys wrote: post #2

Does anybody use R in production services or just for exploratory work? It seems that once you figure out a good model in R, its almost always rewritten into either Scala or Java for real production work.

I used to work in the consulting arm of a software firm and we wrote and deployed R code in production at many Fortune 500 companies. We worked in almost every industry. I spent quite a bit of time refactoring bad R code so it could run reliably in a production environment. There is a ton of bad R code out there that barely works for exploratory analysis, let alone a production environment. So yes, R is used in produ…

R and its libraries are GPL licensed. Is there some corporate license available to prevent companies from being required to publish proprietary code that interacts with R? Or was the usage limited to to internal systems?

Re: Using R to detect fraud at 1M transactions per second [video]

#69

Post by vijucat » Tue, Oct 04, 2016, 7:46 PM UTC

apohn wrote: post #51

vijucat wrote: post #35

Earlier quoted context omitted.

Did you guys separate out the R process (or multiple processes?) from the rest of the transaction-processing / other server infrastructure or embed the REngine (which sounds like a bad idea to me; incorrect data serialization can easily crash the whole process)? What is a stable way to connect (and reconnect!) to R, assuming it was a separate process? I would think that an indirect communication path, such as Server…

We used separate workflows depending on if the data was streaming or batch oriented (e.g. on-demand or triggered by a user). First I'll talk about batch oriented jobs. The company I worked for had a tomcat based product that exposed R via a RESTful API. It was similar to what you get from AzureML now, except it was on-premise. So basically we would call out to this and configure it to restart R sessions if they crash…

Thanks, this is all quite useful! I faced crashes with REngine + rJava, too, and thought of a DB as a intermediary, but your in-memory DB idea adds an interesting twist that adds performance, too.

Re: Using R to detect fraud at 1M transactions per second [video]

#70

Post by vegabook » Thu, Oct 06, 2016, 9:32 AM UTC

ignasl wrote: post #59

vegabook wrote: post #37

Earlier quoted context omitted.

this is not software engineering or production. It is batch jobs / exploratory analysis. It requires little or no structure apart from the analysis itself. also in anything that has not been coded in C directly underneath, Python is 20x faster and C is 500× faster. R is literally the slowest mainstream language today by a long shot. That's a key consideration for production.

Where did you get those numbers from? They are most definitely wrong unless you don't vectorize your code and run loops all around. A lot of R is actually written in C so you can squeeze really good performance if you know what you are doing. I would recommend reading Hadley's Advanced R and profile your code, I think you might be pleasantly surprised.

I make extensive use of vectorization and use as many calls as I possibly can to the built-ins and/or c-based libraries. However as you well know, part of the fun in R is applying your own functions and unless you write these in C, you're back to native R and that's tediously slow. Ggplot another culprit -> amazing library, but if you're chucking out large amounts of custom charts with it it takes ages. Base graphics an order of magnitude faster (if less pretty and convenient for axis training).