Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using R to detect fraud at 1M transactions per second [video]

blog.revolutionanalytics.com

Post reply on HN
Print view View original

21–30 of 72 posts

Re: Using R to detect fraud at 1M transactions per second [video]

#21

Post by blahi »

vegabook wrote: post #19
vasaulys wrote: post #2

Does anybody use R in production services or just for exploratory work? It seems that once you figure out a good model in R, its almost always rewritten into either Scala or Java for real production work.

I have 20k lines of (my own) R code running in production (used intensively by a salesforce of up to 20 people who price bonds with it) and it's an unmitigated nightmare to manage. Slow as crazy. No threading to manage concurrency so constant batch jobs everywhere. Memory hog. On Windows (this is finance), unfortunate fairly frequent crashes. No real time feeds due to the horrible architecture of the interpreter. Tha…

That sounds like bad coders, not that R is bad.

Evidenced by:

>No threading to manage concurrency

R is used in production at EA, Activision, Ebay, Trulia, Google, Microsoft and many, many more. Those are just the ones I've seen give talks about scoring >1TBs regularly with R.

Every time somebody says R can't do be used for large data sets or is slow, I ask for more details and almost universally the programmer's complete lack of initiative is the weak link.

Re: Using R to detect fraud at 1M transactions per second [video]

#22

Post by blahi »

baldfat wrote: post #16
nerdponx wrote: post #7

Earlier quoted context omitted.

I do it at my company. I prototype in R, and then end up having to rewrite chunks of it in Python so it can be worked into our application, which right now is exclusively Python. It's not a matter of performance, it's just because it would be an enormous amount of engineering overhead to start calling R from inside the Python app

That seems like you could simply use http://jupyter.org/ and just run the script with R code inline. http://blog.revolutionanalytics.com/2016/01/pipelining-r-pyt... Also why not just switch to Pandas it really is a pretty close R clone.

How much experience do you have in statistical computing, out of curiosity?

Re: Using R to detect fraud at 1M transactions per second [video]

#23

Post by dhd415 »

The presentation is a little light on the technical details of how the demo was run. What I could get from the presentation was 1M fraud predictions/sec via R stored procedures on data streaming into SQL Server 2016 stored in in-memory column store tables on a 4-socket "commodity" server.

Re: Using R to detect fraud at 1M transactions per second [video]

#24

Post by apohn »

vasaulys wrote: post #2

Does anybody use R in production services or just for exploratory work? It seems that once you figure out a good model in R, its almost always rewritten into either Scala or Java for real production work.

I used to work in the consulting arm of a software firm and we wrote and deployed R code in production at many Fortune 500 companies. We worked in almost every industry.

I spent quite a bit of time refactoring bad R code so it could run reliably in a production environment. There is a ton of bad R code out there that barely works for exploratory analysis, let alone a production environment.

So yes, R is used in production environment in a lot of places.

Re: Using R to detect fraud at 1M transactions per second [video]

#25

Post by nerdponx »

baldfat wrote: post #16
nerdponx wrote: post #7

Earlier quoted context omitted.

I do it at my company. I prototype in R, and then end up having to rewrite chunks of it in Python so it can be worked into our application, which right now is exclusively Python. It's not a matter of performance, it's just because it would be an enormous amount of engineering overhead to start calling R from inside the Python app

That seems like you could simply use http://jupyter.org/ and just run the script with R code inline. http://blog.revolutionanalytics.com/2016/01/pipelining-r-pyt... Also why not just switch to Pandas it really is a pretty close R clone.

It has nothing to do with interoperability on my machine. I use notebooks (and Pandas) all the time, and I consider myself fluent in bith R and Python.

It's because R is a substantial engineering dependency. As I said, our entire stack is Python and Node. Yes, you can call R from Python using Rpy2, but that's a pro-bono project maintained largely by one person. It's great for casual use, but there is far too much risk to start talking about building critical business code around it.

Re: Using R to detect fraud at 1M transactions per second [video]

#26

Post by kgwgk »

baldfat wrote: post #16
nerdponx wrote: post #7

Earlier quoted context omitted.

I do it at my company. I prototype in R, and then end up having to rewrite chunks of it in Python so it can be worked into our application, which right now is exclusively Python. It's not a matter of performance, it's just because it would be an enormous amount of engineering overhead to start calling R from inside the Python app

That seems like you could simply use http://jupyter.org/ and just run the script with R code inline. http://blog.revolutionanalytics.com/2016/01/pipelining-r-pyt... Also why not just switch to Pandas it really is a pretty close R clone.

"Pretty close" as long as you stay within the region of common functionality. I wouldn't say it's a clone.

Re: Using R to detect fraud at 1M transactions per second [video]

#27

Post by kgwgk »

blahi wrote: post #21
vegabook wrote: post #19

Earlier quoted context omitted.

I have 20k lines of (my own) R code running in production (used intensively by a salesforce of up to 20 people who price bonds with it) and it's an unmitigated nightmare to manage. Slow as crazy. No threading to manage concurrency so constant batch jobs everywhere. Memory hog. On Windows (this is finance), unfortunate fairly frequent crashes. No real time feeds due to the horrible architecture of the interpreter. Tha…

That sounds like bad coders, not that R is bad. Evidenced by: >No threading to manage concurrency R is used in production at EA, Activision, Ebay, Trulia, Google, Microsoft and many, many more. Those are just the ones I've seen give talks about scoring >1TBs regularly with R. Every time somebody says R can't do be used for large data sets or is slow, I ask for more details and almost universally the programmer's comp…

Excel is used in production very widely, but I'm sure we all agree it has its limitations.

Re: Using R to detect fraud at 1M transactions per second [video]

#28

Post by vegabook »

blahi wrote: post #21
vegabook wrote: post #19

Earlier quoted context omitted.

I have 20k lines of (my own) R code running in production (used intensively by a salesforce of up to 20 people who price bonds with it) and it's an unmitigated nightmare to manage. Slow as crazy. No threading to manage concurrency so constant batch jobs everywhere. Memory hog. On Windows (this is finance), unfortunate fairly frequent crashes. No real time feeds due to the horrible architecture of the interpreter. Tha…

That sounds like bad coders, not that R is bad. Evidenced by: >No threading to manage concurrency R is used in production at EA, Activision, Ebay, Trulia, Google, Microsoft and many, many more. Those are just the ones I've seen give talks about scoring >1TBs regularly with R. Every time somebody says R can't do be used for large data sets or is slow, I ask for more details and almost universally the programmer's comp…

R just does not have robust software engineering tools for anything that even begins to resemble scale and anybody who says otherwise is denying reality. R can certainly be used in production but the skeleton framework cannot be R. RPC only in my experience with all the structure with something else. R is intrinsically single user / batch with maybe shared database but say goodbye to anything that even starts to approach real time, or multi-node dependent. In my experience the only people who insist that R is robust for production, inevitably have a vested interest. Any objective programmer can see its greatness but also its glaring flaws.

Re: Using R to detect fraud at 1M transactions per second [video]

#29

Post by blahi »

vegabook wrote: post #28
blahi wrote: post #21

Earlier quoted context omitted.

That sounds like bad coders, not that R is bad. Evidenced by: >No threading to manage concurrency R is used in production at EA, Activision, Ebay, Trulia, Google, Microsoft and many, many more. Those are just the ones I've seen give talks about scoring >1TBs regularly with R. Every time somebody says R can't do be used for large data sets or is slow, I ask for more details and almost universally the programmer's comp…

R just does not have robust software engineering tools for anything that even begins to resemble scale and anybody who says otherwise is denying reality. R can certainly be used in production but the skeleton framework cannot be R. RPC only in my experience with all the structure with something else. R is intrinsically single user / batch with maybe shared database but say goodbye to anything that even starts to appr…

Riiight. Everybody else is a bad engineer and you are the good one. With the single threaded R code...

edit: The comment above has been extended quite a bit. Initially it was a single (abrasive) sentence. I still stand by my answer however. Somebody who did not turn on multi-threading does not get to criticize R. It is the first thing you learn in any book about R. You have to be almost actively avoiding learning about it. It's in every 3rd blog post and SO question.

Re: Using R to detect fraud at 1M transactions per second [video]

#30

Post by baldfat »

nerdponx wrote: post #25
baldfat wrote: post #16

Earlier quoted context omitted.

That seems like you could simply use http://jupyter.org/ and just run the script with R code inline. http://blog.revolutionanalytics.com/2016/01/pipelining-r-pyt... Also why not just switch to Pandas it really is a pretty close R clone.

It has nothing to do with interoperability on my machine. I use notebooks (and Pandas) all the time, and I consider myself fluent in bith R and Python. It's because R is a substantial engineering dependency. As I said, our entire stack is Python and Node. Yes, you can call R from Python using Rpy2, but that's a pro-bono project maintained largely by one person. It's great for casual use, but there is far too much ris…

So why not Pandas?
Post reply on HN