Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using R to detect fraud at 1M transactions per second [video]

blog.revolutionanalytics.com

Post reply on HN
Print view View original

31–40 of 72 posts

Re: Using R to detect fraud at 1M transactions per second [video]

#31

Post by nerdponx »

blahi wrote: post #21
vegabook wrote: post #19

Earlier quoted context omitted.

I have 20k lines of (my own) R code running in production (used intensively by a salesforce of up to 20 people who price bonds with it) and it's an unmitigated nightmare to manage. Slow as crazy. No threading to manage concurrency so constant batch jobs everywhere. Memory hog. On Windows (this is finance), unfortunate fairly frequent crashes. No real time feeds due to the horrible architecture of the interpreter. Tha…

That sounds like bad coders, not that R is bad. Evidenced by: >No threading to manage concurrency R is used in production at EA, Activision, Ebay, Trulia, Google, Microsoft and many, many more. Those are just the ones I've seen give talks about scoring >1TBs regularly with R. Every time somebody says R can't do be used for large data sets or is slow, I ask for more details and almost universally the programmer's comp…

Do you have personal experience with this kind of hyper-performant R code?

Re: Using R to detect fraud at 1M transactions per second [video]

#32

Post by nerdponx »

baldfat wrote: post #30
nerdponx wrote: post #25

Earlier quoted context omitted.

It has nothing to do with interoperability on my machine. I use notebooks (and Pandas) all the time, and I consider myself fluent in bith R and Python. It's because R is a substantial engineering dependency. As I said, our entire stack is Python and Node. Yes, you can call R from Python using Rpy2, but that's a pro-bono project maintained largely by one person. It's great for casual use, but there is far too much ris…

So why not Pandas?

Personal preference. I switch back-and-forth based on the project.

R data frames are native and feel native. Pandas data frames are non-native and can be a pain in the ass to work with.

That, and there is a lot mpre to the decision than just which data frame implementation I like better.

Re: Using R to detect fraud at 1M transactions per second [video]

#33

Post by blahi »

nerdponx wrote: post #31
blahi wrote: post #21

Earlier quoted context omitted.

That sounds like bad coders, not that R is bad. Evidenced by: >No threading to manage concurrency R is used in production at EA, Activision, Ebay, Trulia, Google, Microsoft and many, many more. Those are just the ones I've seen give talks about scoring >1TBs regularly with R. Every time somebody says R can't do be used for large data sets or is slow, I ask for more details and almost universally the programmer's comp…

Do you have personal experience with this kind of hyper-performant R code?

I have experience scoring ~ 1TB daily. And a lot of smaller data sets spanning a few hundred gigs.

It's not "hyper performant". Obviously doing things in scala or C++ will be faster. However rewriting the models would take months and an entirely different set of skills. That means separate people.

But if somebody says that they use Python instead of R for the speed... that's just bull. For example one of the fundamental building blocks, pandas is slower than the counterpart in R.

Re: Using R to detect fraud at 1M transactions per second [video]

#34

Post by md2be »

vegabook wrote: post #19
vasaulys wrote: post #2

Does anybody use R in production services or just for exploratory work? It seems that once you figure out a good model in R, its almost always rewritten into either Scala or Java for real production work.

I have 20k lines of (my own) R code running in production (used intensively by a salesforce of up to 20 people who price bonds with it) and it's an unmitigated nightmare to manage. Slow as crazy. No threading to manage concurrency so constant batch jobs everywhere. Memory hog. On Windows (this is finance), unfortunate fairly frequent crashes. No real time feeds due to the horrible architecture of the interpreter. Tha…

R is just S which is just C

Re: Using R to detect fraud at 1M transactions per second [video]

#35

Post by vijucat »

apohn wrote: post #24
vasaulys wrote: post #2

Does anybody use R in production services or just for exploratory work? It seems that once you figure out a good model in R, its almost always rewritten into either Scala or Java for real production work.

I used to work in the consulting arm of a software firm and we wrote and deployed R code in production at many Fortune 500 companies. We worked in almost every industry. I spent quite a bit of time refactoring bad R code so it could run reliably in a production environment. There is a ton of bad R code out there that barely works for exploratory analysis, let alone a production environment. So yes, R is used in produ…

Did you guys separate out the R process (or multiple processes?) from the rest of the transaction-processing / other server infrastructure or embed the REngine (which sounds like a bad idea to me; incorrect data serialization can easily crash the whole process)?

What is a stable way to connect (and reconnect!) to R, assuming it was a separate process? I would think that an indirect communication path, such as Server Database R would work best, but I'd love to hear your battle hardened take on it.

Re: Using R to detect fraud at 1M transactions per second [video]

#36

Post by Jweb_Guru »

buro9 wrote: post #12
apohn wrote: post #10

Earlier quoted context omitted.

That statement is overly vague and sounds like marketing BS. I worked on a project where we scored streaming data in R. The biggest bottleneck was getting the data into and out of the R session. We started out using disk based I/O and ended up using using rJava so our streaming system could communicate with R. In that case we did get a 100X speed up between our first iteration and the final version which used rJava t…

> If Microsoft also has an optimized way to get data from SQL Server to R I can see how they got a 100X speedup. In certain cases using the MKL libraries can give you that as well, but I suspect the speedup just comes from improving the data transfer method. The optimized method is that you can run R inside the database in the latest version of SQL Server. I've actually installed Windows again, just to play with this…

No need to install Windows to get R in a database, you can run with PL/R on Postgres (unless you have a particular desire to run it within SQL Server, of course, but doesn't it run on Linux now?).

http://www.joeconway.com/plr/

Re: Using R to detect fraud at 1M transactions per second [video]

#37

Post by vegabook »

blahi wrote: post #33
nerdponx wrote: post #31

Earlier quoted context omitted.

Do you have personal experience with this kind of hyper-performant R code?

I have experience scoring ~ 1TB daily. And a lot of smaller data sets spanning a few hundred gigs. It's not "hyper performant". Obviously doing things in scala or C++ will be faster. However rewriting the models would take months and an entirely different set of skills. That means separate people. But if somebody says that they use Python instead of R for the speed... that's just bull. For example one of the fundamen…

this is not software engineering or production. It is batch jobs / exploratory analysis. It requires little or no structure apart from the analysis itself.

also in anything that has not been coded in C directly underneath, Python is 20x faster and C is 500× faster. R is literally the slowest mainstream language today by a long shot. That's a key consideration for production.

Re: Using R to detect fraud at 1M transactions per second [video]

#38

Post by sandGorgon »

vegabook wrote: post #19
vasaulys wrote: post #2

Does anybody use R in production services or just for exploratory work? It seems that once you figure out a good model in R, its almost always rewritten into either Scala or Java for real production work.

I have 20k lines of (my own) R code running in production (used intensively by a salesforce of up to 20 people who price bonds with it) and it's an unmitigated nightmare to manage. Slow as crazy. No threading to manage concurrency so constant batch jobs everywhere. Memory hog. On Windows (this is finance), unfortunate fairly frequent crashes. No real time feeds due to the horrible architecture of the interpreter. Tha…

have you considered spark instead of flink/concord ?

Re: Using R to detect fraud at 1M transactions per second [video]

#39

Post by sandGorgon »

blahi wrote: post #33
nerdponx wrote: post #31

Earlier quoted context omitted.

Do you have personal experience with this kind of hyper-performant R code?

I have experience scoring ~ 1TB daily. And a lot of smaller data sets spanning a few hundred gigs. It's not "hyper performant". Obviously doing things in scala or C++ will be faster. However rewriting the models would take months and an entirely different set of skills. That means separate people. But if somebody says that they use Python instead of R for the speed... that's just bull. For example one of the fundamen…

could you talk about some of the learnings you had around scoring 1tb daily in R ?

How do you even load the data into memory ? is it read from a database or s3 files.

Re: Using R to detect fraud at 1M transactions per second [video]

#40

Post by sandGorgon »

vijucat wrote: post #35
apohn wrote: post #24

Earlier quoted context omitted.

I used to work in the consulting arm of a software firm and we wrote and deployed R code in production at many Fortune 500 companies. We worked in almost every industry. I spent quite a bit of time refactoring bad R code so it could run reliably in a production environment. There is a ton of bad R code out there that barely works for exploratory analysis, let alone a production environment. So yes, R is used in produ…

Did you guys separate out the R process (or multiple processes?) from the rest of the transaction-processing / other server infrastructure or embed the REngine (which sounds like a bad idea to me; incorrect data serialization can easily crash the whole process)? What is a stable way to connect (and reconnect!) to R, assuming it was a separate process? I would think that an indirect communication path, such as Server…

i have the same question - how do you use R in production ?
Post reply on HN