Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using R to detect fraud at 1M transactions per second [video]

blog.revolutionanalytics.com

Post reply on HN
Print view View original

41–50 of 72 posts

Re: Using R to detect fraud at 1M transactions per second [video]

#41

Post by baldfat »

kgwgk wrote: post #26
baldfat wrote: post #16

Earlier quoted context omitted.

That seems like you could simply use http://jupyter.org/ and just run the script with R code inline. http://blog.revolutionanalytics.com/2016/01/pipelining-r-pyt... Also why not just switch to Pandas it really is a pretty close R clone.

"Pretty close" as long as you stay within the region of common functionality. I wouldn't say it's a clone.

That is true. I actually started my journey with Pandas and then switched to R for the ecco-system and zero based for data science drove me nuts.

But I do feel that the goal is a clone.

"Python has long been great for data munging and preparation, but less so for data analysis and modeling. pandas helps fill this gap, enabling you to carry out your entire data analysis workflow in Python without having to switch to a more domain specific language like R." http://pandas.pydata.org/

Re: Using R to detect fraud at 1M transactions per second [video]

#42

Post by blahi »

sandGorgon wrote: post #39
blahi wrote: post #33

Earlier quoted context omitted.

I have experience scoring ~ 1TB daily. And a lot of smaller data sets spanning a few hundred gigs. It's not "hyper performant". Obviously doing things in scala or C++ will be faster. However rewriting the models would take months and an entirely different set of skills. That means separate people. But if somebody says that they use Python instead of R for the speed... that's just bull. For example one of the fundamen…

could you talk about some of the learnings you had around scoring 1tb daily in R ? How do you even load the data into memory ? is it read from a database or s3 files.

In that particular case, I used Vertica which loads data in R really, really fast and straight up use a very big machine.

That's not how I approach it most of the time though. I mostly use out-of-memory algorithms, sometimes open source, sometimes Revolution's (now Microsoft). They process things in chunks. You can see BigLM and SpeedGLM for quick examples. h2o is also very popular platform. You should probably check the High Performance Comptuing CRAN Task View.

I have also used Netezza and Hana and both worked well for the purpose. There's also Teradata Aster but I don't have experience with it. There's also the open-source MonetDB which has in-database R threads and also an r package similar to rsqlite.

There are also map/reduce packages for Hadoop.

Re: Using R to detect fraud at 1M transactions per second [video]

#43

Post by vegabook »

blahi wrote: post #29
vegabook wrote: post #28

Earlier quoted context omitted.

R just does not have robust software engineering tools for anything that even begins to resemble scale and anybody who says otherwise is denying reality. R can certainly be used in production but the skeleton framework cannot be R. RPC only in my experience with all the structure with something else. R is intrinsically single user / batch with maybe shared database but say goodbye to anything that even starts to appr…

Riiight. Everybody else is a bad engineer and you are the good one. With the single threaded R code... edit: The comment above has been extended quite a bit. Initially it was a single (abrasive) sentence. I still stand by my answer however. Somebody who did not turn on multi-threading does not get to criticize R. It is the first thing you learn in any book about R. You have to be almost actively avoiding learning abo…

perhaps you might not have started your own comment with the erroneous view that 'bad coders' are to blame when R proves to be deficient at extra-design tasks.

Oh I further note your R consulting vocation. There you go. Vested interest.

BTW, I love R. But my love is not blind.

Re: Using R to detect fraud at 1M transactions per second [video]

#44

Post by vegabook »

sandGorgon wrote: post #38
vegabook wrote: post #19

Earlier quoted context omitted.

I have 20k lines of (my own) R code running in production (used intensively by a salesforce of up to 20 people who price bonds with it) and it's an unmitigated nightmare to manage. Slow as crazy. No threading to manage concurrency so constant batch jobs everywhere. Memory hog. On Windows (this is finance), unfortunate fairly frequent crashes. No real time feeds due to the horrible architecture of the interpreter. Tha…

have you considered spark instead of flink/concord ?

I have, thanks for asking. I must admit that I have a very real priority on (soft) real time. Flink appears attractive but I also have a slight bias to non JVM which is where Concord appears interesting. I also just love Concord's "hot" DAG capability. Agreed (I think) though that I must include Spark micro-batching as a potential candidate. Any experience you have on this...I welcome links/tips. As you can probably tell I am at the very initial exploratory stage on stack choice.

Re: Using R to detect fraud at 1M transactions per second [video]

#45

Post by apathy »

baldfat wrote: post #18
buro9 wrote: post #12

Earlier quoted context omitted.

> If Microsoft also has an optimized way to get data from SQL Server to R I can see how they got a 100X speedup. In certain cases using the MKL libraries can give you that as well, but I suspect the speedup just comes from improving the data transfer method. The optimized method is that you can run R inside the database in the latest version of SQL Server. I've actually installed Windows again, just to play with this…

> I've actually installed Windows again, just to play with this feature (though I cannot make claims to actually putting it to good use yet). I heard the Linux SQL Server is surprisingly decent. https://blogs.microsoft.com/blog/2016/03/07/announcing- sql-server-on-linux/

postgresql used to have a way to embed R into PostGreSQL too:

https://github.com/jconway/plr

I guess it still does, though I haven't used it in years. You can, of course, do the same thing with Python.

Re: Using R to detect fraud at 1M transactions per second [video]

#46

Post by apathy »

blahi wrote: post #42
sandGorgon wrote: post #39

Earlier quoted context omitted.

could you talk about some of the learnings you had around scoring 1tb daily in R ? How do you even load the data into memory ? is it read from a database or s3 files.

In that particular case, I used Vertica which loads data in R really, really fast and straight up use a very big machine. That's not how I approach it most of the time though. I mostly use out-of-memory algorithms, sometimes open source, sometimes Revolution's (now Microsoft). They process things in chunks. You can see BigLM and SpeedGLM for quick examples. h2o is also very popular platform. You should probably check…

I never would have tried MonetDb (ok monetdblite) if not for this great little tutorial on how to load all of SEER into it:

http://www.asdfree.com/2013/07/analyze-surveillance-epidemio...

Yeah the presentation and code isn't beautiful, but it does avoid the need to WRITE THE DAMNED THING YOURSELF, which some people apparently will never understand (although they will once they are unemployed). More importantly, it turns out you don't necessarily need Vertica for fast out-of-core loading and processing.

Granted, there are plenty of other ways to work out of core (hdf5, bigMatrix, any random database, blah blah) but this was one that was new to me. And I like it.

Re: Using R to detect fraud at 1M transactions per second [video]

#47

Post by dandermotj »

md2be wrote: post #34
vegabook wrote: post #19

Earlier quoted context omitted.

I have 20k lines of (my own) R code running in production (used intensively by a salesforce of up to 20 people who price bonds with it) and it's an unmitigated nightmare to manage. Slow as crazy. No threading to manage concurrency so constant batch jobs everywhere. Memory hog. On Windows (this is finance), unfortunate fairly frequent crashes. No real time feeds due to the horrible architecture of the interpreter. Tha…

R is just S which is just C

You're not wrong and absolutely totally wrong at the same time. R is the furthest thing from C you could find in paradigm, syntax and performance, but yes much of the underlying code is C or Fortran.

But really you're missing the point. R's purpose is interactive, exploratory and scientific computing and that's what it is incredibly good at. It wasn't intended for high performance computing, but there are ways of getting it there. Look out for Rho in the future.

Re: Using R to detect fraud at 1M transactions per second [video]

#48

Post by vegabook »

dandermotj wrote: post #47
md2be wrote: post #34

Earlier quoted context omitted.

R is just S which is just C

You're not wrong and absolutely totally wrong at the same time. R is the furthest thing from C you could find in paradigm, syntax and performance, but yes much of the underlying code is C or Fortran. But really you're missing the point. R's purpose is interactive, exploratory and scientific computing and that's what it is incredibly good at. It wasn't intended for high performance computing, but there are ways of get…

So well put. But what is Rho? Intrigued...

Re: Using R to detect fraud at 1M transactions per second [video]

#49

Post by [deleted] »

vegabook wrote: post #43
blahi wrote: post #29

Earlier quoted context omitted.

Riiight. Everybody else is a bad engineer and you are the good one. With the single threaded R code... edit: The comment above has been extended quite a bit. Initially it was a single (abrasive) sentence. I still stand by my answer however. Somebody who did not turn on multi-threading does not get to criticize R. It is the first thing you learn in any book about R. You have to be almost actively avoiding learning abo…

perhaps you might not have started your own comment with the erroneous view that 'bad coders' are to blame when R proves to be deficient at extra-design tasks. Oh I further note your R consulting vocation. There you go. Vested interest. BTW, I love R. But my love is not blind.

[deleted]

Re: Using R to detect fraud at 1M transactions per second [video]

#50

Post by blahi »

Jweb_Guru wrote: post #36
buro9 wrote: post #12

Earlier quoted context omitted.

> If Microsoft also has an optimized way to get data from SQL Server to R I can see how they got a 100X speedup. In certain cases using the MKL libraries can give you that as well, but I suspect the speedup just comes from improving the data transfer method. The optimized method is that you can run R inside the database in the latest version of SQL Server. I've actually installed Windows again, just to play with this…

No need to install Windows to get R in a database, you can run with PL/R on Postgres (unless you have a particular desire to run it within SQL Server, of course, but doesn't it run on Linux now?). http://www.joeconway.com/plr/

That's a lot less than what Microsoft is offering.
Post reply on HN