Ask HN: Fast, In-Memory, Distributed data analysis and machine learning? • Hacker News Board

Re: Ask HN: Fast, In-Memory, Distributed data analysis and machine learning?

#11

Post by x0x0 » Sun, Jul 07, 2013, 6:52 AM UTC

you should check out http://0xdata.com/ ; it's built from the ground up on a custom dkv to do in-memory ML. Reasons to check it out:

1 - it's open source https://github.com/0xdata/h2o

2 - ingest data from hdfs, s3, csv

3 - I've built systems like what you're discussing twice; the ML algorithms are often easier to write than expected while data management (moving data, sending updates, etc) which initially seems easier is much harder. 0xdata handles this for you.

4 - under active development

5 - it cleanly runs on your dev box with 1 or many nodes for development; deploying is a simple as uploading a jar to a cluster and putting a single file on each naming peers in the cluster

5a - see scripts to walk you through doing this

disclosure: I work on it as of very recently =P

Re: Ask HN: Fast, In-Memory, Distributed data analysis and machine learning?

#12

Post by henrythe9th » Sun, Jul 07, 2013, 11:18 AM UTC

karterk wrote: post #10

henrythe9th wrote: post #4

Earlier quoted context omitted.

Thanks for your input. We're roughly talking around 5GB of data. Data growth should be linear in the next 6months. Money is not a big concern. Speed of iteration is key. We frequently run different processing algorithms over the entire stored dataset (stored data doesn't change) and update the calculated features each time. Not sure if this helps narrows things down. Thanks

A little bit of context: I have done a lot of hadoop, and also well aware of spark and storm. Storm is mostly well suited for handling a stream of real-time data. Spark is specifically for running iterative algorithms - it can read from HDFS, and with the expressiveness of Scala, it's great for building machine-learning related stuff. However, 5GB of data is literally nothing, and that statement holds till your data…

Thanks for the suggestion. We've actually thought about just writing a multithreaded system on a single machine. What type of in-memory storage would you recommend in this case? (which hopefully may be extended to a distributed cluster of machines if 1 really large machine becomes expensive)

Thanks

Re: Ask HN: Fast, In-Memory, Distributed data analysis and machine learning?

#13

Post by karterk » Sun, Jul 07, 2013, 1:47 PM UTC

henrythe9th wrote: post #12

karterk wrote: post #10

Earlier quoted context omitted.

A little bit of context: I have done a lot of hadoop, and also well aware of spark and storm. Storm is mostly well suited for handling a stream of real-time data. Spark is specifically for running iterative algorithms - it can read from HDFS, and with the expressiveness of Scala, it's great for building machine-learning related stuff. However, 5GB of data is literally nothing, and that statement holds till your data…

Thanks for the suggestion. We've actually thought about just writing a multithreaded system on a single machine. What type of in-memory storage would you recommend in this case? (which hopefully may be extended to a distributed cluster of machines if 1 really large machine becomes expensive) Thanks

I suggest storing your data in files and just memory mapping them during start-up. JVM can't memory map more than 2GB per file, so just create logical shards, and map them independently.

Since you will be mostly iterating over all records during your iterative algorithms, storing them in a separate in-memory DB makes no sense (have to call external process via socket).

You can then use a framework like zookeeper/akka for managing nodes in the event that you have to scale out. Even a simple master/slave set-up using thrift services will do.

Re: Ask HN: Fast, In-Memory, Distributed data analysis and machine learning?

#14

Post by nihar » Wed, Jul 10, 2013, 9:25 PM UTC

henrythe9th wrote: post #6

nihar wrote: post #2

Have you looked at Oracle Coherence? It's pretty light weight and has clustering features as well.

Thanks for the suggestion. Looks very interesting, but couldn't find much information about it besides on Oracle. How's the community and use cases for Coherence? Thanks

Not much in terms of the open source community, but Oracle forums have some good support for this. Plus, the documentation that comes with the product is pretty decent, and a lot of really large firms use the solution.