If you're looking for data sets to play with, check out Kaggle [0]. Companies post data sets there along with questions they want answered, and people compete to find the best way to answer them.
[0] www.kaggle.com
Ask HN: All of you working with Big Data, what is your Data?
71–80 of 103 posts
Re: Ask HN: All of you working with Big Data, what is your Data?
#72I think the most interesting datasets are within reach but require curation yourself. For example there are extremely powerful scraping libraries in just about every popular language today, not to mention APIs such as Twitter's. If you're looking for a cool dataset to play with, I think it is more productive to ask yourself what questions you want to answer and then find/curate the data VS find a dataset and then ask…
> I think it is more productive to ask yourself what questions you want to answer
I second that. An old remark is, "We often find that a good question is more important than a good answer.", as I recall, due to Richard Bellman, say, the leading proponent of dynamic programming, i.e., usually a case of optimal control, either for the deterministic or stochastic (the system gets random exogenous inputs while we are trying to control it). Bellman was into a lot in pure and applied mathematics, engineering, medicine, etc. Bright guy. As I recall, his Ph.D. was in stability of solutions of initial value problems for ordinary differential equations, from Princeton.
Re: Ask HN: All of you working with Big Data, what is your Data?
#73Genetic sequence data, mostly from cancer.
(Tools are terrible, data sizes up to hundreds of gigabytes per patient)
Re: Ask HN: All of you working with Big Data, what is your Data?
#74Earlier quoted context omitted.
I've worked on relational databases of similar size. There's two challenges. The first is maintaining the relational model at that scale is quite tricky, tradeoffs need to be made. The second is the systems-level management of that large a deployment requires a bit more than standard configuration management. These days Amazon have Multi-AZ RDS, which should handle the 2nd item.
The problem with databases that are 50tb or more is that you soon run into limits with the relational model. I have been reading up on different modeling techniques for converting relational models into cassandra's column family stores.
[deleted]
Re: Ask HN: All of you working with Big Data, what is your Data?
#75Earlier quoted context omitted.
I've worked on relational databases of similar size. There's two challenges. The first is maintaining the relational model at that scale is quite tricky, tradeoffs need to be made. The second is the systems-level management of that large a deployment requires a bit more than standard configuration management. These days Amazon have Multi-AZ RDS, which should handle the 2nd item.
The problem with databases that are 50tb or more is that you soon run into limits with the relational model. I have been reading up on different modeling techniques for converting relational models into cassandra's column family stores.
The issues are a bit more fundamental.
You can't practically fit 50TB on one machine and have reasonable performance, that means multiple machines with the data spread across them.
There's then two potential issues: 1) You're doing 1-to-1 joins across tables in a query, network latency may be an issue at high query rates 2) You're going 1-to-many or many-to-many joins across tables in a query, the resulting combinatorial explosion of data is too much to handle
You want to have your inner loops/joins as deep down in the stack as possible. If you can structure things so all the heavy lifting stays inside one rack/machine/NUMA node/ processor/core you'll be able to scale a good bit further further.
Designing things not to require joins, denormalising and putting it in a column store like Cassandra is also a good approach.
Re: Ask HN: All of you working with Big Data, what is your Data?
#76I think I would consider anything 100Tb and up to be big data. There is no big data that is "easily accessible"; that's why it's "big" because it requires extremely powerful hardware and advanced techniques to work on. Otherwise its just "data". NOTE: there are people in the world who would laugh at my definition and say that big data starts at 1Pb.
Those numbers make me think of "store all the things" rather than useful statistical data. 1PB is arguably enough data to store genetic variation across all human beings .
1pb / 100gb / genome = 10,000 genome sequences. And that's just raw data from one platform. If you're interested in e.g. splicing diversity you would want to do long read RNA sequencing. Leaving room for intermediate results (alignments, assemblies) you would only have room for a thousand people.
Re: Ask HN: All of you working with Big Data, what is your Data?
#77If you want to work in a "big data"-type role as a developer, I wouldn't worry about finding huge data sets. There's a dearth of candidates, especially ones who actually have hands-on experience, and having deep knowledge of (and a little experience with) a broad range of tools will make you a pretty good candidate: Fire up a VM with a single-node install on it [1] and just grab any old CSVs. Load them into HDFS, que…
Wow! I loved the way you explained it so clearly. Is it possible I could contact you off the site to get further guidance?
Re: Ask HN: All of you working with Big Data, what is your Data?
#78I think I would consider anything 100Tb and up to be big data. There is no big data that is "easily accessible"; that's why it's "big" because it requires extremely powerful hardware and advanced techniques to work on. Otherwise its just "data". NOTE: there are people in the world who would laugh at my definition and say that big data starts at 1Pb.
Actually you can normally get away without resorting to "big data" techniques at 1PB scale, it's probably around 10PB that you're really forced to start thinking about things like hadoop style map-reduce/rack-locality.
[deleted]
Re: Ask HN: All of you working with Big Data, what is your Data?
#79Governmental health records and survey data. A lot of the really big stuff we use requires academic licenses, but there's still a lot of publicly accessible data.
For the U.S. try
- CDC's National Center for Health Statistics: http://www.cdc.gov/nchs/
- CDC WONDER: http://wonder.cdc.gov/
- NIH's Unified Medical Language System: http://www.nlm.nih.gov/research/umls/
And for global try the Global Health Data Exchange: http://ghdx.healthdata.org
Re: Ask HN: All of you working with Big Data, what is your Data?
#80Earlier quoted context omitted.
Those numbers make me think of "store all the things" rather than useful statistical data. 1PB is arguably enough data to store genetic variation across all human beings .
1pb / 100gb / genome = 10,000 genome sequences. And that's just raw data from one platform. If you're interested in e.g. splicing diversity you would want to do long read RNA sequencing. Leaving room for intermediate results (alignments, assemblies) you would only have room for a thousand people.
Thanks for picking up on this :) As I said, it's arguable.
I was working on the principle that the effective population size of humans is 10,000.
(And your genome is oversized, no? 3 billion base-pairs is less than 1 Gigabyte)