GPS and transport data.
Same. >1TB/hour of vehicles and sensor data. Including video, gps and radar
Ask HN: All of you working with Big Data, what is your Data?
81–90 of 103 posts
Re: Ask HN: All of you working with Big Data, what is your Data?
#82If you want to work in a "big data"-type role as a developer, I wouldn't worry about finding huge data sets. There's a dearth of candidates, especially ones who actually have hands-on experience, and having deep knowledge of (and a little experience with) a broad range of tools will make you a pretty good candidate: Fire up a VM with a single-node install on it [1] and just grab any old CSVs. Load them into HDFS, que…
Definitely - playing with any data is the best way to learn the tools.
Just for giggles I built a tool that extracts all oracle permissions, sums that up into relationship information using PIG (this schema owner reads from here, writes to there, etc) and used first R/ggplot2 and then later Gephi to plot the results.
None of the data sets could be called big data by any stretch, and I could have done the processing more quickly with perl or python, or even a mix of shell commands. But that wasn't the point. It was to expand on the one day of training I'd had and help cement the ideas, and frankly it was to have fun.
Find something that you're passionate about or just plain sounds like fun and then use the tool you want to learn to solve your problem.
Re: Ask HN: All of you working with Big Data, what is your Data?
#83Here's some stuff I have done in the past year, I work for a small company, but run a personal computing cluster of 167 servers that I pay for out of my own pocket. I really enjoy loading "big" datasets into them and working on improving algorithms or gaining insight into the data. I (try and) network around London and offer my services for free to people who have interesting problems. - Very high resolution FMRI dat…
Do you actually have 167 servers running in your home or do you rent them on e.g. EC2 when you need them? If it's the former case, is it because it makes financial sense (I'd be surprised) or is it for the experience/fun?
I have 3 blade servers in my house, these act as command and control machines, the 167 are rented on EC2
Re: Ask HN: All of you working with Big Data, what is your Data?
#84I think I would consider anything 100Tb and up to be big data. There is no big data that is "easily accessible"; that's why it's "big" because it requires extremely powerful hardware and advanced techniques to work on. Otherwise its just "data". NOTE: there are people in the world who would laugh at my definition and say that big data starts at 1Pb.
> NOTE: there are people in the world who would laugh at my definition and say that big data starts at 1Pb. I commend them for having a larger penis ^H^H^H^H^H^H data stack than you. I thought big data was less about the actual size of the data store and more about where it comes from (typically passive collection from user activity) and how it's accessed (through some kind of large map-reduce style framework) and us…
From the technology standpoint, where it comes from and how it's used doesn't really make a difference - if it can be processed on a single very beefy machine when done properly, then the appropriate/efficient way to work with this data is by avoiding big data techniques.
If it cannot, then you pay the price of all the complexity and overheads of big data processing techniques so that you can get your processing done.
It's correlated with data size, bot not so strictly - you can get, for example, NLP processing problems where you need a painful pipeline split over a huge cluster for a single gb of input data, and you can have problems where the best way to process a petabyte dataset is just to stick a single powerful machine to get the performance benefits of locality and low latency, and avoid managing splits/failed nodes/whatever.
So, in the first problem you would need to use Big Data techniques and the second problem you don't, it's not related to big data and the recommendations on how best to do that won't help people who need to do big data processing.
Re: Ask HN: All of you working with Big Data, what is your Data?
#85Here's some stuff I have done in the past year, I work for a small company, but run a personal computing cluster of 167 servers that I pay for out of my own pocket. I really enjoy loading "big" datasets into them and working on improving algorithms or gaining insight into the data. I (try and) network around London and offer my services for free to people who have interesting problems. - Very high resolution FMRI dat…
What do you pay for such cluster if I may ask? I assume it's not in EC2? Have you thought of adding a web-crawling service on the top of that?
It is on EC2. I pay about £2500 a month - but the monthly cost is lowered because a lot of the machines are on reserved instances - with a big chunk paid up front (sometimes by the people I work with) ... I usually work for free because I'm not really interested in financial gain but more interested in interesting problems ... but I gotta pay the bills sometimes, and the people I work with are usually more than happy to chip in if I need it :)
Re: Ask HN: All of you working with Big Data, what is your Data?
#86The twitter social graph (follow connections between people) is my data source, I extract it from the API and cache it in a database. The mariadb table storing this information currently takes a bit more than 500GB, it has about 4 billion rows (based on the statistics, I don't run SELECT count(*) on it anymore). I usually don't use the term "big data" because the buzzword is so popular that it doesn't mean anything a…
Won't SELECT count(*) be super slow? Isn't SELECT Count(some_primary_key) a better idea?
[deleted]
Re: Ask HN: All of you working with Big Data, what is your Data?
#87Pornography
Re: Ask HN: All of you working with Big Data, what is your Data?
#88If you want to work in a "big data"-type role as a developer, I wouldn't worry about finding huge data sets. There's a dearth of candidates, especially ones who actually have hands-on experience, and having deep knowledge of (and a little experience with) a broad range of tools will make you a pretty good candidate: Fire up a VM with a single-node install on it [1] and just grab any old CSVs. Load them into HDFS, que…
Is that really all that "big data" positions need? In past experience (Google), there were all sorts of problems that working with actual huge data sets introduced that wasn't handled by the frameworks available (not even MapReduce, which by most accounts is significantly more advanced than Hadoop). Things like:
1. With a big data set, there is no easy way to verify the correctness of your algorithms. The data is too big to hand inspect, and so assuming your code is syntactically well-formed and doesn't crash, you will get an answer. Is your answer correct? Well, you don't actually know, and any number of logic errors might throw it off without causing a detectable programming error.
2. Big data is messy. There will be some records in your data set that are formatted differently than you expect, or contain data that means something semantically different than you expect. Best case, your Hadoop job crashes 4 hours in. Worst case, it silently succeeds, and you have no idea that your results were polluted by spurious results that you had no idea existed.
3. Big data will expose basically every code path and combination of code paths in your analysis program, so it all better be bulletproof. Learn how to write code correctly the first time, or you're going to be spending a lot of time waiting for the script to run and then fixing crashes several hours in.
4. Big data contains outliers. Oftentimes, the outliers will dominate your results, and so if you don't have a way of filtering them out or making your algorithm less sensitive to them, you will get garbage as your final answer.
There are techniques to deal with these, but they are techniques that are built into your workflow as a data scientist, and not the tools that are available. One thing that always amazed me at Google was how much time the data scientists on staff spent not writing code. Writing your MapReduces takes perhaps 5-10% of your day; most of the rest of it is mundane stuff like staring at data and compiling golden sets.
Re: Ask HN: All of you working with Big Data, what is your Data?
#89If you're looking for data sets to play with, check out Kaggle [0]. Companies post data sets there along with questions they want answered, and people compete to find the best way to answer them. [0] www.kaggle.com
Interesting idea. I'm going to start posting my company's workload online in the form of competitions, letting people work for the possibility of being compensated at sub-market rates.
Re: Ask HN: All of you working with Big Data, what is your Data?
#90I think I would consider anything 100Tb and up to be big data. There is no big data that is "easily accessible"; that's why it's "big" because it requires extremely powerful hardware and advanced techniques to work on. Otherwise its just "data". NOTE: there are people in the world who would laugh at my definition and say that big data starts at 1Pb.
In practice it's more a marketing term, and how big is big depends on what the nature of the data is and what you're doing with it. If it fits in RAM on your laptop, it isn't big data. If you can't process/handle it in a reasonable time on a single machine and your methods need to explicitly worry about how to scale to handle the data volumes it probably is "Big Data". Problems that are embarrassingly parallel need f…
I think that needing disk parallelism because you have a workload that demands table scans and maintaining indexes is impractical due to dynamism in the data is one feature.
Another is not having pockets deep enough to solve it with intellectual property, either in the form of a parallel proprietary rdbs (spensive) or the need to implement clever stuff.
Big data as a technology is about dumb as brick, cheap as chips, brute force.