Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: All of you working with Big Data, what is your Data?

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 103 posts

Re: Ask HN: All of you working with Big Data, what is your Data?

#91

Post by kiyoto »

alanctgardner3 wrote: post #16

If you want to work in a "big data"-type role as a developer, I wouldn't worry about finding huge data sets. There's a dearth of candidates, especially ones who actually have hands-on experience, and having deep knowledge of (and a little experience with) a broad range of tools will make you a pretty good candidate: Fire up a VM with a single-node install on it [1] and just grab any old CSVs. Load them into HDFS, que…

>If you can talk intelligently about the whole grab bag of stuff these teams use, that'll get you in the door. Understanding RDBMSes, data warehousing concepts, and ETL is a big plus for people doing infrastructure work.

This is sadly true, for now. I don't think folks here disagree with the "true" part. Let me explain why it is "sadly" and "for now".

The biggest issue with big data is most of it sits unused. In many organizations, HDFS ends up being an alternative to NetApp storage servers, storing terabytes of data with the hopes of them being useful one day.

In fact, if you already get to that stage of using HDFS as a storage server, you must have a decent ETL team that can put data into HDFS with a menacing combination of ad hoc scripts and a workflow that looks like a cobweb produced by a deranged spider. For now, knowing the ins and outs of various semi-functional open source components and the tenacity, patience and skill to deal with the gnarliest of ETL tasks get you a high-paying data engineering job.

But, in the long term, there will be a big change.

1. Tools are getting better: many data practitioners are realizing there are huge gaps between different data infrastructure components, and they are trying to fill these gaps. There is a lot of attention given to query execution engines (Presto, Impala, Spark, etc.) but I find data collection/workflow management tools are just as critical (if not higher leverage) right now. Tools like Fluentd (log collector) [1], Luigi (workflow engine) are OSS software in this direction.

2. Data-related cloud services are becoming really, really good: huge kudos to services like AWS, GCP, Heroku (through Addons). They are quickly building a great ecosystem of data processing/analysis/database components that frankly work better than most self-administered OSS counterparts. (Disclaimer: my perception might be colored here since I work for a data processing/collaboration SaaS myself [3])

So, back to the question. I think aspiring data engineers have two distinct career paths:

1. Becoming an expert in a particular data engineering component: this would be building a query execution engine, designing a distributed stream processing system, etc. (It would be awesome if you decide to release as open source)

2. Becoming an expert on quickly and effectively deploying cloud services to get the job done: this is the skill most desired among data engineers at startups.

What not to become is one of these OSS DIY bigots: not good enough to build truly differentiating technology, but adamant about building and running their own . These folks will be wiped out in the next decade or so.

[1] https://www.fluentd.org [2] http://luigi.readthedocs.org [3] http://www.treasuredata.com

Re: Ask HN: All of you working with Big Data, what is your Data?

#93

Post by jghn »

gaius wrote: post #9

I think I would consider anything 100Tb and up to be big data. There is no big data that is "easily accessible"; that's why it's "big" because it requires extremely powerful hardware and advanced techniques to work on. Otherwise its just "data". NOTE: there are people in the world who would laugh at my definition and say that big data starts at 1Pb.

Count me in as someone who says the word better start with at least "peta"

It's not just eh size though, IMO it implies a certain dimensionality and/or lack of structure. At work were sittin on several petabytes and I don't view it as Big Data because it's actually pretty simple. We share many of the same problems as Big Data but not all

Re: Ask HN: All of you working with Big Data, what is your Data?

#94

Post by calinet6 »

spacefight wrote: post #52
calinet6 wrote: post #43

I work at Localytics. We have analytics data from billions of mobile and web users, including specific user actions, usage in general, and user profiles. It really is a fascinating dataset.

Are you really supposed to look at your customers data?

I never said I looked at it—we work with it, meaning we develop products which allow customers to operate on their own data.

Re: Ask HN: All of you working with Big Data, what is your Data?

#95

Post by gaius »

calinet6 wrote: post #41
gaius wrote: post #28

Earlier quoted context omitted.

So some other definition of "big" than umm "big" then?

Yes, absolutely. When people talk about big data, more often than not it's a measure of complexity and difficulty, not size.

Do you think that when people make 1000+ table relational databases it's because a) it's fun b) they're stupid or c) because it's modelling something that is inherently complex?

Big data is neither big nor particularly complex.

Re: Ask HN: All of you working with Big Data, what is your Data?

#96

Post by laughfactory »

I am the data modeler for an organization which lends to small businesses. In my experience "big data" is all in the eye of the beholder, and it's not all about how many gigabytes of data you work with, how wide, or how long it is. The challenges are the same: how to use the data in relevant ways to forward organizational goals. In my case the days isn't particularly long in terms of number of rows, but it is exceptionally wide in terms of potential variables. It's enough data that I have to spend a reasonable amount of time thinking about the most efficient way to model (statistically) and data mine. The issues are similar to other data oriented jobs I've had: how to determine which variables are relevant, clean and transform the data... And ultimately how to turn a big pile of data into a model which effectively predicts likelihood of charge off if the loan were to be approved. Scintillating stuff, but obscenely difficult. Of course, it's harder too because I'm the only modeler and am fairly inexperienced. My last experience building predictive models was a couple classes in college... Which was also my last experience using R (which I prefer to SAS.

To answer your implied question, I'd recommend picking up ANY size real world data and playing with it. Build statistical models (predictive or otherwise), apply supervised and unsupervised machine learning methods to it, but above all develop a foundation of experience working with real world data. In class in college we used "canned" data sets which were already cleaned, validated, organized, and so forth. This made it unrealistically easy to model. In the real world just working with the data effectively is a hard won skill. So from the get go you need to learn how to explore data, visualize it, interpret plots and statistics, clean/transform/normalize it, formulate a question your data can answer, and apply the relevant methods in pursuit of the answers you seek. Once you have the fundamentals down the size of the data is immaterial--only requiring you to put additional thought into what you can computationally achieve (for instance, how to determine which of 150 candidate variables are statistically relevant).

Re: Ask HN: All of you working with Big Data, what is your Data?

#98

Post by jleahy »

Demiurge wrote: post #18
jleahy wrote: post #14

Earlier quoted context omitted.

Actually you can normally get away without resorting to "big data" techniques at 1PB scale, it's probably around 10PB that you're really forced to start thinking about things like hadoop style map-reduce/rack-locality.

Probably depends on what you're doing with the data?

Right, I'm thinking of the "embarrassingly parallel" sort of problems, where you can shard the data (for example by city, country, company or some other obvious classifier) and give one shard to one server.

I'm you were trying to do anything that's O(n.log n) or O(n^2) (think graph processing) then you'll run into trouble at much smaller scales.

Re: Ask HN: All of you working with Big Data, what is your Data?

#99

Post by dmichulke »

noname123 wrote: post #34
dmichulke wrote: post #32

Financial data (tick to EOD), network traffic data (TCP packet level sends / receives) and farm data (sensor + farm ERP data) All of them are basically time series with some master data, none of them is more than a few dozen GB So in any case, I think time series data is worth a look.

I run backtesting for options trading as a hobby and storing EOD tick data, querying it and extracting it is a pain. I dl my source data currently from a retail historical data provider, then store it in MongoDB in AWS. What would you recommend tech stackwise to do backtesting on time seriea data?

I also first used MongoDB but I switched to cassandra and I find it much better.

MongoDB was bad for this case because it devours memory and it lacks a primary key (so saving a tick that's already creates a second tick).

cassandra is better (it has a PK, although it's a bit weird because of its distributed-first attitude) but in the long run I think postgres would be even better (because it's space efficient).

Apart from that I use some Java libraries and clojure/incanter and program the rest around it myself.

Re: Ask HN: All of you working with Big Data, what is your Data?

#100

Post by alanctgardner3 »

pknerd wrote: post #77
alanctgardner3 wrote: post #16

If you want to work in a "big data"-type role as a developer, I wouldn't worry about finding huge data sets. There's a dearth of candidates, especially ones who actually have hands-on experience, and having deep knowledge of (and a little experience with) a broad range of tools will make you a pretty good candidate: Fire up a VM with a single-node install on it [1] and just grab any old CSVs. Load them into HDFS, que…

Wow! I loved the way you explained it so clearly. Is it possible I could contact you off the site to get further guidance?

alanctgardner@gmail.com, feel free :)
Post reply on HN