Ask HN: All of you working with Big Data, what is your Data? • Hacker News Board

Re: Ask HN: All of you working with Big Data, what is your Data?

#101

Post by alanctgardner3 » Mon, Jan 26, 2015, 2:10 PM UTC

vijayr wrote: post #53

alanctgardner3 wrote: post #16

If you want to work in a "big data"-type role as a developer, I wouldn't worry about finding huge data sets. There's a dearth of candidates, especially ones who actually have hands-on experience, and having deep knowledge of (and a little experience with) a broad range of tools will make you a pretty good candidate: Fire up a VM with a single-node install on it [1] and just grab any old CSVs. Load them into HDFS, que…

If someone with a typical web dev background (knows how to handle databases like oracle or MySQL, but nothing about tools like Hadoop etc), could you recommend a course/book to start big data with? Also, with so much to learn, how does one go about deciding what field within big data to specialize in?

There's a lot of stuff under the "Big Data" umbrella: I focused on Hadoop below because that's my focus right now. I'm sure I'm missing some roles here, but the specialities I can think of are:

- getting data out of production systems and transforming it (infrastructure or ETL) - analytical querying and reporting - system administration - machine learning

There's also the wide world of NoSQL data stores, which people lump in with big data, but which require vastly different skills.

The Hadoop VM I linked to above is good for working through exercises for all of the above.

As a starting point, this book[1] walks through the motivation behind Hadoop, and then gets a little into internals and use cases. It's out of date, but you can work through it and get into the right frame of mind, understand HDFS, etc. It's a good starting point.

AMP Camp (that I linked to above) is an introduction to Spark for people with a little Hadoop experience. Spark is getting a lot of attention, you could run into it in a number of roles.

If you're going to be planning the whole pipeline, or doing any sort of infrastructure role, I recommend Hadoop Application Architecture[2] for more modern tools and design patterns. This blog post[3] is a pretty good overview of distributed logs, which are essential for horizontal scale. Understanding Kafka and ZooKeeper is really useful for infrastructure roles, maybe less so for admins.

If you're planning to be in the reporting layer, having a deep understanding of SQL and data warehousing is useful. This book[4] is old hat, but I would say it's expected knowledge for anyone planning a warehouse, and it's interesting to understand best practices. Most places will also expect knowledge of Tableau or a similar BI tool, but that's tougher to learn on your own since licenses are brutal. Visualization with D3 is nice to have in this space, especially if you're coming from a web background - Scott Murray's tutorials [5] are a good starting place.

It's harder to point to resources for sysadmins - if you weren't a sysadmin before, you need to understand a lot of other concepts before you worry about Hadoop stuff. ML is similar - you need to understand the principles and be able to work on a single node. There's lots of good resources out there about getting started in data science.

1. http://shop.oreilly.com/product/0636920021773.do

2. http://shop.oreilly.com/product/0636920033196.do

3. http://engineering.linkedin.com/distributed-systems/log-what...

4. http://ca.wiley.com/WileyCDA/WileyTitle/productCd-0471200247...

5. http://alignedleft.com/tutorials

Re: Ask HN: All of you working with Big Data, what is your Data?

#102

Post by alanctgardner3 » Mon, Jan 26, 2015, 2:45 PM UTC

kiyoto wrote: post #91

alanctgardner3 wrote: post #16

If you want to work in a "big data"-type role as a developer, I wouldn't worry about finding huge data sets. There's a dearth of candidates, especially ones who actually have hands-on experience, and having deep knowledge of (and a little experience with) a broad range of tools will make you a pretty good candidate: Fire up a VM with a single-node install on it [1] and just grab any old CSVs. Load them into HDFS, que…

>If you can talk intelligently about the whole grab bag of stuff these teams use, that'll get you in the door. Understanding RDBMSes, data warehousing concepts, and ETL is a big plus for people doing infrastructure work. This is sadly true, for now. I don't think folks here disagree with the "true" part. Let me explain why it is "sadly" and "for now". The biggest issue with big data is most of it sits unused. In many…

> The biggest issue with big data is most of it sits unused

This is really variable. If you're at a place where they jumped on the bandwagon, then yes. There are also lots of companies (and not just Google/FB/LinkedIn) that build mission critical reporting and ML infratstructure on Hadoop. These companies appreciate the value of workflow coordination, and they wouldn't move ahead without (at least) Oozie/Azkaban in place to give some visibility into their workflow.

> But, in the long term, there will be a big change.

I think more types of work will become commoditized. If you just want log processing, there are lots of on-premises and cloud options. Splunk has been doing this forever. Ostensibly with good-enough BI software you could just focus on ingest, and everything else is drag and drop. On a long enough time frame, hand-rolling pipelines will become obsolete. This is like a 10+ year timeline for any player to get significant market share. In the meantime, people have to actually get stuff done, and their skills will be transferable because they understand distributed systems, ETL, warehousing, and a lot of other stuff that hasn't really changed in a decade.

> Becoming an expert in a particular data engineering component

Are you advocating that nobody writes Spark Streaming jobs, because they should rewrite Spark instead? Don't learn to work with Impala, learn to rewrite Impala? I disagree, the tools are only getting better, and it's going to take more and more work to replace the entrenched players. Working on top of solid tools will make you far more productive than engaging in NIH and making your own SQL engine.

> Becoming an expert on quickly and effectively deploying cloud services to get the job done

Like RedShift, EMR and Amazon Data Pipeline? They're hardly turn-key solutions. Amazon's Kinesis is just Kafka with paid throughput - you can absolutely re-use your skills in the cloud, without having to cave and get locked in to a single vendor serving one specific use-case.

> What not to become is one of these OSS DIY bigots: not good enough to build truly differentiating technology, but adamant about building and running their own

So in your mind you either pick a vendor to handle all your data for you, or you're an "OSS DIY bigot"? Something like owning your entire user analytics pipeline isn't mission critical for a startup, it's stupid to build it yourself?

> These folks will be wiped out in the next decade or so.

Even though Oracle is amazing and great, lots of people still use Postgres, MySQL, etc. There's always going to be a continuum from "We should buy his turnkey thing" to "we started by rolling our own SQL query engine". You need to be able to identify when each is appropriate, not shoehorn in a one-size-fits-all solution.

Re: Ask HN: All of you working with Big Data, what is your Data?

#103

Post by billyhoffman » Mon, Jan 26, 2015, 6:14 PM UTC

icebraining wrote: post #62

billyhoffman wrote: post #55

Earlier quoted context omitted.

Won't SELECT count(*) be super slow? Isn't SELECT Count(some_primary_key) a better idea?

It should be the same. count(*) only needs to return the number of columns (regardless of its value), so it can use only indexes, while count(column) must only count non-null values. But since the primary is non-null, it should end up taking the same time.

Wow, just tested this against a table with some pretty wide columns and ~ 7M rows. They take exactly the same time! I would have thought COUNT() would be like SELECT , but I guess the query planner is smart enough to know what to do.

Thanks!