Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Wrangling at Slack

slack.engineering

Post reply on HN
Print view View original

61–70 of 76 posts

Re: Data Wrangling at Slack

#61

Post by user5994461 »

bhntr3 wrote: post #25

Seems like a pretty typical set of problems. Dependency conflicts hard. Schema evolution hard. Upgrades hard. The big data space still feels like an overengineered, fractured, buggy mess to me. I was hoping spark would simplify the user experience but it's as much of a clusterf*ck as anything else. How hard can fast, reliable distributed computation and storage for petabytes of data be? He said ironically.

Trivial. Just put everything in BigQuery. Use looker for visualization. Job done.

Any single other component you may try to add will increase the complexity factorially. Better stick to the basics ;)

Re: Data Wrangling at Slack

#62

Post by joaodlf »

user5994461 wrote: post #56
joaodlf wrote: post #53

Earlier quoted context omitted.

I feel like this happens because Data Science can only work when two professional areas clash and mix: Programming and Maths. The two are very well connected, of course, but the concepts behind the maths of Data Science are much deeper than what the typical programmer is used to. Programmers need Mathematicians as much as Mathematicians need Programmers. This is where it gets hard: Programmers find it hard to impleme…

It is not my impression that Data Science mixes programming and maths. Unless in a limited field of finance where all data and analysis are maths heavy.

I felt the same when our stats were based on simple arithmetics, "sum those revenue figures", "divide that by the total amount of users", "percentage of returning members"...

It can easily spiral into, "Pearson's Correlation" or "Give me the Linear regression of the bastard".

Re: Data Wrangling at Slack

#63

Post by draw_down »

tyingq wrote: post #52
guessmyname wrote: post #50

Earlier quoted context omitted.

What surprises me the most about the Slack's job page is that most — if not all — the positions are on-site. It surprises me because most of the companies that I know are remote-friendly use Slack as their main communication method, so I would expect Slack itself to have some remote positions just for the Dogfooding [1]. I have applied 3 times for a regular SDE position there and two times I was rejected because I wa…

This article talks about that specifically: http://readwrite.com/2014/11/06/slack-office-communication-p... An excerpt... Which raises the question: With such a good tool for team communication, why does Slack need an office? Why not do all your work virtually? Slack CEO Stewart Butterfield gives product manager Mat Mullen advice, and a ukulele serenade. “There are some conversations that are much easier in person,”…

Revealing.

Re: Data Wrangling at Slack

#64

Post by user5994461 »

cyberpunk wrote: post #13
mastratton3 wrote: post #3

We're actually having a debate now as we're starting to process larger datasets as to whether or not we should keep everything on S3 or start using HDFS w/ Hive. I'm curious if you guys considered HDFS and why you decided to go strictly with S3, and additionally, are there any issues you encounter with S3.

Depends on your definition of 'larger' -- if this data is on S3 currently I can't imagine we're talking multi-TB working sets here? Generally speaking, HDFS is going to be a clusterfuck to support unless you give a load of cash to cloudera (actually, it will be regardless but slightly better with the bill) -- even then you'll get the typical db vendor line of 'not running -some patchset ver-, then upgrade. Which is r…

S3 is ideal for multi TB working set.

That should be the de-factor standard for TB scale. In fact, don't bother comparing other products if you're TB scale, just use S3.

Re: Data Wrangling at Slack

#65

Post by user5994461 »

vikiomega9 wrote: post #2

I'm curious about how much time is spent moving data back and forth from S3. It sounds like they don't currently have an ETL per say.

Pick one solution among:

- alooma.io (SaaS queing and transformation pipeline that saves to S3)

- segment.io (Saas analytics platform that can save to S3)

- snowplowanalytics (clusterfuck open source self hosted analytics pipeline)

Re: Data Wrangling at Slack

#66

Post by cyberpunk »

user5994461 wrote: post #64
cyberpunk wrote: post #13

Earlier quoted context omitted.

Depends on your definition of 'larger' -- if this data is on S3 currently I can't imagine we're talking multi-TB working sets here? Generally speaking, HDFS is going to be a clusterfuck to support unless you give a load of cash to cloudera (actually, it will be regardless but slightly better with the bill) -- even then you'll get the typical db vendor line of 'not running -some patchset ver-, then upgrade. Which is r…

S3 is ideal for multi TB working set. That should be the de-factor standard for TB scale. In fact, don't bother comparing other products if you're TB scale, just use S3.

Really?

Say you're going to ETL or Map/Reduce over all that data a lot of times, you're telling me that reading it all for processing over S3's rest api (which is the only method?) instead of, say, a local array of 15k sas's over pcie hba's is ideal?

It's pretty expensive and inefficient to my eyes, what am I missing? I

In what way would S3 be better than running this on your own gear if cost and perf are clearly not going to be better (which are really the big factors in this decision)?

Re: Data Wrangling at Slack

#67

Post by user5994461 »

cyberpunk wrote: post #66
user5994461 wrote: post #64

Earlier quoted context omitted.

S3 is ideal for multi TB working set. That should be the de-factor standard for TB scale. In fact, don't bother comparing other products if you're TB scale, just use S3.

Really? Say you're going to ETL or Map/Reduce over all that data a lot of times, you're telling me that reading it all for processing over S3's rest api (which is the only method?) instead of, say, a local array of 15k sas's over pcie hba's is ideal? It's pretty expensive and inefficient to my eyes, what am I missing? I In what way would S3 be better than running this on your own gear if cost and perf are clearly not…

You're missing that S3 is the storage system for RedShift and EMR (emr = managed hadoop on AWS).

They are pretty cheap, efficient and simple to use ;)

Re: Data Wrangling at Slack

#68

Post by user5994461 »

joaodlf wrote: post #62
user5994461 wrote: post #56

Earlier quoted context omitted.

It is not my impression that Data Science mixes programming and maths. Unless in a limited field of finance where all data and analysis are maths heavy.

I felt the same when our stats were based on simple arithmetics, "sum those revenue figures", "divide that by the total amount of users", "percentage of returning members"... It can easily spiral into, "Pearson's Correlation" or "Give me the Linear regression of the bastard".

Still not hard maths. If all you have to do is apply a simple standard well documented algorithm, there is really no obstacle to your success =)

That being said. I guess that having had maths classes in my engineering degree skews my point of view, combined with working with Quants at times, who do analysis way more advanced than that.

Re: Data Wrangling at Slack

#69

Post by joaodlf »

user5994461 wrote: post #68
joaodlf wrote: post #62

Earlier quoted context omitted.

I felt the same when our stats were based on simple arithmetics, "sum those revenue figures", "divide that by the total amount of users", "percentage of returning members"... It can easily spiral into, "Pearson's Correlation" or "Give me the Linear regression of the bastard".

Still not hard maths. If all you have to do is apply a simple standard well documented algorithm, there is really no obstacle to your success =) That being said. I guess that having had maths classes in my engineering degree skews my point of view, combined with working with Quants at times, who do analysis way more advanced than that.

If you are familiar with those concepts, I would count that as a big step over what I typically see in "data science". Surely a big step over what a lot of people think data analytics to be.

Like yourself, I had quite a bit of contact with maths during my engineering degree - whether I took most of it in is a different question :) (Financial Calculus nearly destroyed me).

Developers aren't typically aware of concepts outside basic statistics, and even though a lot of algorithms are readily available for everyone to implement and benefit from, how can you use what you don't know conceptually?

I guess everyone has a different experience, depends where you're working, really. I do know of quite a few shops where the push for analytics came from the tech people, mostly because companies don't employ people with the math knowledge to identify these business gains.

Re: Data Wrangling at Slack

#70

Post by user5994461 »

joaodlf wrote: post #69
user5994461 wrote: post #68

Earlier quoted context omitted.

Still not hard maths. If all you have to do is apply a simple standard well documented algorithm, there is really no obstacle to your success =) That being said. I guess that having had maths classes in my engineering degree skews my point of view, combined with working with Quants at times, who do analysis way more advanced than that.

If you are familiar with those concepts, I would count that as a big step over what I typically see in "data science". Surely a big step over what a lot of people think data analytics to be. Like yourself, I had quite a bit of contact with maths during my engineering degree - whether I took most of it in is a different question :) (Financial Calculus nearly destroyed me). Developers aren't typically aware of concepts…

They are really just maths algorithm, seen in maths courses or found with a quick google search.

The typical reddit developer who got a job without a degree is unaware of many many things.

The typical developer who got a job with a hardcore interview at random financial company and is surrounded by other master's and PhD. Not so much.

The typical tech company doesn't need much advanced analysis. If they could figure out how many recurring users and revenues they have, that would be a good start :D

Post reply on HN