Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Wrangling at Slack

slack.engineering

Post reply on HN
Print view View original

51–60 of 76 posts

Re: Data Wrangling at Slack

#51

Post by pixelmonkey »

ransom1538 wrote: post #37

For what it is worth, every company I have worked for - and almost every company I know -builds their own bizarre stats system. Each presentation I attend (last one being uber) the ideas for storing columnar data gets even nuttier. Frankly I gave up. Now I just installed new relic insights and I can run queries, have dashboards, and infinite scale. I understand that slack has scale - but why on earth hook together 30…

I've actually been running an analytics company for a few years (http://parse.ly), and as a result of seeing what you're describing and thinking it was pretty strange -- that many companies have "not invented here" syndrome about analytics -- we actually turned our data collection and event enrichment infrastructure into a fully-managed cloud service. It's called Parse.ly Data Pipeline, and is described here: http://parse.ly/data-pipeline.

Together with cloud SQL tools like BigQuery or Redshift, it gets rid of the need to build a "full analytics stack" on your own. You can license the data collection/enrichment from us (we've already scaled it to billions of monthly events), you can use our clean starting schema (over 100 enriched fields per event), and then you can pipe the data into a fully-managed analytics warehouse, or just analyze it in raw form. Then you can actually spend all your time focusing on insights, rather than fussing about data collection clusters, pipelines, ETLs, etc.

I would love to hear what you think of the idea; it was launched just a few months ago.

Re: Data Wrangling at Slack

#52

Post by tyingq »

guessmyname wrote: post #50
vs2370 wrote: post #46

Well for its worth my experience interviewing for the data team there was terrible. A long coding exercise that when submitted resulted in a 7 day wait and a 2 liner email. Wouldn't recommend.

What surprises me the most about the Slack's job page is that most — if not all — the positions are on-site. It surprises me because most of the companies that I know are remote-friendly use Slack as their main communication method, so I would expect Slack itself to have some remote positions just for the Dogfooding [1]. I have applied 3 times for a regular SDE position there and two times I was rejected because I wa…

This article talks about that specifically: http://readwrite.com/2014/11/06/slack-office-communication-p...

An excerpt...

Which raises the question: With such a good tool for team communication, why does Slack need an office? Why not do all your work virtually?

Slack CEO Stewart Butterfield gives product manager Mat Mullen advice, and a ukulele serenade. “There are some conversations that are much easier in person,” says Brady Archambo, Slack’s head of iOS engineering.

Re: Data Wrangling at Slack

#53

Post by joaodlf »

coldcode wrote: post #48

Sometimes looking at people's stacks I wonder if we've made computing so complicated most of the time is spent dealing with stuff that is broken, and little time is left to do anything useful. Data science seems even more into this that programming in general; and sometimes you wonder if the result is actually worth all the pain.

I feel like this happens because Data Science can only work when two professional areas clash and mix: Programming and Maths. The two are very well connected, of course, but the concepts behind the maths of Data Science are much deeper than what the typical programmer is used to. Programmers need Mathematicians as much as Mathematicians need Programmers. This is where it gets hard: Programmers find it hard to implement these concepts. On the other hand, Mathematicians don't understand what good software is.

Good data analytics software can only come when these two areas learn to teach each other. Programmers need to learn maths to the point where they are comfortable enough to implement a valid solution, Mathematicians need to learn about building software that others can use.

Re: Data Wrangling at Slack

#54

Post by treebeard901 »

ransom1538 wrote: post #45
joaodlf wrote: post #44

Earlier quoted context omitted.

Warning: I build bizarre stats systems for a living :) I totally get where you are coming from. Right now I'm thinking about a web API that feeds data into Kafka, to be processed (in Python, maybe Go?), stored into Cassandra and later on be the target of large Spark jobs, by the way, I need to present this info through pretty graphs and tables - Pandas will come in handy! Sometimes it's better to just use what someon…

I am super curious. Most analytic questions I run into: give me a month over month, which Test won, why is x happening, etc. These could be solved with just some sql queries. What questions do you run into where you need Kafka + pig +fig+ hive+ all messaged with scribe + redshift. Doesn't it even make it more difficult to answer questions?

This is a valid question. In some cases it has to do with the amount of data you're working with. Most database management systems have made progress for aggregating large amounts of data. In many cases it is still necessary to distribute the workload, which in turn creates the need to build out the rest of the distribution system.

With that being said, and to your point, I would not be surprised if these systems were often over engineered when a sql query could get the job done.

Re: Data Wrangling at Slack

#55

Post by rb808 »

ransom1538 wrote: post #37

For what it is worth, every company I have worked for - and almost every company I know -builds their own bizarre stats system. Each presentation I attend (last one being uber) the ideas for storing columnar data gets even nuttier. Frankly I gave up. Now I just installed new relic insights and I can run queries, have dashboards, and infinite scale. I understand that slack has scale - but why on earth hook together 30…

I'm surprised too. I work at companies that have their own data center so can't use new relic, datadog etc. I'm really surprised there aren't more free open source analytics platforms for small projects. I'm going to start one when I "get some spare time". lol.

Anyone know of anything out there?

Re: Data Wrangling at Slack

#56

Post by user5994461 »

joaodlf wrote: post #53
coldcode wrote: post #48

Sometimes looking at people's stacks I wonder if we've made computing so complicated most of the time is spent dealing with stuff that is broken, and little time is left to do anything useful. Data science seems even more into this that programming in general; and sometimes you wonder if the result is actually worth all the pain.

I feel like this happens because Data Science can only work when two professional areas clash and mix: Programming and Maths. The two are very well connected, of course, but the concepts behind the maths of Data Science are much deeper than what the typical programmer is used to. Programmers need Mathematicians as much as Mathematicians need Programmers. This is where it gets hard: Programmers find it hard to impleme…

It is not my impression that Data Science mixes programming and maths. Unless in a limited field of finance where all data and analysis are maths heavy.

Re: Data Wrangling at Slack

#57

Post by sciurus »

ransom1538 wrote: post #37

For what it is worth, every company I have worked for - and almost every company I know -builds their own bizarre stats system. Each presentation I attend (last one being uber) the ideas for storing columnar data gets even nuttier. Frankly I gave up. Now I just installed new relic insights and I can run queries, have dashboards, and infinite scale. I understand that slack has scale - but why on earth hook together 30…

The kind of "stats" collected by new relic are only one of many inputs toa data warehouse like Slack is describing. You can't import your mysql databases into new relic, for example.

Re: Data Wrangling at Slack

#58

Post by user5994461 »

rb808 wrote: post #55
ransom1538 wrote: post #37

For what it is worth, every company I have worked for - and almost every company I know -builds their own bizarre stats system. Each presentation I attend (last one being uber) the ideas for storing columnar data gets even nuttier. Frankly I gave up. Now I just installed new relic insights and I can run queries, have dashboards, and infinite scale. I understand that slack has scale - but why on earth hook together 30…

I'm surprised too. I work at companies that have their own data center so can't use new relic, datadog etc. I'm really surprised there aren't more free open source analytics platforms for small projects. I'm going to start one when I "get some spare time". lol. Anyone know of anything out there?

Using paid tools has no relation with having or not having your own datacenter.

If you have a small project (what is "small"?) you just deal with Google analytics or direct SQL requests to the single database you have. Don't need fancy tools.

The two free stuff I can think of are piwik and snowplowanalytics. They clearly suffer from "free open source" when compared to the paid tools out there.

Re: Data Wrangling at Slack

#59

Post by user5994461 »

treebeard901 wrote: post #54
ransom1538 wrote: post #45

Earlier quoted context omitted.

I am super curious. Most analytic questions I run into: give me a month over month, which Test won, why is x happening, etc. These could be solved with just some sql queries. What questions do you run into where you need Kafka + pig +fig+ hive+ all messaged with scribe + redshift. Doesn't it even make it more difficult to answer questions?

This is a valid question. In some cases it has to do with the amount of data you're working with. Most database management systems have made progress for aggregating large amounts of data. In many cases it is still necessary to distribute the workload, which in turn creates the need to build out the rest of the distribution system. With that being said, and to your point, I would not be surprised if these systems wer…

> With that being said, and to your point, I would not be surprised if these systems were often over engineered when a sql query could get the job done.

RedShift takes SQL queries.

Re: Data Wrangling at Slack

#60

Post by rb808 »

rb808 wrote: post #55
ransom1538 wrote: post #37

For what it is worth, every company I have worked for - and almost every company I know -builds their own bizarre stats system. Each presentation I attend (last one being uber) the ideas for storing columnar data gets even nuttier. Frankly I gave up. Now I just installed new relic insights and I can run queries, have dashboards, and infinite scale. I understand that slack has scale - but why on earth hook together 30…

I'm surprised too. I work at companies that have their own data center so can't use new relic, datadog etc. I'm really surprised there aren't more free open source analytics platforms for small projects. I'm going to start one when I "get some spare time". lol. Anyone know of anything out there?

Actually I forgot that I played with nagios vs graphite - if anyone knows of other backends similar to those that would be appreciated.
Post reply on HN