Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Wrangling at Slack

slack.engineering

Post reply on HN
Print view View original

71–76 of 76 posts

Re: Data Wrangling at Slack

#71

Post by buremba »

BrandonBradley wrote: post #47
buremba wrote: post #20

We actually have pretty similar architecture and use Presto for ad-hoc analysis, Avro is used for hot data and ORC is used as columnar storage at https://rakam.io . Similar to Slack, we have append-only schema (stored on Mysql instead of Hive), since Avro has field ordering the parser uses the latest schema and if it gets EOF in the middle of the buffer, fills the unread columns as null. We modified the Presto engine…

Are you using Avro because of your own choices or Confluent's toolset (which uses Avro on Kafka)?

We tried Avro, Thrift and Protobuf and Avro was our choice. The schema of collections in Rakam is dynamic and with both Thrift and Protobuf schema evolution is not that easy at runtime. Avro is easier to use in Java and doesn't enforce code generation, the dynamic classes are optimized for performance so it's a better option for us.

Re: Data Wrangling at Slack

#72

Post by dgudkov »

joaodlf wrote: post #35
zaptheimpaler wrote: post #28

Earlier quoted context omitted.

IMO one major problem is integration between different projects. Like you said, its a hard problem, and any solution typically depends on many many different open source projects because of the scope of challenges. All of those projects go forward without much coordination between the teams because they're open source. Then we end up in this fun, fun clusterfuck.

There is some sort of hope, Apache Arrow is (in my opinion) a step in the right direction - A common In-Memory data layer for storage and data analysis systems? Yes please. It's important to start thinking about how all these big data storage/analytics tools can bridge the gap between themselves, hopefully projects like Apache Arrow will help... As long as there is adoption.

A common in-memory columnar data layer would make a lot of sense because a) columnar is generally better for analytics, and b) converting from one columnar format to another can theoretically be done without decompression because columnar data is typically compressed using standard algorithms (vocabulary compression, LRE, etc). Here I wrote a few suggestions for such open-source data layer: http://bi-review.blogspot.ca/2015/06/the-world-needs-open-so...

Re: Data Wrangling at Slack

#73

Post by vithlani »

coldcode wrote: post #48

Sometimes looking at people's stacks I wonder if we've made computing so complicated most of the time is spent dealing with stuff that is broken, and little time is left to do anything useful. Data science seems even more into this that programming in general; and sometimes you wonder if the result is actually worth all the pain.

Your musing is in the right direction, just take it to the next logical step: who actually benefits by all this complexity? Answer: vendors, CPU providers, disk providers, network providers, etc. And thus, a certain class of field in the IT sector gets hyped up no end.

Big Data is the worst I have seen as the hype comes from all directions (readily fueled by a huge variety of parasitic vendors).

Re: Data Wrangling at Slack

#74

Post by joaodlf »

dgudkov wrote: post #72
joaodlf wrote: post #35

Earlier quoted context omitted.

There is some sort of hope, Apache Arrow is (in my opinion) a step in the right direction - A common In-Memory data layer for storage and data analysis systems? Yes please. It's important to start thinking about how all these big data storage/analytics tools can bridge the gap between themselves, hopefully projects like Apache Arrow will help... As long as there is adoption.

A common in-memory columnar data layer would make a lot of sense because a) columnar is generally better for analytics, and b) converting from one columnar format to another can theoretically be done without decompression because columnar data is typically compressed using standard algorithms (vocabulary compression, LRE, etc). Here I wrote a few suggestions for such open-source data layer: http://bi-review.blogspot.…

Good read!

Re: Data Wrangling at Slack

#75

Post by disgruntledphd2 »

poorman wrote: post #43

Apparently the concept on sampling has been lost in time.

I think that many people don't trust sampling.

I like sampling for figuring out how something works, it allows me to iterate much, much quicker.

However, if you need individual level predictions, sampling probably isn't going to help.

Re: Data Wrangling at Slack

#76

Post by infinite8s »

dgudkov wrote: post #72
joaodlf wrote: post #35

Earlier quoted context omitted.

There is some sort of hope, Apache Arrow is (in my opinion) a step in the right direction - A common In-Memory data layer for storage and data analysis systems? Yes please. It's important to start thinking about how all these big data storage/analytics tools can bridge the gap between themselves, hopefully projects like Apache Arrow will help... As long as there is adoption.

A common in-memory columnar data layer would make a lot of sense because a) columnar is generally better for analytics, and b) converting from one columnar format to another can theoretically be done without decompression because columnar data is typically compressed using standard algorithms (vocabulary compression, LRE, etc). Here I wrote a few suggestions for such open-source data layer: http://bi-review.blogspot.…

Have you seen the Apache Arrow project? https://arrow.apache.org/
Post reply on HN