Data Wrangling at Slack
slack.engineering
Data Wrangling at Slack
1–10 of 76 posts
Re: Data Wrangling at Slack
#2I'm curious about how much time is spent moving data back and forth from S3. It sounds like they don't currently have an ETL per say.
Re: Data Wrangling at Slack
#3We're actually having a debate now as we're starting to process larger datasets as to whether or not we should keep everything on S3 or start using HDFS w/ Hive. I'm curious if you guys considered HDFS and why you decided to go strictly with S3, and additionally, are there any issues you encounter with S3.
Re: Data Wrangling at Slack
#4We're actually having a debate now as we're starting to process larger datasets as to whether or not we should keep everything on S3 or start using HDFS w/ Hive. I'm curious if you guys considered HDFS and why you decided to go strictly with S3, and additionally, are there any issues you encounter with S3.
We've considered HDFS, but we really liked the idea of having compute only clusters and have our data kept completely separate. Clusters failure happen and having data on S3 makes us worry less if a cluster goes down. Just spin up a new one and you're good to go.
There is a bit of more latency when using S3 compared to HDFS, but it's not bad and the benefits overcame that. We do have a couple of jobs that store some intermediate results in HDFS, but in the end everything lands in S3.
We encountered a few issues with S3 at the beginning mostly around the eventual consistency, but nothing that could not be fixed.
Re: Data Wrangling at Slack
#5We're actually having a debate now as we're starting to process larger datasets as to whether or not we should keep everything on S3 or start using HDFS w/ Hive. I'm curious if you guys considered HDFS and why you decided to go strictly with S3, and additionally, are there any issues you encounter with S3.
We've considered HDFS, but we really liked the idea of having compute only clusters and have our data kept completely separate. Clusters failure happen and having data on S3 makes us worry less if a cluster goes down. Just spin up a new one and you're good to go. There is a bit of more latency when using S3 compared to HDFS, but it's not bad and the benefits overcame that. We do have a couple of jobs that store some…
Oh great, thanks for the reply. I think thats about where I think we'll land... keep S3 as the primary source, but have HDFS be used for intermediate jobs.
Re: Data Wrangling at Slack
#6Data Engineering is about developing technology for data management. Data management/analysis is about using this technology to produce results.
So this is not about data engineering, but data management/analysis.
Re: Data Wrangling at Slack
#7Earlier quoted context omitted.
We've considered HDFS, but we really liked the idea of having compute only clusters and have our data kept completely separate. Clusters failure happen and having data on S3 makes us worry less if a cluster goes down. Just spin up a new one and you're good to go. There is a bit of more latency when using S3 compared to HDFS, but it's not bad and the benefits overcame that. We do have a couple of jobs that store some…
Oh great, thanks for the reply. I think thats about where I think we'll land... keep S3 as the primary source, but have HDFS be used for intermediate jobs.
Good luck and have fun! :D
Re: Data Wrangling at Slack
#8Isn't moving data back and forth from s3 rather expensive?
Re: Data Wrangling at Slack
#9We (adtech) use a very similar approach. We're consuming a ton of data through Kafka and then using Secor to store it on S3 as Parquet files. We then use Spark for both aggregations as well as ad-hoc analyses.
One thing that sounds very interesting and worked surprisingly well when I played around with it was Amazon's Athena (https://aws.amazon.com/athena/) which lets you query Parquet data directly without relying on Spark which can get expensive quickly. I wouldn't trust production use cases just yet and it ties you more and more into the AWS ecosystem but might be worth exploring as a simple way to do basic queries on top of Parquet data. I suspect it's simply a managed service on top of Apache Drill (https://drill.apache.org/).
Re: Data Wrangling at Slack
#10Isn't moving data back and forth from s3 rather expensive?
AWS doesn't charge to put data in to s3. It's free to pull data out from its region to any AWS service within the same region. It can get expensive to pull data out across regions or out of AWS infrastructure (ie. to your private data center).