Isn't moving data back and forth from s3 rather expensive?
[deleted]
Data Wrangling at Slack
11–20 of 76 posts
Re: Data Wrangling at Slack
#12We're actually having a debate now as we're starting to process larger datasets as to whether or not we should keep everything on S3 or start using HDFS w/ Hive. I'm curious if you guys considered HDFS and why you decided to go strictly with S3, and additionally, are there any issues you encounter with S3.
We've considered HDFS, but we really liked the idea of having compute only clusters and have our data kept completely separate. Clusters failure happen and having data on S3 makes us worry less if a cluster goes down. Just spin up a new one and you're good to go. There is a bit of more latency when using S3 compared to HDFS, but it's not bad and the benefits overcame that. We do have a couple of jobs that store some…
netflix i think said they see about a 10% perf hit using s3 instead of hdfs, using emr where they launch temporary clusters that do a job and shut down, and that performance cost was well worth the flexibility of being to launch independent clusters whenever they need.
Re: Data Wrangling at Slack
#13We're actually having a debate now as we're starting to process larger datasets as to whether or not we should keep everything on S3 or start using HDFS w/ Hive. I'm curious if you guys considered HDFS and why you decided to go strictly with S3, and additionally, are there any issues you encounter with S3.
Depends on your definition of 'larger' -- if this data is on S3 currently I can't imagine we're talking multi-TB working sets here?
Generally speaking, HDFS is going to be a clusterfuck to support unless you give a load of cash to cloudera (actually, it will be regardless but slightly better with the bill) -- even then you'll get the typical db vendor line of 'not running -some patchset ver-, then upgrade. Which is really risky on a large cluster which pretty much works as you want.
Also, unless you've got a load of hardware you can dedicate to this environment, then you're going to be spending a lot of money on IAAS bills and your performance is probably not going to be very good. (Yeah sure you can virtualize HDFS but generally I passthrough local storage to the VM's, and only run demo on AWS etc).
There was been a push towards such mental complexity and folks convincing themselves they needed to solve their problems in this manner, and now a bit of an ebb backwards (at least, in the general space) now that your avg deployer found out how hard it is to do this stuff even with good support. Massive data ingestion and huge batch jobs might be a solution to a given problem you have, but it's probably not the only one whereas it's almost certainly going to be the most difficult and expensive.
Personally, I'd avoid hdfs, flume, hfs, zookeeper and all the rest of the nightmares until you're absolutely sure that you need them (and if you're not already, then you probably don't).
Also: Check out manta from joyent. :}
Re: Data Wrangling at Slack
#14We (adtech) use a very similar approach. We're consuming a ton of data through Kafka and then using Secor to store it on S3 as Parquet files. We then use Spark for both aggregations as well as ad-hoc analyses. One thing that sounds very interesting and worked surprisingly well when I played around with it was Amazon's Athena ( https://aws.amazon.com/athena/ ) which lets you query Parquet data directly without relying…
not drill, its on top of presto. presto is quite good, but the open source s3 support is definitely second class because fb doesnt use it, hopefully aws is contributing their connector back. likewise, fb use orc, and parquet is more externally supported.
Since s3 listing is so awful, and the huge number of partitions we needed, we had to write a custom connector that was aware of the file structure on s3, instead of the hive metastore which has lots of limitations, so im a little wary of athena. create table as select is amazing too, write sql to generate temporary parquet/orc files back to s3 to query later, i hope will support this if it doesn't already.
Re: Data Wrangling at Slack
#15I had very similar experience with Parquet and cross system pains. Pretty much the whole big data space is a giant cluster fuck of poorly documented and ever so slightly incompatible technologies.. with hidden config flags you need to find to get it to work the way you want, classpath issues, tiny incompatibilities between data storage formats and SQL dialects and so on..
Hoping someone on this thread could answer a related question - how do you store data in Parquet when the schema is not known ahead of time? Currently we create an RDD and use Spark to save as Parquet (which I believe has an encoder/decoder for Rows) but this is a problem because we can't stream each record as it comes and use a lot of memory to buffer before writing to disk.
Re: Data Wrangling at Slack
#16Isn't moving data back and forth from s3 rather expensive?
AWS doesn't charge to put data in to s3. It's free to pull data out from its region to any AWS service within the same region. It can get expensive to pull data out across regions or out of AWS infrastructure (ie. to your private data center).
AWS does indeed. They charge $0.005 per 1000 PUT requests (which is 12.5x more expensive than GET requests) and then you're immediately paying for storage space as well.
Re: Data Wrangling at Slack
#17We are implementing a very similar architecture, and have decided to use Avro for schema validation / serialization, rather than Parquet.
Does anyone have experience with both that can talk to their strengths / weaknesses?
Re: Data Wrangling at Slack
#18Earlier quoted context omitted.
AWS doesn't charge to put data in to s3. It's free to pull data out from its region to any AWS service within the same region. It can get expensive to pull data out across regions or out of AWS infrastructure (ie. to your private data center).
AWS does indeed. They charge $0.005 per 1000 PUT requests (which is 12.5x more expensive than GET requests) and then you're immediately paying for storage space as well.
Wow, I hadn't noticed that before. It's less than a third of the cost to store data in S3 than to pull it across the wire (2.3c/G store, 9c/G wire, in us-east-1)
Re: Data Wrangling at Slack
#19We are implementing a very similar architecture, and have decided to use Avro for schema validation / serialization, rather than Parquet. Does anyone have experience with both that can talk to their strengths / weaknesses?
Parquet is a columnar storage type whereas Avro is row-oriented serialization framework. If you have lots of columns and want to perform ad-hoc analysis, Parquet will be better than Avro due to the mechanics of the columnar storage types.
Re: Data Wrangling at Slack
#20We actually have pretty similar architecture and use Presto for ad-hoc analysis, Avro is used for hot data and ORC is used as columnar storage at https://rakam.io. Similar to Slack, we have append-only schema (stored on Mysql instead of Hive), since Avro has field ordering the parser uses the latest schema and if it gets EOF in the middle of the buffer, fills the unread columns as null. We modified the Presto engine and built a real-time data warehouse, Avro is used when pushing data to Kafka, the consumers fetch the data in micro-batches, process and convert it to ORC format and save it to the both local SSD + AWS S3.