Hey! I’m the author of this post. I’m pretty chuffed to see this here. Happy to answer any questions.
Why didn't you push the data into a traditional data warehouse or sql database?
[deleted]
Goodreads offloads DynamoDB tables to S3 and queries them with Athena
41–50 of 110 posts
Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena
#42Hey! I’m the author of this post. I’m pretty chuffed to see this here. Happy to answer any questions.
Happy to post it, but you can thank Corey Quinn from https://lastweekinaws.com/ for pointing me to it.
I did! I'm a big fan of the Last Week in AWS newsletter.
https://twitter.com/iamsomewalrus/status/1016368717880389632
Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena
#43Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena
#44Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena
#45Earlier quoted context omitted.
For a lot of teams, S3 is a data warehouse, and you can treat it just like HDFS for the most part with most things in the big-data ecosystem. Presto works well for letting you access it from these locations without having to explicitly import it (assuming it's in a traditional data warehouse or a common SQL DB).
I wonder if anyone here has a good heuristic for identifying the conditions under which using S3 + SQL layer as a data warehouse is a better choice than a SQL database? I've been exploring the former and it seems to only make sense if the size of your data is at a scale that is beyond what a single SQL database instance can handle, and even then, you can continue to scale out with systems like Citus so the limit isn'…
The decision to add SQL on top of S3 probably had a lot to do with a very common use case: people had structured data in S3 but no way to query it.
However, it is also very useful if the following two things are true: 1. You have a very large stream of incoming structured data that is mostly write-once-read-never, like logs. 2. Your query use cases are relatively simple and static. If those fit your use case, then S3 + parquet + Athena is very easy and very cheap.
Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena
#46Earlier quoted context omitted.
For a lot of teams, S3 is a data warehouse, and you can treat it just like HDFS for the most part with most things in the big-data ecosystem. Presto works well for letting you access it from these locations without having to explicitly import it (assuming it's in a traditional data warehouse or a common SQL DB).
I wonder if anyone here has a good heuristic for identifying the conditions under which using S3 + SQL layer as a data warehouse is a better choice than a SQL database? I've been exploring the former and it seems to only make sense if the size of your data is at a scale that is beyond what a single SQL database instance can handle, and even then, you can continue to scale out with systems like Citus so the limit isn'…
Having been back in SQL land for a bit (vanilla MySQL on RDS) I have to say that I _love_ a well designed SQL database. I forgot how much I had given up in NoSQL land.
Goodreads hit scaling issues a while ago with Active Record and a single database so we broke up the data into separate MySQL servers. At that point joining data across DB servers is impossible so we went with Redshift for BI. Nowadays we would probably go with a datalake on S3.
Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena
#47Earlier quoted context omitted.
For a lot of teams, S3 is a data warehouse, and you can treat it just like HDFS for the most part with most things in the big-data ecosystem. Presto works well for letting you access it from these locations without having to explicitly import it (assuming it's in a traditional data warehouse or a common SQL DB).
:this: A concept that's underlying the move to a datalake architecture (read: keeping your data in its rawest form, and its transforms in S3 or HDFS) is decoupling your compute from storage. Motivating example: you have huge tables in Redshift that are either infrequently accessed or the usefulness of the data decays over time (website logs, customer order information). In this scenario you're paying a lot just to ke…
BigQuery on GCP does this as well, without any extra work.
(Disclaimer: I work for Google)
Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena
#48Earlier quoted context omitted.
Come join my team and I’ll be happy to spill all the beans!
Do you guys have internships on the Goodreads team?
Absolutely! We have internships in San Francisco and Seattle. Usually, they coincide with the summer, but we on occasion have Fall interns.
Reach out to me feeneyj @ amazon if you'd like I can forward you to the right people.
Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena
#49Earlier quoted context omitted.
For a lot of teams, S3 is a data warehouse, and you can treat it just like HDFS for the most part with most things in the big-data ecosystem. Presto works well for letting you access it from these locations without having to explicitly import it (assuming it's in a traditional data warehouse or a common SQL DB).
I wonder if anyone here has a good heuristic for identifying the conditions under which using S3 + SQL layer as a data warehouse is a better choice than a SQL database? I've been exploring the former and it seems to only make sense if the size of your data is at a scale that is beyond what a single SQL database instance can handle, and even then, you can continue to scale out with systems like Citus so the limit isn'…
I've often wondered the same, thanks for raising this question! I hope somebody with direct experience can chime in!