Hey! I’m the author of this post. I’m pretty chuffed to see this here. Happy to answer any questions.
It's a great post. Thanks for taking the time to write it!
Goodreads offloads DynamoDB tables to S3 and queries them with Athena
101–110 of 110 posts
Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena
#102Earlier quoted context omitted.
For a lot of teams, S3 is a data warehouse, and you can treat it just like HDFS for the most part with most things in the big-data ecosystem. Presto works well for letting you access it from these locations without having to explicitly import it (assuming it's in a traditional data warehouse or a common SQL DB).
I wonder if anyone here has a good heuristic for identifying the conditions under which using S3 + SQL layer as a data warehouse is a better choice than a SQL database? I've been exploring the former and it seems to only make sense if the size of your data is at a scale that is beyond what a single SQL database instance can handle, and even then, you can continue to scale out with systems like Citus so the limit isn'…
The serverless capabilities is the big plus IMO. You pay for query. If you go with a typical OLAP system like redshift you need a cluster with a minimum number of machines I believe.
I think it compares more with something like BigQuery but if you already have your data in S3 maybe you get a more well integrated system if you stick with AWS tools.
Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena
#103How does Athena work? The charge is $5 per terrabyte scanned, which indicates (maybe) that no indices are used and queries are processed by a scan through the data. Is this correct?
yes. It uses schema-on-read so it doesn't support indices as far as I know. And yeah, it scans through the data using the schema you define previously
Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena
#104How does Athena work? The charge is $5 per terrabyte scanned, which indicates (maybe) that no indices are used and queries are processed by a scan through the data. Is this correct?
The docs are a good place to start to get a sense of what Athena can do: https://docs.aws.amazon.com/athena/latest/ug/what-is.html
There are no indexes in the traditional MySQL / Postgres sense of the word. You can, however, layout the data to make your querying more efficient. See: https://docs.aws.amazon.com/athena/latest/ug/partitions.html
Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena
#105With the launch of https://aws.amazon.com/about-aws/whats-new/2018/07/aws-glue-... Wouldn't using ETL Glue job to directly dump data in parquet format be better?
100%. Expect a follow up in the future with a simplified pipeline.
Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena
#106Earlier quoted context omitted.
The website search seems to not be tokenized, it seems to use the entire word only... so if there's a slight difference it won't find anything. The main thing I've noticed in Goodreads since they were bought is a strong focus on Kindle integration. The newer Kindles all have Goodreads integration, you can send notes etc. to the Goodreads profile, the site itself has changed very little (although Goodreads changed a l…
Feature-rich text search with elasticsearch is stupidly easy. It's a real shame.
or postgres! https://news.ycombinator.com/item?id=17638169
Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena
#107Earlier quoted context omitted.
I know that the tuning you made is probably very context-dependent, but could you elaborate a bit on what kind of things you tuned up? I am about to start using ES for a project and just knowing which kinds of things could be useful to tune would be helpful. Thanks!
Sure thing! I'll see if we can make a blog post, but here are a bunch of things that jump out (I work at a company that sells tickets to live events, so our users search for 'live events' like sporting events / concerts and 'performers' like teams and musicians): - word order matters _a lot_. we did a lot of fiddling with the n-gram tokenzier ( https://www.elastic.co/guide/en/elasticsearch/reference/curr... ).. we en…
Awesome answer!
I'm hoping you do find the time to write a blog post on this.
Thanks a lot!
Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena
#108Earlier quoted context omitted.
I know that the tuning you made is probably very context-dependent, but could you elaborate a bit on what kind of things you tuned up? I am about to start using ES for a project and just knowing which kinds of things could be useful to tune would be helpful. Thanks!
Check out https://www.manning.com/books/relevant-search (Disclaimer: I used to work with one of the authors at Eventbrite)
Thanks!
Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena
#109Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena
#110Hey! I’m the author of this post. I’m pretty chuffed to see this here. Happy to answer any questions.
iamsomewalrus: Since you're using the same s3 key prefix for all dbexport data does that slow down your Athena queries because of key map partitioning? [1] Also, do you ever see 503 Slowdown errors from Athena requests to s3?
[1]: https://aws.amazon.com/blogs/aws/amazon-s3-performance-tips-...