Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goodreads offloads DynamoDB tables to S3 and queries them with Athena

aws.amazon.com

Post reply on HN
Print view View original

71–80 of 110 posts

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#71

Post by psychometry »

a_bonobo wrote: post #24
kaycebasques wrote: post #8

I use Goodreads as my main book tracker and reviewer. My impression is that Amazon bought it for whatever reason, and then ignored it. It seems like there's so much more potential to a social community around books than what Goodreads offers. The UI also never fails to disappoint. One of the core user features, searching for books, has all sorts of weird quirks. I'll use a very specific query that should guarantee a…

The website search seems to not be tokenized, it seems to use the entire word only... so if there's a slight difference it won't find anything. The main thing I've noticed in Goodreads since they were bought is a strong focus on Kindle integration. The newer Kindles all have Goodreads integration, you can send notes etc. to the Goodreads profile, the site itself has changed very little (although Goodreads changed a l…

Feature-rich text search with elasticsearch is stupidly easy. It's a real shame.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#72

Post by CobrastanJorji »

iamsomewalrus wrote: post #55
Jemaclus wrote: post #54

I like the approach, and I've been considering building something similar. Nice writeup. Is this used only for BI or is it used for real-time queries? The reason I ask is because goodreads.com is slow AF, and performance is a concern for me at this point.

We use carrier pigeons from an S3 data center high in the Himalayas to a CloudFront distribution center in Atlantic City (don't ask me, ask the pigeons) to serve all requests. Just BI.

It annoys me that AWS only supports RFC 1149. When will RFC 2549 support be added?

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#73

Post by iamsomewalrus »

RangerScience wrote: post #66
RangerScience wrote: post #64

Earlier quoted context omitted.

Heya! I have been working on almost exactly the same thing for the last couple months :) I haven't had a chance to actually read your post yet, but I'm really looking forward to it.

Okay, neat! Had a chance to skim. It's really cool that you have the Cloud Formation code up and accessible (as well as the Glue script). But, is the lamdba accessible from anywhere? (or did I just miss the link?) We used Terraform instead of CloudFormation, although there are a few places it didn't/doesn't cover. We're also doing protobuf -> parquet instead of JSON; Scala instead of Python, and one of our major feat…

The Lambda is included in the DynamoDB exports CloudFormation templates. It's embedded in the file template itself.

Thumbs up on the Glue Dev endpoint. It's been killer. I had trouble setting up a Notebook (I wanted to get fancy with Docker) and I usually use the Python repl link that's provided.

I'm working on a follow up post that removes the Data Pipeline -> Lambda and uses the new Glue DynamoDB integration.

Looking forward to your blog post!

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#74

Post by iamsomewalrus »

bcbrown wrote: post #63
iamsomewalrus wrote: post #4

Hey! I’m the author of this post. I’m pretty chuffed to see this here. Happy to answer any questions.

I'd love to hear more about "A serverless Apache Spark environment". How is that set up? How long do those jobs tend to run for? Are those written in Java/Scala or Python? What are the pros and cons for choosing to go serverless instead of a dedicated cluster or ephemeral on-demand clusters?

The AWS Glue service provides a serverless Spark environment for running jobs. Here's a link: https://docs.aws.amazon.com/glue/latest/dg/author-job.html

* The default timeout is ~ 48 hours and you pay per Data Processing Unit (DPU) that you've provisioned the Job. * Currently it supports Python and Scala. As far as I'm aware you can't run Java jobs directly, but you can upload JAR libraries and use them in your code.

Re serverless vs dedicated / ephemeral clusters: Like with any serverless runtime environment you are trading convenience (across a few dimensions) for flexibility.

The Glue environment runs in a few limited runtimes and uses a specific version of Spark that you have no control over updating. Given that it's pretty quick to author a job, you can set the required DPU and Glue handles that, and you don't have to worry about sizing the cluster for your data size. For me most of my jobs fit within those constraints.

At some point on the cost curve it may make sense for you to move all of your jobs from Glue into a dedicated cluster on EMR. You may also get there sooner if you need to use specific frameworks or libraries.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#75

Post by iamsomewalrus »

CobrastanJorji wrote: post #72
iamsomewalrus wrote: post #55

Earlier quoted context omitted.

We use carrier pigeons from an S3 data center high in the Himalayas to a CloudFront distribution center in Atlantic City (don't ask me, ask the pigeons) to serve all requests. Just BI.

It annoys me that AWS only supports RFC 1149. When will RFC 2549 support be added?

You, dear person, win my undying support and affection.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#76

Post by iamsomewalrus »

ryanmarsh wrote: post #67

This is pretty much straight from the DynamoDB best practices. Offload infrequently accessed data (think time series data from previous months) to S3 and use another tool to query it. https://docs.aws.amazon.com/amazondynamodb/latest/developerg...

+1 Everything I've learned the hard way with DynamoDB was in the Best Practices documentation the whole time.

Now I've learned to read the Best Practices section of any AWS service before I start implementing. It saves a lot of heartache

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#79

Post by byrnehollander »

psychometry wrote: post #71
a_bonobo wrote: post #24

Earlier quoted context omitted.

The website search seems to not be tokenized, it seems to use the entire word only... so if there's a slight difference it won't find anything. The main thing I've noticed in Goodreads since they were bought is a strong focus on Kindle integration. The newer Kindles all have Goodreads integration, you can send notes etc. to the Goodreads profile, the site itself has changed very little (although Goodreads changed a l…

Feature-rich text search with elasticsearch is stupidly easy. It's a real shame.

I haven’t found this to be exactly the case - I recently worked with an engineer for more than a month tuning Elasticsearch (which required non-trivial changes to our catalog service). (It also took a good amount of effort to coordinate these changes across devices.)

Like Goodreads, we have a fairly constrained catalog and we probably get a similar amount of queries.

The search results are now much, much better, but ES has a pretty poor edit distance / fuzziness algorithm so they still aren’t perfect.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#80

Post by Jemaclus »

iamsomewalrus wrote: post #55
Jemaclus wrote: post #54

I like the approach, and I've been considering building something similar. Nice writeup. Is this used only for BI or is it used for real-time queries? The reason I ask is because goodreads.com is slow AF, and performance is a concern for me at this point.

We use carrier pigeons from an S3 data center high in the Himalayas to a CloudFront distribution center in Atlantic City (don't ask me, ask the pigeons) to serve all requests. Just BI.

Excellent. Thanks. (Have you considered using unladen swallows instead?)
Post reply on HN