Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goodreads offloads DynamoDB tables to S3 and queries them with Athena

aws.amazon.com

Post reply on HN
Print view View original

61–70 of 110 posts

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#61

Post by Roritharr »

a_bonobo wrote: post #24
kaycebasques wrote: post #8

I use Goodreads as my main book tracker and reviewer. My impression is that Amazon bought it for whatever reason, and then ignored it. It seems like there's so much more potential to a social community around books than what Goodreads offers. The UI also never fails to disappoint. One of the core user features, searching for books, has all sorts of weird quirks. I'll use a very specific query that should guarantee a…

The website search seems to not be tokenized, it seems to use the entire word only... so if there's a slight difference it won't find anything. The main thing I've noticed in Goodreads since they were bought is a strong focus on Kindle integration. The newer Kindles all have Goodreads integration, you can send notes etc. to the Goodreads profile, the site itself has changed very little (although Goodreads changed a l…

What drives me nuts is that the Goodreads integration never made it to Germany. It's frustrating to manually carry over the data while the code is just sitting there behind a country flag.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#62

Post by iamsomewalrus »

manojlds wrote: post #2

First lesson - it's taken Goodreads so long and so many users to actually start moving to microservices.

In general, it's not uncommon for startups of the Goodreads vintage to outgrow a simple Rails app and use a more service oriented architecture. We've actually had services for about 5 years now, but we haven't been very vocal about it.

As I mentioned in another comment I, personally, wouldn't advocate for new teams or startups to use a service oriented architecture right out the gate. It's too easy to end up with a distributed monolith (circular dependencies between services, services uptimes that are tightly coupled). Engineers also tend to underestimate the build tooling, observability excellence and discipline you need to make it seamless.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#63

Post by bcbrown »

iamsomewalrus wrote: post #4

Hey! I’m the author of this post. I’m pretty chuffed to see this here. Happy to answer any questions.

I'd love to hear more about "A serverless Apache Spark environment". How is that set up? How long do those jobs tend to run for? Are those written in Java/Scala or Python? What are the pros and cons for choosing to go serverless instead of a dedicated cluster or ephemeral on-demand clusters?

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#64

Post by RangerScience »

iamsomewalrus wrote: post #4

Hey! I’m the author of this post. I’m pretty chuffed to see this here. Happy to answer any questions.

Heya! I have been working on almost exactly the same thing for the last couple months :)

I haven't had a chance to actually read your post yet, but I'm really looking forward to it.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#65

Post by RangerScience »

iamsomewalrus wrote: post #31
kaycebasques wrote: post #8

I use Goodreads as my main book tracker and reviewer. My impression is that Amazon bought it for whatever reason, and then ignored it. It seems like there's so much more potential to a social community around books than what Goodreads offers. The UI also never fails to disappoint. One of the core user features, searching for books, has all sorts of weird quirks. I'll use a very specific query that should guarantee a…

We can, and will do better in the areas you’ve mentioned. I’ve been bitten by that same search bug :-(. I’ll cut a ticket for that.

Oh hey! I would like to use Goodreads, but I whenever I try to load the page (with JS enabled) I get stuck in some kind of weird automatic-login->login-fail loop. I can go hit the page and give you more details if you want.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#66

Post by RangerScience »

RangerScience wrote: post #64
iamsomewalrus wrote: post #4

Hey! I’m the author of this post. I’m pretty chuffed to see this here. Happy to answer any questions.

Heya! I have been working on almost exactly the same thing for the last couple months :) I haven't had a chance to actually read your post yet, but I'm really looking forward to it.

Okay, neat! Had a chance to skim. It's really cool that you have the Cloud Formation code up and accessible (as well as the Glue script). But, is the lamdba accessible from anywhere? (or did I just miss the link?)

We used Terraform instead of CloudFormation, although there are a few places it didn't/doesn't cover.

We're also doing protobuf -> parquet instead of JSON; Scala instead of Python, and one of our major feature/issues is that the incoming data is out-of-order; Firehose outputs to partitions based on when the event arrived, and we're repartitioning based on when the event occurred (according to a timestamp in the message)

I see you ran into the capitalization issue too :) I ran across docs somewhere that said something along the lines of Glue downcasing column names, which definitely fits observed behavior.

I'm hoping to turn what we've done into a similar post, although I think it'll be a month or two before I can get to that.

Edit: Oh, I also got a lot of mileage out of the Zeppelin Notebooks. Way better than a raw dev endpoint, but watch out on the cost for both :)

The notebooks can be provisioned with almost a single click from the Dev endpoint console, but they changed the recipe halfway through my work on it, and now you have to also SSH into the box and run a script to setup some of the security. :/ Still totally worth it, tho.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#67

Post by ryanmarsh »

This is pretty much straight from the DynamoDB best practices. Offload infrequently accessed data (think time series data from previous months) to S3 and use another tool to query it.

https://docs.aws.amazon.com/amazondynamodb/latest/developerg...

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#68

Post by favorited »

kaycebasques wrote: post #8

I use Goodreads as my main book tracker and reviewer. My impression is that Amazon bought it for whatever reason, and then ignored it. It seems like there's so much more potential to a social community around books than what Goodreads offers. The UI also never fails to disappoint. One of the core user features, searching for books, has all sorts of weird quirks. I'll use a very specific query that should guarantee a…

Better integration with Audible (another Amazon property) seems like it would be a slam dunk. One of my largest frustrations with my audiobook habit is that there is no single source of truth for my to-read list. I need the books I'm interested in to be in an Audible wish list so I can get alerts when they're on sale, but then it's divorced from my currently-reading and completed indices in GR.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#69

Post by djhworld »

kaycebasques wrote: post #8

I use Goodreads as my main book tracker and reviewer. My impression is that Amazon bought it for whatever reason, and then ignored it. It seems like there's so much more potential to a social community around books than what Goodreads offers. The UI also never fails to disappoint. One of the core user features, searching for books, has all sorts of weird quirks. I'll use a very specific query that should guarantee a…

I've always found the recommendations feature a bit rubbish as well, but recommendations are a hard problem so I'll cut them some slack

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#70

Post by sdoering »

favorited wrote: post #68
kaycebasques wrote: post #8

I use Goodreads as my main book tracker and reviewer. My impression is that Amazon bought it for whatever reason, and then ignored it. It seems like there's so much more potential to a social community around books than what Goodreads offers. The UI also never fails to disappoint. One of the core user features, searching for books, has all sorts of weird quirks. I'll use a very specific query that should guarantee a…

Better integration with Audible (another Amazon property) seems like it would be a slam dunk. One of my largest frustrations with my audiobook habit is that there is no single source of truth for my to-read list. I need the books I'm interested in to be in an Audible wish list so I can get alerts when they're on sale, but then it's divorced from my currently-reading and completed indices in GR.

I'd second this. Not only the kindle integration but also audible. They could even tie in my order history for books.

That would mostly create a good base of about 90% of my books in the last 10 years.

Post reply on HN