Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goodreads offloads DynamoDB tables to S3 and queries them with Athena

aws.amazon.com

Post reply on HN
Print view View original

31–40 of 110 posts

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#31

Post by iamsomewalrus »

kaycebasques wrote: post #8

I use Goodreads as my main book tracker and reviewer. My impression is that Amazon bought it for whatever reason, and then ignored it. It seems like there's so much more potential to a social community around books than what Goodreads offers. The UI also never fails to disappoint. One of the core user features, searching for books, has all sorts of weird quirks. I'll use a very specific query that should guarantee a…

We can, and will do better in the areas you’ve mentioned. I’ve been bitten by that same search bug :-(. I’ll cut a ticket for that.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#32

Post by [deleted] »

starchand wrote: post #5
polskibus wrote: post #3

Weren't Goodreads bought by Amazon a couple of years ago? If so, they might've been pushed to do the move (to microservices, s3, etc) to comply with corporate guidelines/policy not because there wasn't a better/more efficient way to scale.

Yeah 2013. You get goodreads info inside your profile section of the kindle apps.

[deleted]

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#33

Post by iamsomewalrus »

toomuchtodo wrote: post #25
iamsomewalrus wrote: post #18

Earlier quoted context omitted.

There’s no real policy that I’m aware of internally for teams to use microservices. Amazon has a lot of tooling to make it easy to spin up services, however. The first major project after being acquired was to make a pared down Goodreads experience available on the Kindle Paperwhite. Our first services came out of that initiative to provide a buffer between the Kindle traffic and the Goodreads Rails app. That being s…

Are internal Amazon projects cross charged for their AWS use? Or is it essentially free for internal use?

Come join my team and I’ll be happy to spill all the beans!

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#34

Post by veritas3241 »

kaycebasques wrote: post #8

I use Goodreads as my main book tracker and reviewer. My impression is that Amazon bought it for whatever reason, and then ignored it. It seems like there's so much more potential to a social community around books than what Goodreads offers. The UI also never fails to disappoint. One of the core user features, searching for books, has all sorts of weird quirks. I'll use a very specific query that should guarantee a…

I noticed recently that the Android UI was updated and a lot of my frustrations have gone away. Primarily around updating my current progress and adding to different shelves seems wot work much better.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#35

Post by veritas3241 »

iamsomewalrus wrote: post #31
kaycebasques wrote: post #8

I use Goodreads as my main book tracker and reviewer. My impression is that Amazon bought it for whatever reason, and then ignored it. It seems like there's so much more potential to a social community around books than what Goodreads offers. The UI also never fails to disappoint. One of the core user features, searching for books, has all sorts of weird quirks. I'll use a very specific query that should guarantee a…

We can, and will do better in the areas you’ve mentioned. I’ve been bitten by that same search bug :-(. I’ll cut a ticket for that.

Is there a public tracker for this?

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#36

Post by iamsomewalrus »

veritas3241 wrote: post #35
iamsomewalrus wrote: post #31

Earlier quoted context omitted.

We can, and will do better in the areas you’ve mentioned. I’ve been bitten by that same search bug :-(. I’ll cut a ticket for that.

Is there a public tracker for this?

No, but we have a really responsive customer service team that watches the Goodreads feedback group very closely.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#37

Post by iamsomewalrus »

veritas3241 wrote: post #34
kaycebasques wrote: post #8

I use Goodreads as my main book tracker and reviewer. My impression is that Amazon bought it for whatever reason, and then ignored it. It seems like there's so much more potential to a social community around books than what Goodreads offers. The UI also never fails to disappoint. One of the core user features, searching for books, has all sorts of weird quirks. I'll use a very specific query that should guarantee a…

I noticed recently that the Android UI was updated and a lot of my frustrations have gone away. Primarily around updating my current progress and adding to different shelves seems wot work much better.

That’s good to hear! The underlying architecture project has been in motion for a few years. We’re excited that it’s out.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#38

Post by sctb »

hitekker wrote: post #7

Why is the author’s HN comment dead? Joe feeneys HN account is indeed iamawalrus; he wasn’t impersonating anyone. Further his comment just said: “Hey! I’m the author of this post. I’m pretty chuffed to see this here. Happy to answer any questions.” I want to hear what he has to say.

Their posts were caught by a software filter. If you email us at hn@ycombinator.com we can be sure to see it—otherwise we may or may not notice on the site.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#39

Post by iamsomewalrus »

ecnahc515 wrote: post #30
mooreds wrote: post #22

Earlier quoted context omitted.

Why didn't you push the data into a traditional data warehouse or sql database?

For a lot of teams, S3 is a data warehouse, and you can treat it just like HDFS for the most part with most things in the big-data ecosystem. Presto works well for letting you access it from these locations without having to explicitly import it (assuming it's in a traditional data warehouse or a common SQL DB).

:this: A concept that's underlying the move to a datalake architecture (read: keeping your data in its rawest form, and its transforms in S3 or HDFS) is decoupling your compute from storage.

Motivating example: you have huge tables in Redshift that are either infrequently accessed or the usefulness of the data decays over time (website logs, customer order information). In this scenario you're paying a lot just to keep data in Redshift (storage) but a large subset of the data is laying dormant (no compute).

If you're bought into the Redshift ecosystem this is where Redshift Spectrum comes in. If you're a smaller company you could just store the data in S3 and "spin up" the compute when you need it (Athena, Glue jobs, or Elastic Map Reduce clusters).

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#40

Post by wenc »

ecnahc515 wrote: post #30
mooreds wrote: post #22

Earlier quoted context omitted.

Why didn't you push the data into a traditional data warehouse or sql database?

For a lot of teams, S3 is a data warehouse, and you can treat it just like HDFS for the most part with most things in the big-data ecosystem. Presto works well for letting you access it from these locations without having to explicitly import it (assuming it's in a traditional data warehouse or a common SQL DB).

I wonder if anyone here has a good heuristic for identifying the conditions under which using S3 + SQL layer as a data warehouse is a better choice than a SQL database?

I've been exploring the former and it seems to only make sense if the size of your data is at a scale that is beyond what a single SQL database instance can handle, and even then, you can continue to scale out with systems like Citus so the limit isn't a hard one. SQL gives one so much (data mutability, consistency, indexes, etc.) that I am hesitant to give it up unless the tradeoffs make sense.

Post reply on HN