Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goodreads offloads DynamoDB tables to S3 and queries them with Athena

aws.amazon.com

Post reply on HN
Print view View original

21–30 of 110 posts

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#21

Post by mooreds »

iamsomewalrus wrote: post #18
polskibus wrote: post #3

Weren't Goodreads bought by Amazon a couple of years ago? If so, they might've been pushed to do the move (to microservices, s3, etc) to comply with corporate guidelines/policy not because there wasn't a better/more efficient way to scale.

There’s no real policy that I’m aware of internally for teams to use microservices. Amazon has a lot of tooling to make it easy to spin up services, however. The first major project after being acquired was to make a pared down Goodreads experience available on the Kindle Paperwhite. Our first services came out of that initiative to provide a buffer between the Kindle traffic and the Goodreads Rails app. That being s…

> That being said I’ll be the first to caution small teams should avoid microservices at first for fear of creating a distributed monolith.

What is your heuristic for determining when it is the right time to look at microservices?

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#23

Post by hkchad »

mapleoin wrote: post #10

What I don't get about Athena is what happens after you've put the data in Athena? Fine, you've got SQL and tabular data, but the type of BI I've had to do usually has a graph or some other visual representation at the end rather than a table. There's only so much data you can import into Excel from a CSV that Athena produces. Usually I find periscope/cluvio to be much better tools for this and then you need to go to…

There are a bunch of BI tools that can connect to Athena, we use Tableau to cut/slice the data and test out various hypothesis before building a real backend to utilize the data in new ways. I've found RedShift to be a bit expensive for such use cases as our Tableau users only query for a few hrs/day so keeping a RedShift cluster up and running is way overkill, Athena is a good stopgap.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#24

Post by a_bonobo »

kaycebasques wrote: post #8

I use Goodreads as my main book tracker and reviewer. My impression is that Amazon bought it for whatever reason, and then ignored it. It seems like there's so much more potential to a social community around books than what Goodreads offers. The UI also never fails to disappoint. One of the core user features, searching for books, has all sorts of weird quirks. I'll use a very specific query that should guarantee a…

The website search seems to not be tokenized, it seems to use the entire word only... so if there's a slight difference it won't find anything.

The main thing I've noticed in Goodreads since they were bought is a strong focus on Kindle integration. The newer Kindles all have Goodreads integration, you can send notes etc. to the Goodreads profile, the site itself has changed very little

(although Goodreads changed a lot of rules around the ability to delete user reviews if they focus too much on author behaviour around the time the Amazon purchase went through https://www.washingtonpost.com/blogs/compost/wp/2013/09/23/i... )

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#25

Post by toomuchtodo »

iamsomewalrus wrote: post #18
polskibus wrote: post #3

Weren't Goodreads bought by Amazon a couple of years ago? If so, they might've been pushed to do the move (to microservices, s3, etc) to comply with corporate guidelines/policy not because there wasn't a better/more efficient way to scale.

There’s no real policy that I’m aware of internally for teams to use microservices. Amazon has a lot of tooling to make it easy to spin up services, however. The first major project after being acquired was to make a pared down Goodreads experience available on the Kindle Paperwhite. Our first services came out of that initiative to provide a buffer between the Kindle traffic and the Goodreads Rails app. That being s…

Are internal Amazon projects cross charged for their AWS use? Or is it essentially free for internal use?

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#26

Post by iamsomewalrus »

dglass wrote: post #6

We did something similar at a previous position I had, except we set up an amazon lambda function that was triggered on every insert or update to a DynamoDB table. The lambda function flattened the updated record and inserted it into our redshift cluster, which gave us a real-time ETL pipeline for our DynamoDB data. That allowed us to report on our DynamoDB data just like our relational data.

We have another GR team that does that and it works well. It’s complicated by the fact that at Amazon every team uses their own AWS account. Concretely, a service’s DynamoDB tables don’t exist in the same AWS account let alone the same VPC as the redshift cluster. Obviously, you can figure out the permissions, etc.

We’re trying to get to a place where we have the data in S3 for engineers to build products off of and for the oncalls to do sanity checks and the data in Redshift for our BI needs.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#27

Post by mi100hael »

manojlds wrote: post #2

First lesson - it's taken Goodreads so long and so many users to actually start moving to microservices.

I'm not really surprised. Goodreads doesn't have nearly the same level of "real-time" interaction that other social networks do. People can only read (and therefore review) so many books, so it's not like there are millions of users constantly posting new content/comments/etc.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#28

Post by iamsomewalrus »

mooreds wrote: post #22
iamsomewalrus wrote: post #4

Hey! I’m the author of this post. I’m pretty chuffed to see this here. Happy to answer any questions.

Why didn't you push the data into a traditional data warehouse or sql database?

Data will eventually end up there. Our product and business teams are heavy users of Redshift.

As mentioned in another comment I’ve found having Dynamo snapshots in Athena really useful as an oncall to sanity check snapshots (what was the state of Harry Potter 3 months ago compared to now?) and to answer product questions that can only be answered from the raw production data.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#29

Post by iamsomewalrus »

mooreds wrote: post #21
iamsomewalrus wrote: post #18

Earlier quoted context omitted.

There’s no real policy that I’m aware of internally for teams to use microservices. Amazon has a lot of tooling to make it easy to spin up services, however. The first major project after being acquired was to make a pared down Goodreads experience available on the Kindle Paperwhite. Our first services came out of that initiative to provide a buffer between the Kindle traffic and the Goodreads Rails app. That being s…

> That being said I’ll be the first to caution small teams should avoid microservices at first for fear of creating a distributed monolith. What is your heuristic for determining when it is the right time to look at microservices?

That’s “the question”. I will do my best to bluff through it.

Engineering team size: you have teams large enough (~3-4) to own and iterate on a subset of related functionality for the long term.

Tooling: you have a builder tools type team that provides a tooling and observability happy path.

Traffic scale: you have functionality that operates at 2 or more order of magnitude higher that the rest of your application.

Decoupled: you have functionality that can be decoupled from your main app and, most importantly, isn’t required for your main app’s uptime. Like a search service or something.

A combination of the above.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#30

Post by ecnahc515 »

mooreds wrote: post #22
iamsomewalrus wrote: post #4

Hey! I’m the author of this post. I’m pretty chuffed to see this here. Happy to answer any questions.

Why didn't you push the data into a traditional data warehouse or sql database?

For a lot of teams, S3 is a data warehouse, and you can treat it just like HDFS for the most part with most things in the big-data ecosystem. Presto works well for letting you access it from these locations without having to explicitly import it (assuming it's in a traditional data warehouse or a common SQL DB).
Post reply on HN