Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goodreads offloads DynamoDB tables to S3 and queries them with Athena

aws.amazon.com

Post reply on HN
Print view View original

11–20 of 110 posts

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#11

Post by awinder »

mapleoin wrote: post #10

What I don't get about Athena is what happens after you've put the data in Athena? Fine, you've got SQL and tabular data, but the type of BI I've had to do usually has a graph or some other visual representation at the end rather than a table. There's only so much data you can import into Excel from a CSV that Athena produces. Usually I find periscope/cluvio to be much better tools for this and then you need to go to…

Quicksight (https://aws.amazon.com/quicksight/)? There are some other 3rd party sources that can utilize athena querying like ChartIO and Looker as well.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#12

Post by hitekker »

js2 wrote: post #9
hitekker wrote: post #7

Why is the author’s HN comment dead? Joe feeneys HN account is indeed iamawalrus; he wasn’t impersonating anyone. Further his comment just said: “Hey! I’m the author of this post. I’m pretty chuffed to see this here. Happy to answer any questions.” I want to hear what he has to say.

Unsure, but I vouched for it so it's undead now.

Thanks. I’m guessing it must have triggered some kind of filter.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#13

Post by djhworld »

mapleoin wrote: post #10

What I don't get about Athena is what happens after you've put the data in Athena? Fine, you've got SQL and tabular data, but the type of BI I've had to do usually has a graph or some other visual representation at the end rather than a table. There's only so much data you can import into Excel from a CSV that Athena produces. Usually I find periscope/cluvio to be much better tools for this and then you need to go to…

Athena has a JDBC driver and AWS Quicksight is supported if you want to use that

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#14

Post by eggman222 »

iamsomewalrus wrote: post #4

Hey! I’m the author of this post. I’m pretty chuffed to see this here. Happy to answer any questions.

Hi, thank you for taking the time. I have a couple questions: * Why is the DB scrape written as json instead of directly proto/avro/parquet? Isn't it a lot more costly to store and to handle? * How many events can aws lambda scale to in this kind of architecture?

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#15

Post by iamsomewalrus »

eggman222 wrote: post #14
iamsomewalrus wrote: post #4

Hey! I’m the author of this post. I’m pretty chuffed to see this here. Happy to answer any questions.

Hi, thank you for taking the time. I have a couple questions: * Why is the DB scrape written as json instead of directly proto/avro/parquet? Isn't it a lot more costly to store and to handle? * How many events can aws lambda scale to in this kind of architecture?

Did we just become best friends with our “I am the Walrus” references?

The DB scrape uses a template from Data Pipeline that under the hood uses the Dynamo DB Scan API. Not really surprising, but that API uses JSON. I wanted to use as much off the shelf software as I could to get data into Athena.

In this architecture Lambda is only used to listen to the SNS topic that fires when the Data Pipeline job succeeds or fails so we’re not pushing the limits of Lambda at all. You’d probably hit an EC2 limit Witt Data Pipeline before hitting your Lambda limit on the account.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#16

Post by cthalupa »

hitekker wrote: post #7

Why is the author’s HN comment dead? Joe feeneys HN account is indeed iamawalrus; he wasn’t impersonating anyone. Further his comment just said: “Hey! I’m the author of this post. I’m pretty chuffed to see this here. Happy to answer any questions.” I want to hear what he has to say.

I just had to vouch for his other post in here as well.

Looks like some automated filter, for sure.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#17

Post by iamsomewalrus »

mapleoin wrote: post #10

What I don't get about Athena is what happens after you've put the data in Athena? Fine, you've got SQL and tabular data, but the type of BI I've had to do usually has a graph or some other visual representation at the end rather than a table. There's only so much data you can import into Excel from a CSV that Athena produces. Usually I find periscope/cluvio to be much better tools for this and then you need to go to…

Athena is just a front end for the data that a typical user can understand (SQL!). the real value is that the data is: in S3, in a more efficient format (parquet), and available in the Glue catalog.

The other replies got it right w.r.t. other BI tools. If you’re using Tableau I think it integrates with Redshift, right? In that case Redshift Spectrum is an option.

If you don’t have any existing BI tools then Quicksight is an option or alternatively you can spin up an Elastic Map Reduce (EMR) cluster with your fav open source BI tools

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#18

Post by iamsomewalrus »

polskibus wrote: post #3

Weren't Goodreads bought by Amazon a couple of years ago? If so, they might've been pushed to do the move (to microservices, s3, etc) to comply with corporate guidelines/policy not because there wasn't a better/more efficient way to scale.

There’s no real policy that I’m aware of internally for teams to use microservices. Amazon has a lot of tooling to make it easy to spin up services, however.

The first major project after being acquired was to make a pared down Goodreads experience available on the Kindle Paperwhite. Our first services came out of that initiative to provide a buffer between the Kindle traffic and the Goodreads Rails app.

That being said I’ll be the first to caution small teams should avoid microservices at first for fear of creating a distributed monolith.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#20

Post by iamsomewalrus »

hitekker wrote: post #7

Why is the author’s HN comment dead? Joe feeneys HN account is indeed iamawalrus; he wasn’t impersonating anyone. Further his comment just said: “Hey! I’m the author of this post. I’m pretty chuffed to see this here. Happy to answer any questions.” I want to hear what he has to say.

Thanks all. Long time lurker, first time commenter.
Post reply on HN