Goodreads offloads DynamoDB tables to S3 and queries them with Athena • Hacker News Board

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#51

Post by ryanianian » Mon, Jul 30, 2018, 5:53 PM UTC

toomuchtodo wrote: post #25

iamsomewalrus wrote: post #18

Earlier quoted context omitted.

There’s no real policy that I’m aware of internally for teams to use microservices. Amazon has a lot of tooling to make it easy to spin up services, however. The first major project after being acquired was to make a pared down Goodreads experience available on the Kindle Paperwhite. Our first services came out of that initiative to provide a buffer between the Kindle traffic and the Goodreads Rails app. That being s…

Are internal Amazon projects cross charged for their AWS use? Or is it essentially free for internal use?

Former Amazon employee here.

They're cross-charged with lots of internal book-keeping. The rates teams "pay" internally are different from public pricing of course but expensive things externally are still expensive things internally. The cost-accounting used to be (3+ years ago) much more "just looking...no pressure to keep them down", but recently there's been huge efforts to bring internal AWS costs down, especially for EC2 usage. I heard rumors that much of the Prime Day fiasco this year could have been avoided if teams would have been permitted to spin up enough capacity.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#52

Post by ngould » Mon, Jul 30, 2018, 5:57 PM UTC

iamsomewalrus wrote: post #28

mooreds wrote: post #22

Earlier quoted context omitted.

Why didn't you push the data into a traditional data warehouse or sql database?

Data will eventually end up there. Our product and business teams are heavy users of Redshift. As mentioned in another comment I’ve found having Dynamo snapshots in Athena really useful as an oncall to sanity check snapshots (what was the state of Harry Potter 3 months ago compared to now?) and to answer product questions that can only be answered from the raw production data.

This is the first time I've come across the approach of storing database snapshots and saving them in a data lake. Do you find those snapshots are useful/used for analytics or data science end-uses, or are they more there for debugging and answering one-off questions?

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#53

Post by always_good » Mon, Jul 30, 2018, 6:04 PM UTC

ryanianian wrote: post #51

toomuchtodo wrote: post #25

Earlier quoted context omitted.

Are internal Amazon projects cross charged for their AWS use? Or is it essentially free for internal use?

Former Amazon employee here. They're cross-charged with lots of internal book-keeping. The rates teams "pay" internally are different from public pricing of course but expensive things externally are still expensive things internally. The cost-accounting used to be (3+ years ago) much more "just looking...no pressure to keep them down", but recently there's been huge efforts to bring internal AWS costs down, especial…

I'm reminded of a story my ex-Amazon friend would tell where it was a week-long process to get an extra $100-something stick of RAM for his workstation when he started there, needing various sign-off and escalations for such a grave expense.

Meanwhile he was deploying some machine-learning categorization stuff he was working on to some cluster that would cost five figures of compute each time, and nobody batted an eye.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#54

Post by Jemaclus » Mon, Jul 30, 2018, 6:11 PM UTC

I like the approach, and I've been considering building something similar. Nice writeup. Is this used only for BI or is it used for real-time queries? The reason I ask is because goodreads.com is slow AF, and performance is a concern for me at this point.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#55

Post by iamsomewalrus » Mon, Jul 30, 2018, 6:19 PM UTC

Jemaclus wrote: post #54

I like the approach, and I've been considering building something similar. Nice writeup. Is this used only for BI or is it used for real-time queries? The reason I ask is because goodreads.com is slow AF, and performance is a concern for me at this point.

We use carrier pigeons from an S3 data center high in the Himalayas to a CloudFront distribution center in Atlantic City (don't ask me, ask the pigeons) to serve all requests.

Just BI.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#56

Post by ryeguy_24 » Mon, Jul 30, 2018, 6:20 PM UTC

kaycebasques wrote: post #8

I use Goodreads as my main book tracker and reviewer. My impression is that Amazon bought it for whatever reason, and then ignored it. It seems like there's so much more potential to a social community around books than what Goodreads offers. The UI also never fails to disappoint. One of the core user features, searching for books, has all sorts of weird quirks. I'll use a very specific query that should guarantee a…

We agree. My co-founder and I built helloreads.com to address the UI concern above. We never felt like Goodreads got the necessary UI upgrade it needed.

http://www.helloreads.com

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#57

Post by some_account » Mon, Jul 30, 2018, 6:22 PM UTC

Cidan wrote: post #47

iamsomewalrus wrote: post #39

Earlier quoted context omitted.

:this: A concept that's underlying the move to a datalake architecture (read: keeping your data in its rawest form, and its transforms in S3 or HDFS) is decoupling your compute from storage. Motivating example: you have huge tables in Redshift that are either infrequently accessed or the usefulness of the data decays over time (website logs, customer order information). In this scenario you're paying a lot just to ke…

BigQuery on GCP does this as well, without any extra work. (Disclaimer: I work for Google)

Google could shut down an entire account without any communication and kill a business in the process.

I would never pick Google for anything important.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#58

Post by iamsomewalrus » Mon, Jul 30, 2018, 6:29 PM UTC

ngould wrote: post #52

iamsomewalrus wrote: post #28

Earlier quoted context omitted.

Data will eventually end up there. Our product and business teams are heavy users of Redshift. As mentioned in another comment I’ve found having Dynamo snapshots in Athena really useful as an oncall to sanity check snapshots (what was the state of Harry Potter 3 months ago compared to now?) and to answer product questions that can only be answered from the raw production data.

This is the first time I've come across the approach of storing database snapshots and saving them in a data lake. Do you find those snapshots are useful/used for analytics or data science end-uses, or are they more there for debugging and answering one-off questions?

I have personally only used them for debugging one off questions. That was my original intent. We do have teams that are considering using the snapshots for ML problems.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#59

Post by bcbrown » Mon, Jul 30, 2018, 6:31 PM UTC

wenc wrote: post #40

ecnahc515 wrote: post #30

Earlier quoted context omitted.

For a lot of teams, S3 is a data warehouse, and you can treat it just like HDFS for the most part with most things in the big-data ecosystem. Presto works well for letting you access it from these locations without having to explicitly import it (assuming it's in a traditional data warehouse or a common SQL DB).

I wonder if anyone here has a good heuristic for identifying the conditions under which using S3 + SQL layer as a data warehouse is a better choice than a SQL database? I've been exploring the former and it seems to only make sense if the size of your data is at a scale that is beyond what a single SQL database instance can handle, and even then, you can continue to scale out with systems like Citus so the limit isn'…

I've worked with a S3 + SQL system. It was used for serving data for a reporting dashboard where the stored data was in the 0.1-10 TB range. As the use case was only semi-interactive (users didn't mind waiting 1-10 seconds for a report), and all the queries were pre-defined, this solution was a good fit.

I think it makes sense when there's no in-place updates; either querying write-once data like logs or the output of batch data processing roll-ups that replace the previous data. The less you need the relational model (like joins), the better, but some of those needs can be met through careful design of the storage schema and denormalization.

I wouldn't advocate this sort of solution if your requirements include in-place updates of existing data, frequent/granular updates of new data, expressive ad-hoc queries that use the full capability of relational algebra, or tight latency requirements. You also lose the safety net of referential integrity and table-level constraints, as those are now enforced in custom code that can have bugs.

I would say maintaining this system cost about a half-engineer for ongoing maintenance and new functionality.

Re: Goodreads offloads DynamoDB tables to S3 and queries them with Athena

#60

Post by toomuchtodo » Mon, Jul 30, 2018, 6:34 PM UTC

ryanianian wrote: post #51

toomuchtodo wrote: post #25

Earlier quoted context omitted.

Are internal Amazon projects cross charged for their AWS use? Or is it essentially free for internal use?

Former Amazon employee here. They're cross-charged with lots of internal book-keeping. The rates teams "pay" internally are different from public pricing of course but expensive things externally are still expensive things internally. The cost-accounting used to be (3+ years ago) much more "just looking...no pressure to keep them down", but recently there's been huge efforts to bring internal AWS costs down, especial…

Thanks, I really appreciate the insight.