Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced the cost of building Mastodon at Twitter-scale by 100x

blog.redplanetlabs.com

Post reply on HN
Print view View original

121–130 of 376 posts

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#121

Post by faitswulff »

nathanmarz wrote: post #100
buro9 wrote: post #86

Measuring "Twitter Scale" by tweets per second seems to be not how I would measure it. Updates per second to end users who follow the 7K tweets per second seems more realistic, it's the timelines and notifications that hurt, not the top of ingest tweets per second prior to the fan out... and then of course it's whether you can do that continuously so as not to back up on it.

That's why we're saying "at 403 fanout". The bottleneck of Mastodon/Twitter is timeline writes, which is posts/second multiplied by the average number of followers per post. So our instance is doing 1.4M timeline writes / second. Another important metric is "time to deliver to follower timelines", which is tricky due to how much variance there can be every second due to the extremely unbalanced social graph. When som…

I heard somewhere that one of the particular challenges of Twitter's scale is not the average fanout, but the outliers where millions or tens of millions of users follow a single account. Does your simulation take that into account?

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#122

Post by StephenAmar »

afro88 wrote: post #118

Looks amazing and incredibly smart. But I found the LOC and implementation time comparisons to Twitter and Threads very disingenuous. It makes me wonder what other wool will be pulled over our eyes with Rama in future (or important real world details missed / future footguns). Still super impressive. Reminds me of when I discovered Elixir while building a social-ish music discovery app. Switching the backend from Rai…

+1 the comparisons are not great. How much engineer-hours did it take to build Rama itself?

The numbers they got for Twitter likely include the time it took to build their infrastructure, common libraries (like finagle,…)

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#123

Post by nathanmarz »

faitswulff wrote: post #121
nathanmarz wrote: post #100

Earlier quoted context omitted.

That's why we're saying "at 403 fanout". The bottleneck of Mastodon/Twitter is timeline writes, which is posts/second multiplied by the average number of followers per post. So our instance is doing 1.4M timeline writes / second. Another important metric is "time to deliver to follower timelines", which is tricky due to how much variance there can be every second due to the extremely unbalanced social graph. When som…

I heard somewhere that one of the particular challenges of Twitter's scale is not the average fanout, but the outliers where millions or tens of millions of users follow a single account. Does your simulation take that into account?

Yes, we discussed this at length in the post.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#125

Post by reilly3000 »

jedberg wrote: post #55
Pxtl wrote: post #25

I've seen many people describe frameworks like this - you know, first you have the slow back-end event-driven master database that you don't query live against, then you've got eventual-consistency flows against the various data-warehouses and data-stores and partitioned sharded databases in useful query-friendly layouts that you actually read live from... and I never see it clearly explained: how do you read a chang…

The short answer is write-through cache. You write the update directly to the cache closest to the user and into the eventually consistent queue. We did this at reddit. When you make a comment the HTML is rendered and put straight into the cache, and the raw text is put into the queue to go into the database. Same with votes. I suspect they do this client side now, which is now the closest cache to the user, but back…

DynamoDB’s DAX cache espouses the same approach.

I have to say in my ~12 years as an active Redditor I can’t recall a time where I saw any real state issues, even with rapidly changing votes, etc. Bravo!? Now that we’re beyond the days of molten servers, I have to say its overall reliability in the face of massive spiky traffic is quite a feat.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#126

Post by nathanmarz »

gfodor wrote: post #58

Something I'm immediately thinking about with this is change management and inertia at the early stages of a new, underdefined project. Less code is great, the big question is how such a system compares to the usual hack-and-slash method of getting a v1 up and running as you search for PMF from the perspectives of ops, cost, data migrations, rapid deployments, and so on. Presumably, the idea here is to start from the…

Thanks, this is a good idea for a another post.

The basic operation Rama provides for evolving an application over time is "module update". This lets you update the code for an existing module, including adding new depots, PStates, and topologies.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#128

Post by nathanmarz »

lionkor wrote: post #116

Very interesting, looking forward to reading the docs once they come out. Why Java?

It's one of the most widely used/known programming languages in the world.

It's a Java API so any JVM language can be used (Clojure, Scala, etc.).

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#129

Post by 3cats-in-a-coat »

nathanmarz wrote: post #95
3cats-in-a-coat wrote: post #66

Earlier quoted context omitted.

If I grasp the essence of Rama: - "Depots" are event streams (for event sourced data repositories) - ETL read one or more streams and project them to indexable read models... - Which read models are called "PStates" and represent nested combinations of indices like hashtables, b-trees, linked lists and so on. The point of those being they have the data in fast to query way. - And you have query engine which splits a…

From the post: Individually, none of these concepts are new. I’m sure you’ve seen them all before. You may be tempted to dismiss Rama’s programming model as just a combination of event sourcing and materialized views. But what Rama does is integrate and generalize these concepts to such an extent that you can build entire backends end-to-end without any of the impedance mismatches or complexity that characterize and…

Thanks, I will read more soon! I'm curious... how do you resolve the "impedance mismatch" between some "canonical" models that business decisions are made, based upon, which need to be synchronous with the depots (and mutually synchronous with other models sharing fragments of the same data), and the eventually consistent read models, which have a more lax constraint on how up to date they are?

How do you ensure consistency here? How do you organize it in the data flow?

Say I update a user, because that user seems to still be there in the query result/indexes, but actually an event for this user being deleted has happened some time ago?

This can also happen I suppose of the depots run queries themselves on PState in order to determine if a certain event is valid at all or not, and how exactly to carry it out.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#130

Post by dang »

Kiro wrote: post #60
dang wrote: post #41

Earlier quoted context omitted.

I'm happy to correct it if anyone suggests a better one. The intention is to find a neutral title that accurately reflects what the article itself is saying. We've learned that when an article's original title generates complaints like https://news.ycombinator.com/item?id=37137317 , the thread is likely to get derailed by shallow arguing about the title. It's in both the author's interest and the community's for us t…

Alright, sounds reasonable. I think the problem here is that the author specifically says (in a sibling comment) that the point is not Mastodon and now it's in the title. Maybe they're fine with it though.

I'm no expert and definitely get things wrong - we only skim things and make a first crack at an attempt, and then rely on other people to refine it. If Nathan or someone else wants to suggest a more accurate and neutral title, we can do that - the goal is simply to clear the discussion space for something more interesting than title fever (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...). But now I'm repeating myself!
Post reply on HN