Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced the cost of building Mastodon at Twitter-scale by 100x

blog.redplanetlabs.com

Post reply on HN
Print view View original

91–100 of 376 posts

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#91

Post by dustingetz »

3cats-in-a-coat wrote: post #66
nathanmarz wrote: post #33

Earlier quoted context omitted.

You're missing the point. Rama is a generic platform that provides a new baseline for how expensive it is to build applications at scale. There's nothing about Rama specific to social networks. What we're showing is that Rama creates a new era in software engineering where the cost of building applications at scale is radically reduced. With Rama, anyone embarking on a new application today has a radically different…

If I grasp the essence of Rama: - "Depots" are event streams (for event sourced data repositories) - ETL read one or more streams and project them to indexable read models... - Which read models are called "PStates" and represent nested combinations of indices like hashtables, b-trees, linked lists and so on. The point of those being they have the data in fast to query way. - And you have query engine which splits a…

you're missing automatic/free linear scaling

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#93

Post by DoesntMatter22 »

nathanmarz wrote: post #83
DoesntMatter22 wrote: post #59

Earlier quoted context omitted.

"We recreated a service from 2007 and it's so much faster!"

As we mentioned in the article, Instagram just spent ~25 person-years building Threads which is a barebones clone of Twitter. Not only did we build our instance 30x faster than that, we have way more features like federation, hashtag follows, polls, DMs, global timelines, and more. And Instagram didn't start from scratch as Instagram/Meta already had infrastructure powering similar products. https://www.washingtonpos…

You said this in another post.

> That's why we're comparing it to the cost of Twitter's original consumer product.

Plus, you cloned a preexisting architecture. FB wrote theirs from scratch. Not apples to apples. This is much easier.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#94

Post by [deleted] »

nathanmarz wrote: post #62
ceejayoz wrote: post #32

Earlier quoted context omitted.

Mastodon and Twitter don't do the same amount of work per post. Mastodon doesn't have a recommendation engine, they don't have an advertising engine, they don't scan every post for CSAM, there's no global search, etc. (Some of these things are good not to have, but they still drastically change the scope.) Claiming to have enabled significant scaling of a Mastodon/ActivityPub-compatible instance is fine. Claiming to…

That's why we're comparing it to the cost of Twitter's original consumer product. As a demonstration of Rama, we scoped this project to the entirety of Mastodon which is roughly equivalent to Twitter's original consumer product (actually, it's probably greater in scope with additional features like hashtag follows and more complex filter/mute capabilities). All those use cases you listed absolutely can be implemented…

[deleted]

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#95

Post by nathanmarz »

3cats-in-a-coat wrote: post #66
nathanmarz wrote: post #33

Earlier quoted context omitted.

You're missing the point. Rama is a generic platform that provides a new baseline for how expensive it is to build applications at scale. There's nothing about Rama specific to social networks. What we're showing is that Rama creates a new era in software engineering where the cost of building applications at scale is radically reduced. With Rama, anyone embarking on a new application today has a radically different…

If I grasp the essence of Rama: - "Depots" are event streams (for event sourced data repositories) - ETL read one or more streams and project them to indexable read models... - Which read models are called "PStates" and represent nested combinations of indices like hashtables, b-trees, linked lists and so on. The point of those being they have the data in fast to query way. - And you have query engine which splits a…

From the post:

Individually, none of these concepts are new. I’m sure you’ve seen them all before. You may be tempted to dismiss Rama’s programming model as just a combination of event sourcing and materialized views. But what Rama does is integrate and generalize these concepts to such an extent that you can build entire backends end-to-end without any of the impedance mismatches or complexity that characterize and overwhelm existing systems.

You have the general model correct, but here are a few clarifications:

- PStates are partitioned, durable, replicated indexes that are represented as arbitrary combinations of data structures. A PState can be as simple an an integer per partition, or it can be complex like a map of lists of maps of sets. PStates allow you to shape your indexes to perfectly match your application's use cases.

- I wouldn't call Rama queries an "engine", as it's considerably more straightforward in how it works than something like SQL. The base query API is called "paths", which are an imperative way to concisely reach into one partition of one PState to fetch or aggregate values. There's also "query topologies" which are predefined, on-demand distributed computations that can fetch and aggregate data from many partitions of many PStates.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#97

Post by 3cats-in-a-coat »

dustingetz wrote: post #91
3cats-in-a-coat wrote: post #66

Earlier quoted context omitted.

If I grasp the essence of Rama: - "Depots" are event streams (for event sourced data repositories) - ETL read one or more streams and project them to indexable read models... - Which read models are called "PStates" and represent nested combinations of indices like hashtables, b-trees, linked lists and so on. The point of those being they have the data in fast to query way. - And you have query engine which splits a…

you're missing automatic/free linear scaling

Systems that promise "free linear scaling" without qualifiers either withhold or have not analyzed/realized their bottlenecks yet. Say if there is eventual consistency maybe the "eventuality" becomes so long that the service fails at its purpose. Or the communication link bandwidth is exhausted between key business logic (mutation event generating) services, and so on.

The only systems that scale linearly are stateless systems. Mastodon is not stateless. And even stateless systems hit some bottlenecks eventually, as they exist and run in a scale-variant Universe.

So this claim by itself doesn't immediately impress me, just turns my red lights on, awaiting further investigation. But we can of course discuss why this claim is made and how is it supported. The article is long so I've not had the chance to read it entirely yet.

But we have X number of event streams mapped through Y number of ETLs to produce Z number of read model indices, in a shape that seems to form a highly interlinked DAG, which eventually loops back on itself in terms of message flow. Just the increased cross-chatter here as we introduce more features suggests non-linear scaling.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#98

Post by roguas »

erlend_sh wrote: post #84
nathanmarz wrote: post #53

Earlier quoted context omitted.

We're actually not asking anyone to give up anything. First off, it has a simple integration API (which you'll be able to see the details of next week) that allows it to seamlessly integrate with any other backend tool (databases, monitoring systems, queues, etc.). So Rama can be incrementally introduced into any existing architecture. Second, Rama has a pure Java API and is not a bespoke language. So no new language…

So Rama-powered apps need to be written in Java? Or will any JVM language work? And the Rama core will remain closed-source? That part seems like the toughest sell of all, at a time when the vast majority of developer tooling and backends are open source or at the very least source-available.

Since all jvm languages usually have "ffi" to javaapis/javalibs, I would say yes.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#100

Post by nathanmarz »

buro9 wrote: post #86

Measuring "Twitter Scale" by tweets per second seems to be not how I would measure it. Updates per second to end users who follow the 7K tweets per second seems more realistic, it's the timelines and notifications that hurt, not the top of ingest tweets per second prior to the fan out... and then of course it's whether you can do that continuously so as not to back up on it.

That's why we're saying "at 403 fanout". The bottleneck of Mastodon/Twitter is timeline writes, which is posts/second multiplied by the average number of followers per post. So our instance is doing 1.4M timeline writes / second.

Another important metric is "time to deliver to follower timelines", which is tricky due to how much variance there can be every second due to the extremely unbalanced social graph. When someone with 20M followers posts, that multiples the number of needed timeline writes by 15x. We went into depth in our post on how we handled that to provide fairness by preventing these big users from hogging all the resources all at once.

Post reply on HN