Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced the cost of building Mastodon at Twitter-scale by 100x

blog.redplanetlabs.com

Post reply on HN
Print view View original

251–260 of 376 posts

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#251

Post by falsandtru »

dataangel wrote: post #192

I do C++ backend work in a non-web industry and this entire post is Greek to me. Even though this is targeted at developers, you need a better pitch. I get "we did this 100x faster" but the obvious followup question is "how" but then the answer seems to be a ton of flow diagrams with way too many nodes that tell me approximately nothing and some handwaving about something called P-States that are basically defined to…

> Whereas Twitter stores home timelines in a dedicated in-memory database, in Rama they’re stored in-memory in the same processes executing the ETL for timeline fanout. So instead of having to do network operations, serialization, and deserialization, the reads and writes to home timelines in our implementation are literally just in-memory operations on a hash map. This is dramatically simpler and more efficient than operating a separate in-memory database. The timelines themselves are stored like this:

> To minimize memory usage and GC pressure, we use a ring buffer and Java primitives to represent each home timeline. The buffer contains pairs of author ID and status ID. The author ID is stored along with the status ID since it is static information that will never change, and materializing it means that information doesn’t need to be looked up at query time. The home timeline stores the most recent 600 statuses, so the buffer size is 1,200 to accommodate each author ID and status ID pair. The size is fixed since storing full timelines would require a prohibitive amount of memory (the number of statuses times the average number of followers).

> Each user utilizes about 10kb of memory to represent their home timeline. For a Twitter-scale deployment of 500M users, that requires about 4.7TB of memory total around the cluster, which is easily achievable.

Isn't this where the most difficult(expensive) part is and Rama has little to do with it? It appears that the other parts also do not have to be Rama.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#252

Post by nathanmarz »

endisneigh wrote: post #215
sdwr wrote: post #207

Earlier quoted context omitted.

In a typical architecture, the DB stores data, and the backend calls the DB to make updates and compile views. Here, the "views" are defined formally (the P-states), and incrementally, automatically updated when the underlying data changes. Example problem: Get a list of accounts that follow account 1306 "Classic architecture": - Naive approach. Search through all accounts follow lists for "1306". Super slow, scales…

I read their post and honestly it’s not really that much different than just materialized views in a regular database plus async jobs to do the long running tasks. It’s a ridiculous amount of fluff to describe that. Not to mention it’s proprietary and only supports the JVM and doesn’t integrate with the tons of tooling designed about RDBMS unless you stream everything to them, defeating the purpose. What really irks…

We are very open in the post that the core concepts are not new:

  Individually, none of these concepts are new. I’m sure you’ve seen them all before. You may be tempted to dismiss Rama’s programming model as just a combination of event sourcing and materialized views. But what Rama does is integrate and generalize these concepts to such an extent that you can build entire backends end-to-end without any of the impedance mismatches or complexity that characterize and overwhelm existing systems.
Indexes as arbitrary data structures that you shape to perfectly meet your use cases, a powerful computation API that's like a "distributed programming language", and everything being so integrated make a world of difference.

I understand the desire to see all the code, and that's coming in two weeks. That said, the code in the post isn't trivial as it's showing almost the complete implementations of two major parts of Mastodon: the social graph and timeline fanout.

Next week you'll be able to play with Rama when we release a build of it, and the documentation will help with that.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#253

Post by nathanmarz »

falsandtru wrote: post #251
dataangel wrote: post #192

I do C++ backend work in a non-web industry and this entire post is Greek to me. Even though this is targeted at developers, you need a better pitch. I get "we did this 100x faster" but the obvious followup question is "how" but then the answer seems to be a ton of flow diagrams with way too many nodes that tell me approximately nothing and some handwaving about something called P-States that are basically defined to…

> Whereas Twitter stores home timelines in a dedicated in-memory database, in Rama they’re stored in-memory in the same processes executing the ETL for timeline fanout. So instead of having to do network operations, serialization, and deserialization, the reads and writes to home timelines in our implementation are literally just in-memory operations on a hash map. This is dramatically simpler and more efficient than…

We're storing those in-memory within the Rama modules materializing the home timelines. And the query topologies that refresh home timelines for lost partitions is colocated with that. This is dramatically simpler than operating a separate in-memory database, and Rama has everything to do with that.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#254

Post by legi0nary »

nathanmarz wrote: post #120
NoraCodes wrote: post #2

I would argue that this is not "a Mastodon instance", since it is not running Mastodon - other than that, very very neat work! I'm excited for that "Source Code" link to be live :)

We call it a "Mastodon instance" because we implemented the entire Mastodon API ( https://docs.joinmastodon.org/api/ ). This is in addition to also implementing the ActivityPub API which Mastodon also implements for federation.

If you can do this with Bluesky once it federates we might be able to get away from twitter for good.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#255

Post by falsandtru »

nathanmarz wrote: post #253
falsandtru wrote: post #251

Earlier quoted context omitted.

> Whereas Twitter stores home timelines in a dedicated in-memory database, in Rama they’re stored in-memory in the same processes executing the ETL for timeline fanout. So instead of having to do network operations, serialization, and deserialization, the reads and writes to home timelines in our implementation are literally just in-memory operations on a hash map. This is dramatically simpler and more efficient than…

We're storing those in-memory within the Rama modules materializing the home timelines. And the query topologies that refresh home timelines for lost partitions is colocated with that. This is dramatically simpler than operating a separate in-memory database, and Rama has everything to do with that.

It appears simpler and better without Rama.

> So instead of having to do network operations, serialization, and deserialization, the reads and writes to home timelines in our implementation are literally just in-memory operations on a hash map. This is dramatically simpler and more efficient than operating a separate in-memory database.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#256

Post by datavirtue »

nathanmarz wrote: post #176
chubot wrote: post #151

Earlier quoted context omitted.

Yeah definitely, these ideas always sound very appealing to me, in theory -- I almost wonder why nobody has built it before e.g. they mention "event sourcing" and "materialized views" in the post -- sounds good But I thought I heard from a few people who were like "we ripped event sourcing" out of our codebase and so forth And yeah your question is an obvious good one, and the Reddit answer of "write through cache" .…

When you step back and consider the incredible amount of manpower and resources that have been put into these applications, it's amazing how buggy these applications are. To put it simply, they're buggy because the underlying infrastructure and techniques used to build them are so complex that the implementation is beyond the realm of human understanding. The way applications are built, and have been built since befo…

You should try using Facebook marketplace. It is so rickety. I have to get on a desktop to use it at all.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#257

Post by datavirtue »

nathanmarz wrote: post #238
chubot wrote: post #193

Earlier quoted context omitted.

Yes, I 100% agree with you. I would like something like this to succeed, and agree the problem is real. But what are the tradeoffs? There's nothing that comes with 100x benefit with no tradeoffs (side note: I worked on Google Code for a short while in 2008, concurrent with Github's founding ... I think Github moved a lot faster in a large part because they weren't dealing with distributed systems at first -- they had…

Rama is a much broader platform than a database, so the consistency semantics you get depend on how you use it. When using Rama, you're not mutating indexes directly like you do with a database, but adding source data that then gets materialized into any number of indexes. You get read-after-write consistency for any PStates in a streaming ETL colocated with the depot you appended to. This is if you do the depot appe…

I would like your newsletter.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#259

Post by dpweb »

nathanmarz wrote: post #168
RomanPushkin wrote: post #163

> ...10k lines of code. This is 100x less code than the ~1M lines Twitter I wish I didn't see this comparison, which is not fair at all. Everyone in their right mind understands that the number of features is much less, that's why you have 10k lines. Add large-scale distributed live video support at the top of that, and you won't get any close to 10k lines. It's only one of many many examples. I really wish you compa…

We're comparing just to the original consumer product, which is about the same as Mastodon is today. That's why we said "original consumer product" and not "Twitter's current consumer product". Mastodon actually has more features than the original Twitter consumer product like hashtag follows, global timelines, and more sophisticated filtering/muting capabilities. Some people argue it's not so expensive to build a sc…

Built some log databases and back end frameworks myself with some of the same concepts. I applaud the creativity in rethinking how back ends should work. Please now do frontends next! :)

"But it's not a fully functioning 2023 Twitter!!!" I think some people miss the point. This is not about hey we built a Twitter clone. This is about a POC for a novel app architecture.

We need to be constantly examining and re-examining our thoughts about the best way to deal with distributed systems, scale, developer workflow. Even inventing new ones.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#260

Post by dools »

nathanmarz wrote: post #120
NoraCodes wrote: post #2

I would argue that this is not "a Mastodon instance", since it is not running Mastodon - other than that, very very neat work! I'm excited for that "Source Code" link to be live :)

We call it a "Mastodon instance" because we implemented the entire Mastodon API ( https://docs.joinmastodon.org/api/ ). This is in addition to also implementing the ActivityPub API which Mastodon also implements for federation.

"Originally, Twitter was one, monolithic application built with Ruby on Rails. But now, it's divided into about two hundred self-contained services that talk to each other. Each runs atop the JVM, with most written in Scala and some in Java and Clojure"[1]

So is Twitter not a Twitter instance? Like if it looks, walks and toots like a Mastodon, is it not a Mastodon instance?

[1] https://www.wired.com/2013/09/the-second-coming-of-java/

Post reply on HN