Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced the cost of building Mastodon at Twitter-scale by 100x

blog.redplanetlabs.com

Post reply on HN
Print view View original

171–180 of 376 posts

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#171

Post by nathanmarz »

jedberg wrote: post #160
nathanmarz wrote: post #156

Earlier quoted context omitted.

Rama is extremely fast, as you can see for yourself by playing with our Mastodon instance.

It’s fast until it’s not. Making a post and then hitting reload and not seeing it can be very jarring for the user. Definitely something to think about.

What do you mean? Every post I do shows up instantly.

Reloading the page from scratch can be slow due to Soapbox doing a lot of stuff asynchronously from scratch (Soapbox is the open-source Mastodon interface that we're using to serve the frontend). https://soapbox.pub/

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#172

Post by nathanmarz »

jitl wrote: post #161
nathanmarz wrote: post #156

Earlier quoted context omitted.

Rama is extremely fast, as you can see for yourself by playing with our Mastodon instance.

is there a way to guarantee reading your own writes from a client perspective?

Yes. Depot appends by default don't return success until colocated streaming topologies have completed processing the data. So this is one way to coordinate the frontend with changes on the backend.

Within an ETL, when the computations you do on PStates are colocated with them, you always read your own writes.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#173

Post by squeaky-clean »

jedberg wrote: post #55
Pxtl wrote: post #25

I've seen many people describe frameworks like this - you know, first you have the slow back-end event-driven master database that you don't query live against, then you've got eventual-consistency flows against the various data-warehouses and data-stores and partitioned sharded databases in useful query-friendly layouts that you actually read live from... and I never see it clearly explained: how do you read a chang…

The short answer is write-through cache. You write the update directly to the cache closest to the user and into the eventually consistent queue. We did this at reddit. When you make a comment the HTML is rendered and put straight into the cache, and the raw text is put into the queue to go into the database. Same with votes. I suspect they do this client side now, which is now the closest cache to the user, but back…

In Nathan Marz's (the article author) book, Big Data, he describes this and calls it the Speed Layer. I haven't fully finished the article yet, but the components it's describing seem to be equivalent to what he calls the Batch Layer and the Serving Layer in his book.

But I'm kind of getting the impression this works without any speed layer and is expected to be fast enough as-is.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#174

Post by [deleted] »

nathanmarz wrote: post #159
sitzkrieg wrote: post #137

Earlier quoted context omitted.

did you ever consider starting from something already technically performant like pleroma or misskey?

Well, we didn't start from anything as we implemented this completely from scratch. I believe Mastodon is much more widely used than those so it seemed like a better target for this.

[deleted]

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#175

Post by squeaky-clean »

nathanmarz wrote: post #171
jedberg wrote: post #160

Earlier quoted context omitted.

It’s fast until it’s not. Making a post and then hitting reload and not seeing it can be very jarring for the user. Definitely something to think about.

What do you mean? Every post I do shows up instantly. Reloading the page from scratch can be slow due to Soapbox doing a lot of stuff asynchronously from scratch (Soapbox is the open-source Mastodon interface that we're using to serve the frontend). https://soapbox.pub/

I think the concern is will this still be true if Mastodon reaches Twitter scale?

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#176

Post by nathanmarz »

chubot wrote: post #151
Pxtl wrote: post #25

I've seen many people describe frameworks like this - you know, first you have the slow back-end event-driven master database that you don't query live against, then you've got eventual-consistency flows against the various data-warehouses and data-stores and partitioned sharded databases in useful query-friendly layouts that you actually read live from... and I never see it clearly explained: how do you read a chang…

Yeah definitely, these ideas always sound very appealing to me, in theory -- I almost wonder why nobody has built it before e.g. they mention "event sourcing" and "materialized views" in the post -- sounds good But I thought I heard from a few people who were like "we ripped event sourcing" out of our codebase and so forth And yeah your question is an obvious good one, and the Reddit answer of "write through cache" .…

When you step back and consider the incredible amount of manpower and resources that have been put into these applications, it's amazing how buggy these applications are. To put it simply, they're buggy because the underlying infrastructure and techniques used to build them are so complex that the implementation is beyond the realm of human understanding.

The way applications are built, and have been built since before I was born, is by combining together potentially dozens of narrow tools together: databases, computation systems, caches, monitoring tools, etc. There has never been a cohesive model capable of expressing arbitrary backends end-to-end, and every application built has to be twisted to fit onto the existing narrow pieces.

Rama is a lot more than just "event sourcing" and "materialized views". Those are two concepts at its foundation, but the real breakthrough is being that cohesive model capable of expressing diverse backends in their entirety. It took me more than five years of dedicated research to discover this model, and it was extremely difficult.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#178

Post by ceejayoz »

wizofaus wrote: post #158
ceejayoz wrote: post #73

Earlier quoted context omitted.

> Don't even want to wait for an HTTP roundtrip for some of these, e.g. "liking" a post should fill in the heart icon or whatever instantly. HN does this, and on slow days, about half of my upvotes don't go through.

You actually check your list of upvoted comments?

No, I just notice it when I come back to the thread later in the day and a bunch of comments I know I upvoted are back to normal.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#179

Post by 5636588 »

ltr_ wrote: post #169

i always had this question: how realistically is to, having an standard spec and interoperable protocols, for toxic apps of big international tech companies that provides """services""", so instances of implementations can be maintained by municipalities or local tech business and talent with 100x less employees and money? what policies should be in place to achieve that? what would be the challenges? it would be bet…

The EU already has regulations in place regarding open banking with its Payment Services Directive. I'd imagine a similar framework could be applied to big social tech.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#180

Post by colonwqbang »

polishdude20 wrote: post #61

"We spent nine person-months building our scalable Mastodon instance. " Nono, you can't say that when later on you say it's built on top of Rama. You literally spent 10 years building the framework to even make this. And yes, you built this in 10k lines of code but how many lines of code is Rama? This seems disingenuous.

The JVM took years to write. It took decades to develop the technology necessary to build a modern microcomputer. Before that, millennia to invent written language. And now that those platforms (including Rama) all exist, one can deliver a Mastodon server on top of them in about 9 man-months.
Post reply on HN