Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced the cost of building Mastodon at Twitter-scale by 100x

blog.redplanetlabs.com

Post reply on HN
Print view View original

151–160 of 376 posts

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#151

Post by chubot »

Pxtl wrote: post #25

I've seen many people describe frameworks like this - you know, first you have the slow back-end event-driven master database that you don't query live against, then you've got eventual-consistency flows against the various data-warehouses and data-stores and partitioned sharded databases in useful query-friendly layouts that you actually read live from... and I never see it clearly explained: how do you read a chang…

Yeah definitely, these ideas always sound very appealing to me, in theory -- I almost wonder why nobody has built it before

e.g. they mention "event sourcing" and "materialized views" in the post -- sounds good

But I thought I heard from a few people who were like "we ripped event sourcing" out of our codebase and so forth

And yeah your question is an obvious good one, and the Reddit answer of "write through cache" ... is less than satisfying to me

I FREQUENTLY have the problem where I reload the page and Reddit shows me stale data. It's SUPER buggy.

---

Anyway I definitely look forward to hearing people try this and what their longer term impressions are !

I basically want to know what the tradeoffs are -- it sounds good, but there are always tradeoffs

So is the tradeoff "eventual consistency" ? What are the other tradeoffs?

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#152

Post by theptip »

jitl wrote: post #146

This architecture seems very similar to existing offerings in the "in-memory data grid" category, like Apache Ignite and Hazelcast. I'm more familiar with Ignite (I built a toy Notion backend with it over a few afternoons in 2020). The way Ignite works overall is similar. You make a cluster of JVM processes, your data partitioned and replicated across the cluster, and you upload some JARs of business logic to the clu…

Hazelcast has been on my list to explore for a while. Anyone have pointers to a good sample project / deep-dive in the same sort of spirit as the OP here?

Also would love to hear folks’ thoughts on the sort of usecase where this data grid excels.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#153

Post by 3cats-in-a-coat »

nathanmarz wrote: post #143
3cats-in-a-coat wrote: post #129

Earlier quoted context omitted.

Thanks, I will read more soon! I'm curious... how do you resolve the "impedance mismatch" between some "canonical" models that business decisions are made, based upon, which need to be synchronous with the depots (and mutually synchronous with other models sharing fragments of the same data), and the eventually consistent read models, which have a more lax constraint on how up to date they are? How do you ensure cons…

The impedance mismatches you're used to from using databases are gone because: - You can finely tune your indexes to be exactly the optimal shape for your application (data structure). You can see this in our Mastodon implementation with the big variety of data structures we used for all the use cases. - You're generally just using regular Java objects everywhere: appending to depots, during ETL processing, and store…

Thanks! So we can see these ACKs as "wait and synchronize" signals I suppose? However how can we ensure an "all or nothing" between all parties trying to ACK a conditions they're mutually dependent on? I.e. transactionality or atomicity?

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#154

Post by newaccount74 »

hot_gril wrote: post #79
ceejayoz wrote: post #73

Earlier quoted context omitted.

> Don't even want to wait for an HTTP roundtrip for some of these, e.g. "liking" a post should fill in the heart icon or whatever instantly. HN does this, and on slow days, about half of my upvotes don't go through.

Messaging apps often have a checkmark to indicate the message actually went to the server, and maybe another checkmark to indicate it was received on the other end. Maybe HN needs an icon indicating that your vote went through.

Make the arrows grey to indicate the click registered, make them disappear to indicate the server successfully registered the vote?

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#155

Post by CharlesW »

nathanmarz wrote: post #120
NoraCodes wrote: post #2

I would argue that this is not "a Mastodon instance", since it is not running Mastodon - other than that, very very neat work! I'm excited for that "Source Code" link to be live :)

We call it a "Mastodon instance" because we implemented the entire Mastodon API ( https://docs.joinmastodon.org/api/ ). This is in addition to also implementing the ActivityPub API which Mastodon also implements for federation.

> We call it a "Mastodon instance" because we implemented the entire Mastodon API…

Except "Mastodon instance" means an instance of Mastodon, which is open source. Whether or not it was intended to be deceptive (I'd think a group of smart people would know better), this personally left a bad taste in my mouth.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#156

Post by nathanmarz »

jitl wrote: post #150
jedberg wrote: post #55

Earlier quoted context omitted.

The short answer is write-through cache. You write the update directly to the cache closest to the user and into the eventually consistent queue. We did this at reddit. When you make a comment the HTML is rendered and put straight into the cache, and the raw text is put into the queue to go into the database. Same with votes. I suspect they do this client side now, which is now the closest cache to the user, but back…

Rama should bundle a write-through cache! Another in-memory JVM cluster thingamabob (Apache Ignite) used to propose write-through caching as it's primary selling point: https://ignite.apache.org/use-cases/in-memory-cache.html#:~:... . Or, maybe their pitch is that the streaming bits are so fast, you can just await the downstream commit of some write to a depot and it'll be as fast as a normal SQL UPDATE.

Rama is extremely fast, as you can see for yourself by playing with our Mastodon instance.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#157

Post by chubot »

chubot wrote: post #151
Pxtl wrote: post #25

I've seen many people describe frameworks like this - you know, first you have the slow back-end event-driven master database that you don't query live against, then you've got eventual-consistency flows against the various data-warehouses and data-stores and partitioned sharded databases in useful query-friendly layouts that you actually read live from... and I never see it clearly explained: how do you read a chang…

Yeah definitely, these ideas always sound very appealing to me, in theory -- I almost wonder why nobody has built it before e.g. they mention "event sourcing" and "materialized views" in the post -- sounds good But I thought I heard from a few people who were like "we ripped event sourcing" out of our codebase and so forth And yeah your question is an obvious good one, and the Reddit answer of "write through cache" .…

Hilariously, I went to edit the above comment, and HN was overloaded. Then it served me three or four 500's, AND it served me stale data in between

I was pissed off that I would have to type my comment again, but actually it did save it, and refreshing worked.

From what I understand Hacker News is architected more in-memory, on one big box ... Perhaps similar to the event sourcing model

(not knocking hacker news -- it's generally a very fast site, MUCH better than Reddit. Just that scaling beyond a single machine is difficult and full of gotchas )

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#158

Post by wizofaus »

ceejayoz wrote: post #73
hot_gril wrote: post #57

Earlier quoted context omitted.

This is usually what I do. Don't even want to wait for an HTTP roundtrip for some of these, e.g. "liking" a post should fill in the heart icon or whatever instantly. One famous example of this going to far: Mac Mail app used to play a whoosh sound when your email is actually sent. They changed it to whoosh instantly no matter what. Given how often an email might fail to send or get delayed, this meant an actually use…

> Don't even want to wait for an HTTP roundtrip for some of these, e.g. "liking" a post should fill in the heart icon or whatever instantly. HN does this, and on slow days, about half of my upvotes don't go through.

You actually check your list of upvoted comments?

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#159

Post by nathanmarz »

sitzkrieg wrote: post #137
nathanmarz wrote: post #13

Earlier quoted context omitted.

That is indeed an intentional part of Mastodon's design, which we tried to be faithful to as much as possible. We originally implemented search across all statuses and had to reimplement it when we realized Mastodon is a little different.

did you ever consider starting from something already technically performant like pleroma or misskey?

Well, we didn't start from anything as we implemented this completely from scratch. I believe Mastodon is much more widely used than those so it seemed like a better target for this.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#160

Post by jedberg »

nathanmarz wrote: post #156
jitl wrote: post #150

Earlier quoted context omitted.

Rama should bundle a write-through cache! Another in-memory JVM cluster thingamabob (Apache Ignite) used to propose write-through caching as it's primary selling point: https://ignite.apache.org/use-cases/in-memory-cache.html#:~:... . Or, maybe their pitch is that the streaming bits are so fast, you can just await the downstream commit of some write to a depot and it'll be as fast as a normal SQL UPDATE.

Rama is extremely fast, as you can see for yourself by playing with our Mastodon instance.

It’s fast until it’s not. Making a post and then hitting reload and not seeing it can be very jarring for the user. Definitely something to think about.
Post reply on HN