Live data from Hacker News

We reduced the cost of building Mastodon at Twitter-scale by 100x

blog.redplanetlabs.com

21–30 of 376 posts

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#21
post #5

Headline: "building Twitter at Twitter-scale" Article: "building Mastodon at sub-Twitter-scale"

Actually if you read the article you can see we tested way above Twitter-scale. We can easily run this instance at full Twitter-scale by just paying for more servers.

The point isn't the Mastodon instance, but rather that Rama enabled us to build it at scale with in a tiny amount of code and time.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#22

> How is it possible that we’ve reduced the cost of building scalable applications by multiple orders of magnitude? > You can begin to understand this by starting with a simple observation: you can describe Mastodon (or Twitter, Reddit, Slack, Gmail, Uber, etc.) in total detail in a matter of hours. It has profiles, follows, timelines, statuses, replies, boosts, hashtags, search, follow suggestions, and so on. It doe…

~~full text search doesn't appear to work... so it's possible they punted on one of the harder parts, which is fast efficient accurate fuzzy search, which moderation and a lot of those other harder things rely on.~~

eta: they say that had it but removed it because apparently it's not something mastodon supports. so I guess it is a pretty good high level implementation.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#23
What one finds useful from a web application and what the web application actually is are usually two entirely different things.

I work in marketing automation, and I guess I have in one way or another my entire career. The clients who need to use the platform to communicate with their own clients over social networking may never touch our print delivery system, but that doesn't mean that print delivery doesn't exist or isn't important.

If you are unwilling to recreate the totality of the application in terms of functionality, then you are lying if you say that you have recreated it.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#24

They deserve congrats for that since they built the load test to prove this Of course, for actual production use, there's probably a lot of things still, but this is a very nice works nonetheless

I wouldn't call our instance a load test, as it's a legitimate instance available for anyone to use. It's very much production-grade.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#25
I've seen many people describe frameworks like this - you know, first you have the slow back-end event-driven master database that you don't query live against, then you've got eventual-consistency flows against the various data-warehouses and data-stores and partitioned sharded databases in useful query-friendly layouts that you actually read live from... and I never see it clearly explained: how do you read a change back to the user literally just after they made the change? How do you say "other views eventual-consistency is fine but for this view of this bit of info we need it updated now".

This write-up is very detailed but I couldn't find that explanation.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#26
post #20
post #5

Headline: "building Twitter at Twitter-scale" Article: "building Mastodon at sub-Twitter-scale"

Thank - I've changed the title to be consistent with what the article says.

Minor gripe, but there's a misspelling in the title: it should read Mastodon not Mastadon.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#27

They deserve congrats for that since they built the load test to prove this Of course, for actual production use, there's probably a lot of things still, but this is a very nice works nonetheless

I wouldn't call our instance a load test, as it's a legitimate instance available for anyone to use. It's very much production-grade.

This is what I'm calling load test:

> The instance has 100M bots posting 3,500 times per second at 403 average fanout to demonstrate its scale.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#28
post #20
post #5

Headline: "building Twitter at Twitter-scale" Article: "building Mastodon at sub-Twitter-scale"

Thank - I've changed the title to be consistent with what the article says.

Hey dang, I think you have a typo in the title.. says Mastadon instead of Mastodon!

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#30

> How is it possible that we’ve reduced the cost of building scalable applications by multiple orders of magnitude? > You can begin to understand this by starting with a simple observation: you can describe Mastodon (or Twitter, Reddit, Slack, Gmail, Uber, etc.) in total detail in a matter of hours. It has profiles, follows, timelines, statuses, replies, boosts, hashtags, search, follow suggestions, and so on. It doe…

> I'm honestly shocked that the technical implementation of a simplified, coherent platform took a full 9 person-months.

To be fair they developed this whole new platform to build this app with. I guess that's where the effort went.

Post reply on HN