Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced the cost of building Mastodon at Twitter-scale by 100x

blog.redplanetlabs.com

Post reply on HN
Print view View original

31–40 of 376 posts

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#31

Post by Kiro »

dang wrote: post #20
ceejayoz wrote: post #5

Headline: "building Twitter at Twitter-scale" Article: "building Mastodon at sub-Twitter-scale"

Thank - I've changed the title to be consistent with what the article says.

That's not the title of the article and also not what the article says. I would be really pissed if you editorialized the title of my article like that.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#32

Post by ceejayoz »

nathanmarz wrote: post #21
ceejayoz wrote: post #5

Headline: "building Twitter at Twitter-scale" Article: "building Mastodon at sub-Twitter-scale"

Actually if you read the article you can see we tested way above Twitter-scale. We can easily run this instance at full Twitter-scale by just paying for more servers. The point isn't the Mastodon instance, but rather that Rama enabled us to build it at scale with in a tiny amount of code and time.

Mastodon and Twitter don't do the same amount of work per post. Mastodon doesn't have a recommendation engine, they don't have an advertising engine, they don't scan every post for CSAM, there's no global search, etc. (Some of these things are good not to have, but they still drastically change the scope.)

Claiming to have enabled significant scaling of a Mastodon/ActivityPub-compatible instance is fine. Claiming to have replicated Twitter on the cheap is, from the post, not accurate.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#33

Post by nathanmarz »

trollied wrote: post #16

> We spent nine person-months building our scalable Mastodon instance. + the time spent creating Rama, the platform that enables it. Very dishonest leaving that out.

You're missing the point. Rama is a generic platform that provides a new baseline for how expensive it is to build applications at scale. There's nothing about Rama specific to social networks. What we're showing is that Rama creates a new era in software engineering where the cost of building applications at scale is radically reduced. With Rama, anyone embarking on a new application today has a radically different economic outlook for the end-to-end cost of developing that application from prototype through large scale.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#34

Post by phillipcarter »

In the year of our lord 2023, people are still launching immature products with "we built a clone of a tiny subset of Twitter" as their use case? Come on. Twitter is huge because they have to support a huge number of use cases. Using this proprietary framework won't magically make complex use cases go away.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#36

Post by Zak »

trollied wrote: post #16

> We spent nine person-months building our scalable Mastodon instance. + the time spent creating Rama, the platform that enables it. Very dishonest leaving that out.

Not from the perspective of this being a demo application to sell Rama. The pitch is that if you use Rama, you can achieve similar results.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#37

Post by nathanmarz »

_dwt wrote: post #14

Hmmm, "Rama is programmed entirely with a Java API – no custom languages or DSLs" according to the landing page, but this sure looks like an embedded DSL for dataflow graphs to me - Expr and Ops everywhere. Odd angle to take.

I consider "DSL" as something that's its own language with it's own lexer and parser, like SQL. The Rama API is just Java – classes, interfaces, methods, etc. Everything you do in Rama, from defining indexes, performing queries, or writing dataflow ETLs, is done in Java.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#38

Post by dang »

messe wrote: post #26
dang wrote: post #20

Earlier quoted context omitted.

Thank - I've changed the title to be consistent with what the article says.

Minor gripe, but there's a misspelling in the title: it should read Mastodon not Mastadon.

Oops! Fixed now. Thanks to you, Fabricio20, and riffic.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#39

Post by newZWhoDis »

LeifCarrotson wrote: post #12

> How is it possible that we’ve reduced the cost of building scalable applications by multiple orders of magnitude? > You can begin to understand this by starting with a simple observation: you can describe Mastodon (or Twitter, Reddit, Slack, Gmail, Uber, etc.) in total detail in a matter of hours. It has profiles, follows, timelines, statuses, replies, boosts, hashtags, search, follow suggestions, and so on. It doe…

So the things that make it difficult are all things you shouldn’t be doing in the first place? Well that certainly helps.

You shouldn’t be handling PII/raw CC’s anyways (assuming FinTech is not your core business)

Secretly scanning your customers private messages against an illegal and immoral hash table from a pseudo-government entity? Are you law enforcement? No? Then fucking stop.

Copyright claims? Fuck ‘em. Only do what you are absolutely, positively, no way-out legally bound to do. No more no less. Require formal, written requests and comply in the maximum amount of time allowed.

Audits? What kind of audit? If they’re non-financial you’re probably doing something wrong.

Corporate squares have ruined the tech scene, and it’s time to resist.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#40

Post by jokethrowaway »

Pxtl wrote: post #25

I've seen many people describe frameworks like this - you know, first you have the slow back-end event-driven master database that you don't query live against, then you've got eventual-consistency flows against the various data-warehouses and data-stores and partitioned sharded databases in useful query-friendly layouts that you actually read live from... and I never see it clearly explained: how do you read a chang…

You can hack it and optimistically render the data you know about because your client created it - on the frontend, at no additional cost.
Post reply on HN