Live data from Hacker News

We reduced the cost of building Mastodon at Twitter-scale by 100x

blog.redplanetlabs.com

311–320 of 376 posts

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#311

HN seems to be putting you through the wringer, I for one am excited you guys made this and plan to open source it- it looks like a fantastic project.

Additionally to the “why not” sibling comment, I suspect nathanmarz can take it :). (Early Apache storm user here.)

It takes strong conviction to work on something like this for a decade.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#312
post #25

I've seen many people describe frameworks like this - you know, first you have the slow back-end event-driven master database that you don't query live against, then you've got eventual-consistency flows against the various data-warehouses and data-stores and partitioned sharded databases in useful query-friendly layouts that you actually read live from... and I never see it clearly explained: how do you read a chang…

It's called "read-after-write consistency". Write-through cache is one way, or just use a strongly consistent database, haha.

https://avikdas.com/2020/04/13/scalability-concepts-read-aft...

https://en.wikipedia.org/wiki/Cache_%28computing%29#Writing_...

https://cloud.google.com/blog/products/databases/why-you-sho...

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#313

Earlier quoted context omitted.

Yes, it's 100% written in Java.

This looks interesting and a nice weekend read but just skimming through, why is spring needed?

Nevermind (it has to be paired with some web server/framework).

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#314
I appreciate the inversion/melding of the data model and compute. Im curious to know your perspective on two parts: How would multitenancy fit in to rama? Using your mastodon example, providing “hosted mastodon instances as a service”, where you _also_ allow for data governance, per customer encryption at rest, user IDP support, etc. Is it multiple single tenant rama deployments, running independent customers? Multitenant rama clusters with shared depos and each pstate includes “tenant id”? Something else?

Second whats the product/business angle on customer confidence, technical novelty, and your business core competency? A dated example but Im thinking of somewhere like basho with riak. Super cool tool, takes some mental adjustment to “get”, challlenges selling hosting vs software vs pro services.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#315

> ...10k lines of code. This is 100x less code than the ~1M lines Twitter I wish I didn't see this comparison, which is not fair at all. Everyone in their right mind understands that the number of features is much less, that's why you have 10k lines. Add large-scale distributed live video support at the top of that, and you won't get any close to 10k lines. It's only one of many many examples. I really wish you compa…

The 100x less code always reminds me of a hypothetical Half Life 2 game engine. Then your code would be:

   StartHalfLife2LikeGame()
and you'd replace millions LOC with one line. If there is a perfect match between the framework and your app, there is no code. The more your app diverges from the ideal app the framework was written for, the more code you have.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#316
post #157
post #151

Earlier quoted context omitted.

Yeah definitely, these ideas always sound very appealing to me, in theory -- I almost wonder why nobody has built it before e.g. they mention "event sourcing" and "materialized views" in the post -- sounds good But I thought I heard from a few people who were like "we ripped event sourcing" out of our codebase and so forth And yeah your question is an obvious good one, and the Reddit answer of "write through cache" .…

Hilariously, I went to edit the above comment, and HN was overloaded. Then it served me three or four 500's, AND it served me stale data in between I was pissed off that I would have to type my comment again, but actually it did save it, and refreshing worked. From what I understand Hacker News is architected more in-memory, on one big box ... Perhaps similar to the event sourcing model (not knocking hacker news -- i…

StackOverflow used to run on a single (very beefy) machine also for a long time — databases make efficient use of vertical scaling, horizontal scaling is much harder.

Of course, specialist systems can often do much better.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#317

"We spent nine person-months building our scalable Mastodon instance. " Nono, you can't say that when later on you say it's built on top of Rama. You literally spent 10 years building the framework to even make this. And yes, you built this in 10k lines of code but how many lines of code is Rama? This seems disingenuous.

So if one was building a database, and then later building an app against their own database, they should not get to claim how much time the latter took alone? There are enough hyped bullshit in the world to call out, but I would not be so quick to dismiss this one, especially given that the author does know what he talks about (he used to work at twitter)

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#318
post #283
post #85

Earlier quoted context omitted.

Any trademark case is going to have to prove that a reasonable person would think this article is from Mastodon gGmbH, or is talking about their product "Mastodon". The top of the page reads "Red Planet Labs" , the title of the article is "How we reduced the cost of building Twitter at Twitter-scale by 100x" and the first line of the article is "We built a Twitter-scale Mastodon instance from scratch in only 10k line…

there's this though: https://mastodon.redplanetlabs.com/timeline/local and this from the gGmbH trademark policy: > You may not use the Mastodon word mark, or any similar mark, in your domain name, unless you have written permission from Mastodon gGmbH.

Trademark law doesn’t work like that. You don’t get to license your trademark in the same way you get to license copyrighted works.

Otherwise, people could just say you can’t use their trademark in any document that says something negative about them, and then successfully sue the press and angry customers for complaining about them.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#319

ctrl+f "ads" ctrl+f "monetization" ctrl+f "moderation" ctrl+f "existing infrastructure" ctrl+f "personalization" etc etc Yeah about what I expect from a "we rebuilt twitter for cheap" post. There's no point to the comparisons with the Twitter codebase size/cost. It completely distracts from what is probably a perfectly fine project.

The point is that much of that should be unnecessary to sustain the service because hosting costs are significantly (presumably 100x) cheaper.

They never claimed that hosting costs would be 100x cheaper, and given what they show with amount of machines in the scaling chart, I doubt it is.

It's a JVM application with all state duplicated N times, so at least on the memory side it's likely going to be a resource hog.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#320
As someone who has worked in both startups and Enterprise IT for over 30 years (including large Java based systems) I see a use case for Rama in large companies who have a lot of difficulty glueing many different systems together to achieve scalability. So I think that Red Planet Labs could get several contracts in the 100k USD and over range in large Enterprises. This is for enterprises who have the problem of integrating many systems to achieve scalability and who are already large Java shops.

However, I do not see Rama's initial market being startups, since they just want the simplest way possible to build UI + backend and want to iterate super fast with tech that their developers already know in the initial stages.

Post reply on HN