Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced the cost of building Mastodon at Twitter-scale by 100x

blog.redplanetlabs.com

Post reply on HN
Print view View original

241–250 of 376 posts

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#241

Post by doctorpangloss »

nathanmarz wrote: post #168
RomanPushkin wrote: post #163

> ...10k lines of code. This is 100x less code than the ~1M lines Twitter I wish I didn't see this comparison, which is not fair at all. Everyone in their right mind understands that the number of features is much less, that's why you have 10k lines. Add large-scale distributed live video support at the top of that, and you won't get any close to 10k lines. It's only one of many many examples. I really wish you compa…

We're comparing just to the original consumer product, which is about the same as Mastodon is today. That's why we said "original consumer product" and not "Twitter's current consumer product". Mastodon actually has more features than the original Twitter consumer product like hashtag follows, global timelines, and more sophisticated filtering/muting capabilities. Some people argue it's not so expensive to build a sc…

I work in gaming, so I cannot speak to your specific experiences. Entity Component Systems are extremely performant, really good science, and shipping in middlewares like Unity. However, in order to ship an ECS game, in my experience, you have to have already made your whole game first in a normal approach, in order to have everything be fully specified sufficiently that you can correctly create an ECS implementation. In practice, this means ECS is used to make first person shooters, which have decades of well specified traditions and behavior, and V2 of simulators, like for Cities Skylines 2 and Two Point Campus.

So this is not meant to diminish the achievements of what you have built at all, it is more intellectually honest to say that "any high performance framework is most suitable for projects that are exact clones of pre-existing, mature things with battle-hardened specifications and end user behavior." While this might cover some greenfield projects, including the best capitalized ones that may matter to you, it does diminish the appeal of a framework for the vast majority of success stories from small & poorly capitalized teams. Those small & poor teams are very innovation and serendipity driven and hence rarely copying a pre-existing thing. And even if they try to become well-capitalized, they are almost always doing so by having worked on the thing they are copying already (i.e., already shipping version 1.0 for years).

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#242

Post by deltree7 »

nathanmarz wrote: post #168
RomanPushkin wrote: post #163

> ...10k lines of code. This is 100x less code than the ~1M lines Twitter I wish I didn't see this comparison, which is not fair at all. Everyone in their right mind understands that the number of features is much less, that's why you have 10k lines. Add large-scale distributed live video support at the top of that, and you won't get any close to 10k lines. It's only one of many many examples. I really wish you compa…

We're comparing just to the original consumer product, which is about the same as Mastodon is today. That's why we said "original consumer product" and not "Twitter's current consumer product". Mastodon actually has more features than the original Twitter consumer product like hashtag follows, global timelines, and more sophisticated filtering/muting capabilities. Some people argue it's not so expensive to build a sc…

[dead]

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#243

Post by kyle-rb »

nathanmarz wrote: post #212
kyle-rb wrote: post #165

Kinda disappointed by the simulation, where are all the viral posts? I've been digging around for a while and haven't found any posts with more than 20 faves. The accounts I've found with ~1 million followers have little to no engagement. I want to see how a post with a million faves holds up to the promises of "fast constant time". I'm especially curious about these queries — fave-count and has-user-faved — since a…

The load generator generates boosts/favorites for a subset of posts that are randomly picked to be "popular". However, since the rate of posts is so high even individual posts picked to be "popular" are only getting ~70 reactions. Tracking reactions is considerably easier than timeline fanout though, as a favorite does a small handful of things (updates set of users favoriting a status and sending a notification), wh…

Thanks for the response, I'm still curious about the details of the subindexing and how that scales. I'll be keeping an eye out for the release!

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#244

Post by sixo »

endisneigh wrote: post #215
sdwr wrote: post #207

Earlier quoted context omitted.

In a typical architecture, the DB stores data, and the backend calls the DB to make updates and compile views. Here, the "views" are defined formally (the P-states), and incrementally, automatically updated when the underlying data changes. Example problem: Get a list of accounts that follow account 1306 "Classic architecture": - Naive approach. Search through all accounts follow lists for "1306". Super slow, scales…

I read their post and honestly it’s not really that much different than just materialized views in a regular database plus async jobs to do the long running tasks. It’s a ridiculous amount of fluff to describe that. Not to mention it’s proprietary and only supports the JVM and doesn’t integrate with the tons of tooling designed about RDBMS unless you stream everything to them, defeating the purpose. What really irks…

The difference is that the materialized-view logic lives naturally in the application code; there's no step where they go out of the DB to do computations and then reinsert.

Once SQL materialized views aren't enough, you might do this by replicating your database into Kafka, implementing logic in Flink or something, and reinserting into the same DB/Elasticsearch/etc. Very common architecture. (Writ small, could also use a queue processor like RabbitMQ.)

Their approach is to instead--apparently--make all of these first-class elements of the same ecosystem, not by "putting it all in the database", but by putting the database into the application code. Which seems wild, but colocates data, transformation, and view.

Seems like it would open up a lot of cans of worms, but if you solve those, sounds great.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#245

Post by nathanmarz »

teacpde wrote: post #218
nathanmarz wrote: post #172

Earlier quoted context omitted.

Yes. Depot appends by default don't return success until colocated streaming topologies have completed processing the data. So this is one way to coordinate the frontend with changes on the backend. Within an ETL, when the computations you do on PStates are colocated with them, you always read your own writes.

It makes sense, but wouldn’t the write be slow? Especially when you have many streaming pipelines.

That's part of designing Rama applications. Acking is only coordinated with colocated stream topologies – stream topologies consuming that depot from another module don't add any latency.

Internally Rama does a lot of dynamic auto-batching for both depot appends and stream ETLs to amortize the cost of things like replication. So additional colocated stream topologies don't necessarily add much cost (though that depends on how complex the topology is, of course).

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#246

Post by Fomite »

adventured wrote: post #200
hosh wrote: post #190

Earlier quoted context omitted.

How much of Twitter’s code base is dedicated to things like security, compliance, and moderation? Granted, a decentralized platform would eliminate some of those, just by being decentralized

None of those things get eliminated by decentralization, they get distributed to whatever the point of control / ownership is. Mastodon still requires security, compliance and moderation. And those requirements are going to keep getting more challenging by the year. It'll end up being another reason nobody will want to host content in a decentralized manner, the burden will become obnoxious.

During the first wave of Twitter exodus, several people in my professional circle asked if they should be hosting professional, field-specific Mastadon servers, etc.

My answer, born of moderating a modestly sized forum, was "Absolutely not under any circumstances."

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#247

Post by [deleted] »

nathanmarz wrote: post #24
raverbashing wrote: post #8

They deserve congrats for that since they built the load test to prove this Of course, for actual production use, there's probably a lot of things still, but this is a very nice works nonetheless

I wouldn't call our instance a load test, as it's a legitimate instance available for anyone to use. It's very much production-grade.

[deleted]

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#248

Post by Pxtl »

nathanmarz wrote: post #184
squeaky-clean wrote: post #173

Earlier quoted context omitted.

In Nathan Marz's (the article author) book, Big Data, he describes this and calls it the Speed Layer. I haven't fully finished the article yet, but the components it's describing seem to be equivalent to what he calls the Batch Layer and the Serving Layer in his book. But I'm kind of getting the impression this works without any speed layer and is expected to be fast enough as-is.

Rama codifies and integrates the concepts I described in my book, with the high level model being: indexes = function(data) and query = function(indexes). These correspond to "depots" (data) , "ETLs" (functions), "PStates" (indexes), and "queries" (functions). Rama is not batch-based. That is, PStates are not materialized by recomputing from scratch. They're incrementally updated either with stream or microbatch proc…

So the idea is that you could do

1. send event data to depot

2. trigger localized ETLs (or put it high-priority in queue) to recalculate just the impacted data into relevant PStates

3. await completion of aforementioned ETLs

4. run query from updated PStates

Maybe too heavy for an upvote, but very appropriate for a an important transaction like a purchase.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#249

Post by tester457 »

sourcecodeplz wrote: post #112

Who cares. Mastodon was/is destined to fail. Trigger happy mods ban you from a server, then you're banned from a bunch.

It can be the smallest things. Just the other day a CS professor was banned from a programming server for interviewing at the NSA 15 years ago. That was pre-snowden.

The bright side is that it's easy to move to a new server.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#250

Post by reissbaker »

nathanmarz wrote: post #168
RomanPushkin wrote: post #163

> ...10k lines of code. This is 100x less code than the ~1M lines Twitter I wish I didn't see this comparison, which is not fair at all. Everyone in their right mind understands that the number of features is much less, that's why you have 10k lines. Add large-scale distributed live video support at the top of that, and you won't get any close to 10k lines. It's only one of many many examples. I really wish you compa…

We're comparing just to the original consumer product, which is about the same as Mastodon is today. That's why we said "original consumer product" and not "Twitter's current consumer product". Mastodon actually has more features than the original Twitter consumer product like hashtag follows, global timelines, and more sophisticated filtering/muting capabilities. Some people argue it's not so expensive to build a sc…

I worked on a previous iteration of Threads at Instagram, and I don't think the 25 person-years number you're estimating spent on the project was spent primarily on backend engineering. They concurrently launched native iOS and Android apps, and I think you're also factoring in the IC count from the linked article to mean full-time ICs dedicated exclusively to Threads, which is probably not how it worked. (Although TBH I'm not sure how you arrived at 25-person-years from the article, which said the project started with "dozens" of engineers and peaked at 50, and was 7 months long in total — a simple average of 24-50 engineers for 7 months, even if you assume they're all full time, would be about 21 person-years).

When I was at IG, my team was 20-something ICs full-time on our project, bursted to maybe double that as necessary in part-time help from ICs in the wider org. We had a total of three backend engineers, of which I was one, as well as a backend intern for a few months, although we bursted the backend team for a couple of months to four for some ML help.

Your project sounds pretty cool! But I don't think the comparison to Threads productivity is quite right. The majority of IG engineering is focused on building polished native clients, not generally on backend infrastructure. It's true that Meta already has a lot of the backend infra built — if Red Planet Labs can come close to what you get at Meta, that's pretty amazing. But I don't think the numbers you're quoting are apples-to-apples, or mean quite what you think they do. I don't know if this version of Threads operated exactly like the one I worked on (a Snapchat competitor), but I'd be pretty surprised if there was a product team at IG that was majority backend.

(Edit: I also think it's worth keeping in mind the experience levels of who works on these projects — at IG it's usually a smaller number of E6/E5s guiding a larger number of E4/E3s. Person-years are not all equivalent! If you spent ten years building the Red Planet Labs infrastructure based on your time at Twitter, nine person-months of your team's time building a product on your infra might not be the same as nine person-months of someone else's.)

Anyway — I don't mean to downplay your product, and really, if it's anything like the backend productivity I experienced at Meta, that would be pretty groundbreaking. Curious to see what you launch :)

Post reply on HN