Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced the cost of building Mastodon at Twitter-scale by 100x

blog.redplanetlabs.com

Post reply on HN
Print view View original

211–220 of 376 posts

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#211

Post by MikePlacid »

RomanPushkin wrote: post #163

> ...10k lines of code. This is 100x less code than the ~1M lines Twitter I wish I didn't see this comparison, which is not fair at all. Everyone in their right mind understands that the number of features is much less, that's why you have 10k lines. Add large-scale distributed live video support at the top of that, and you won't get any close to 10k lines. It's only one of many many examples. I really wish you compa…

So

>> 100M bots posting 3,500 times per second...

and

> We used the OpenAI API to generate 50,000 statuses for the bots to choose from at random.

I wonder: 100M OpenAI bots talking to each other continuously and with much vigor - how is this affecting OpenAI’s uhm… intellect?

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#212

Post by nathanmarz »

kyle-rb wrote: post #165

Kinda disappointed by the simulation, where are all the viral posts? I've been digging around for a while and haven't found any posts with more than 20 faves. The accounts I've found with ~1 million followers have little to no engagement. I want to see how a post with a million faves holds up to the promises of "fast constant time". I'm especially curious about these queries — fave-count and has-user-faved — since a…

The load generator generates boosts/favorites for a subset of posts that are randomly picked to be "popular". However, since the rate of posts is so high even individual posts picked to be "popular" are only getting ~70 reactions.

Tracking reactions is considerably easier than timeline fanout though, as a favorite does a small handful of things (updates set of users favoriting a status and sending a notification), while fanout has to do an operation on every follower (403 operations on average, sometimes up to 22M).

The code getting the favorite count for a status looks like:

  .localSelect("$$statusIdToFavoriters", Path.key("*statusId").view(Ops.SIZE)).out("*numFavorites")
Because the nested set is subindexed, that's an extremely fast operation (looking at our telemetry, about 0.05ms).

Determining "has-user-faved" looks like:

  .localSelect("$$statusIdToFavoriters", Path.key("*statusId").view(Ops.CONTAINS, "*accountId")).out("\*hasFavorited")
The API server doesn't do these queries individually, which would be two roundtrips. It does them together in a query topology along with fetching other needed information (like number of boosts, number of replies, "has boosted", "has muted this status", etc.).

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#213

Post by rollcat »

HaZeust wrote: post #202
dataangel wrote: post #192

I do C++ backend work in a non-web industry and this entire post is Greek to me. Even though this is targeted at developers, you need a better pitch. I get "we did this 100x faster" but the obvious followup question is "how" but then the answer seems to be a ton of flow diagrams with way too many nodes that tell me approximately nothing and some handwaving about something called P-States that are basically defined to…

... Maybe the post isn't targeted to your audience at all? How is "C++" and "non-web work" adjacent to web work with web launguage audiences?

OP did not specify what their industry actually is. I've been doing "web work" for 17 years and I'm sharing their concern: where's the TL;DR for this? If this somehow can make me 100x as productive, how about starting with a "hello world" example that shows me how is it different from pip install django, etc?

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#215

Post by endisneigh »

sdwr wrote: post #207
dataangel wrote: post #192

I do C++ backend work in a non-web industry and this entire post is Greek to me. Even though this is targeted at developers, you need a better pitch. I get "we did this 100x faster" but the obvious followup question is "how" but then the answer seems to be a ton of flow diagrams with way too many nodes that tell me approximately nothing and some handwaving about something called P-States that are basically defined to…

In a typical architecture, the DB stores data, and the backend calls the DB to make updates and compile views. Here, the "views" are defined formally (the P-states), and incrementally, automatically updated when the underlying data changes. Example problem: Get a list of accounts that follow account 1306 "Classic architecture": - Naive approach. Search through all accounts follow lists for "1306". Super slow, scales…

I read their post and honestly it’s not really that much different than just materialized views in a regular database plus async jobs to do the long running tasks.

It’s a ridiculous amount of fluff to describe that. Not to mention it’s proprietary and only supports the JVM and doesn’t integrate with the tons of tooling designed about RDBMS unless you stream everything to them, defeating the purpose.

What really irks me is that they go on and on bragging about the low LoC count and literally show nothing complete. They should’ve held on this post and released it simultaneously with the code.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#216

Post by nathanmarz »

joelthelion wrote: post #198
nathanmarz wrote: post #74

Earlier quoted context omitted.

Great question. There's actually two ways to look at this: what does it look like to run Rama in a unit test environment, and what does it look like to run a small-scale single-node Rama application in production? For the former, Rama has a class called "InProcessCluster" that works identically to a real cluster. It enables Rama applications to be tested and experimented with end-to-end. There's an example of this in…

Follow up question : do you see Rama as being a good fit for applications that /don't/ need Twitter scale? These have simpler requirements, but I feel the integration you propose could still have value there.

Yes, it's a better model for developing backends in general. Our comparison against Mastodon's official implementation demonstrates this, being at least 44% less code.

It's the ability to avoid the impedance mismatches which dominate existing tooling that makes such a difference. With existing databases, including RDBMS's, you have to twist your application to fit their data models. The existence of things like ORMs help, but they add their own layers of complexity.

With Rama, you mold your indexes to exactly match your application's needs. And you're always just working with objects represented however you want, whether appending data to depots, processing data in ETLs, or storing data in PStates.

That computation and storage are integrated and colocated is another way that Rama simplifies application development and deployment.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#217

Post by nathanmarz »

NoahTheDuke wrote: post #170

Congrats! This looks super cool. Are there any plans for exposing a Clojure API? Given that it's implemented in Clojure, seems like it would be a natural fit. Interop with Java is nice but can be cumbersome compared to the more natural calling conventions and idioms (threading macros instead of `..` builder patterns, etc).

Answered this in another comment: https://news.ycombinator.com/item?id=37138526

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#218

Post by teacpde »

nathanmarz wrote: post #172
jitl wrote: post #161

Earlier quoted context omitted.

is there a way to guarantee reading your own writes from a client perspective?

Yes. Depot appends by default don't return success until colocated streaming topologies have completed processing the data. So this is one way to coordinate the frontend with changes on the backend. Within an ETL, when the computations you do on PStates are colocated with them, you always read your own writes.

It makes sense, but wouldn’t the write be slow? Especially when you have many streaming pipelines.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#219

Post by softwaredoug »

nathanmarz wrote: post #53
softwaredoug wrote: post #42

It’s a massive ask, even if the platform was 100x better, for all developers to give up every programming language and database they’ve ever used to depend on a startups closed source platform for all functionality. It’s hard enough trusting Google or Amazons cloud offerings won’t change. It seems that’s what they’re proposing right? What am I missing?

We're actually not asking anyone to give up anything. First off, it has a simple integration API (which you'll be able to see the details of next week) that allows it to seamlessly integrate with any other backend tool (databases, monitoring systems, queues, etc.). So Rama can be incrementally introduced into any existing architecture. Second, Rama has a pure Java API and is not a bespoke language. So no new language…

I can imagine this being really useful from the ground up. Because it looks like it wants to be the source of truth, with different views on the data.

It’s hard to imagine it for a complex legacy application without having lots of added complexity. It wants to be the unifying programming model for the application. It would seem like running with two RDMS sources of truth simultaneously.

It’s like the xkcd “there are 12 ways of doing X, let’s create a standard to unify them” now there are 13 ways

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#220

Post by sdwr »

MikePlacid wrote: post #211
RomanPushkin wrote: post #163

> ...10k lines of code. This is 100x less code than the ~1M lines Twitter I wish I didn't see this comparison, which is not fair at all. Everyone in their right mind understands that the number of features is much less, that's why you have 10k lines. Add large-scale distributed live video support at the top of that, and you won't get any close to 10k lines. It's only one of many many examples. I really wish you compa…

So >> 100M bots posting 3,500 times per second... and > We used the OpenAI API to generate 50,000 statuses for the bots to choose from at random. I wonder: 100M OpenAI bots talking to each other continuously and with much vigor - how is this affecting OpenAI’s uhm… intellect?

They generated 50,000 statuses once, put them in a text file, and pick between them randomly. So not at all.
Post reply on HN