Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced the cost of building Mastodon at Twitter-scale by 100x

blog.redplanetlabs.com

Post reply on HN
Print view View original

201–210 of 376 posts

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#201

Post by ldayley »

dataangel wrote: post #192

I do C++ backend work in a non-web industry and this entire post is Greek to me. Even though this is targeted at developers, you need a better pitch. I get "we did this 100x faster" but the obvious followup question is "how" but then the answer seems to be a ton of flow diagrams with way too many nodes that tell me approximately nothing and some handwaving about something called P-States that are basically defined to…

Nathan Marz created Apache Storm, coauthored the book "Big Data", and founded an early real-time infrastructure team at Twitter. It's likely the 'curse of knowledge' of working on this specific problem for so long is responsible for the unique and/or unfamiliar style of communication here.

EDIT: Specifics

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#202

Post by HaZeust »

dataangel wrote: post #192

I do C++ backend work in a non-web industry and this entire post is Greek to me. Even though this is targeted at developers, you need a better pitch. I get "we did this 100x faster" but the obvious followup question is "how" but then the answer seems to be a ton of flow diagrams with way too many nodes that tell me approximately nothing and some handwaving about something called P-States that are basically defined to…

... Maybe the post isn't targeted to your audience at all? How is "C++" and "non-web work" adjacent to web work with web launguage audiences?

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#203

Post by justrealist »

ehutch79 wrote: post #197
nathanmarz wrote: post #168

Earlier quoted context omitted.

We're comparing just to the original consumer product, which is about the same as Mastodon is today. That's why we said "original consumer product" and not "Twitter's current consumer product". Mastodon actually has more features than the original Twitter consumer product like hashtag follows, global timelines, and more sophisticated filtering/muting capabilities. Some people argue it's not so expensive to build a sc…

Are you sure about that. With things like twitter, the ui is not the hard part. Things like moderation are the secret sauce. All the corner cases and support for devopsy stuff likely account for a lot. Routing to specific instances for celebrities and such.

Nathan worked at Twitter so while he might be wrong, I don't think it's reasonable to assume he's just naive http://nathanmarz.com/blog/leaving-twitter.html.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#204

Post by CuriousCosmic »

StephenAmar wrote: post #122
afro88 wrote: post #118

Looks amazing and incredibly smart. But I found the LOC and implementation time comparisons to Twitter and Threads very disingenuous. It makes me wonder what other wool will be pulled over our eyes with Rama in future (or important real world details missed / future footguns). Still super impressive. Reminds me of when I discovered Elixir while building a social-ish music discovery app. Switching the backend from Rai…

+1 the comparisons are not great. How much engineer-hours did it take to build Rama itself? The numbers they got for Twitter likely include the time it took to build their infrastructure, common libraries (like finagle,…)

Honestly I'm willing to accept the number they gave since the author (Nathan Marz) was one of the lead/founding devs for twitter's streaming compute backend in the past.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#205

Post by alberth »

FridgeSeal wrote: post #195

Semi-related: Their homepage ( https://redplanetlabs.com/ ) has to be one of the best looking websites I’ve seen in a while, buttery smooth as well. I love it.

If you like the pretty static background look with light text, checkout https://carrd.co/

It’s a website builder with lots of themes similar in design.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#206

Post by hosh »

adventured wrote: post #200
hosh wrote: post #190

Earlier quoted context omitted.

How much of Twitter’s code base is dedicated to things like security, compliance, and moderation? Granted, a decentralized platform would eliminate some of those, just by being decentralized

None of those things get eliminated by decentralization, they get distributed to whatever the point of control / ownership is. Mastodon still requires security, compliance and moderation. And those requirements are going to keep getting more challenging by the year. It'll end up being another reason nobody will want to host content in a decentralized manner, the burden will become obnoxious.

An organization trying to maintain an ISO certification will have drastically different policies and controls than a small shop, or even a hobbyist group.

Everyone (theoretically) would be complying to statues in its broadest sense, but jurisdiction, regulations, industry best practices, reporting requirements, and appetite for risk is going to be different from organization to organization. It’s not one-size-fits-all.

So some of these things are eliminated because the people hosting those are not put under the same kind of scrutiny as say, a Twitter.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#207

Post by sdwr »

dataangel wrote: post #192

I do C++ backend work in a non-web industry and this entire post is Greek to me. Even though this is targeted at developers, you need a better pitch. I get "we did this 100x faster" but the obvious followup question is "how" but then the answer seems to be a ton of flow diagrams with way too many nodes that tell me approximately nothing and some handwaving about something called P-States that are basically defined to…

In a typical architecture, the DB stores data, and the backend calls the DB to make updates and compile views.

Here, the "views" are defined formally (the P-states), and incrementally, automatically updated when the underlying data changes.

Example problem:

Get a list of accounts that follow account 1306

"Classic architecture":

- Naive approach. Search through all accounts follow lists for "1306". Super slow, scales terribly with # of accounts.

- Normal approach. Create a "followed by" table, update it whenever an account follows / unfollows / is deleted / is blocked.

Normal sounds good, but add 10x features, or 1000x users, and it gets trickier. You need to make a new table for each feature, and add conditions to the update calls, and they start overlapping... Or you have to split the database up so it scales, but then you have to pay attention to consistency, and watch which order stuff gets updated in.

Their solution is separating the "true" data tables from the "view" tables, formally defining the relationship between the two, and creating the "view" tables magically behind the scenes.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#208

Post by ThinkBeat »

I am confused.

This is meant to be hyped to sell your Rama platform/product/framework? That you have spent 10 years building in secret? During that time you have built a datastore and a Kafke competitor and ?

Should not those 10 years be factored into the time it took to develop this technical demo?

Is it 100x less code including every LOC in all of Rama?

I mean I am sure you picked a use cast that is well suited to creating a Twitterish architecture implementation.

If I went off and wrote a ThinkBeat platform for creating Twitterish systems and then created a Twitterish implementation on top if it, its real easy to reach low LOCs.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#209

Post by dunk010 »

bo1024 wrote: post #188
dunk010 wrote: post #145

Earlier quoted context omitted.

Exactly, why are so many people missing this point. It's not "we built a narrow, tedious framework for knocking off Twitter clones", it's "We built a platform that turns data processing on its head and look in a couple of months you can clone Twitter just imagine what YOU can do with this." I see parallels though to Datomic, where they turned the database inside out, co-located the app logic and data and indexes, etc…

I think they didn't do a good job making the point clear for people who just clicked the link without context. It starts off talking a lot about the Mastodon clone and then gradually starts talking about Rama as it goes.

People should probably close the TikTok and pick up a book instead to increase their attention spans then :-D

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#210

Post by slim »

HaZeust wrote: post #202
dataangel wrote: post #192

I do C++ backend work in a non-web industry and this entire post is Greek to me. Even though this is targeted at developers, you need a better pitch. I get "we did this 100x faster" but the obvious followup question is "how" but then the answer seems to be a ton of flow diagrams with way too many nodes that tell me approximately nothing and some handwaving about something called P-States that are basically defined to…

... Maybe the post isn't targeted to your audience at all? How is "C++" and "non-web work" adjacent to web work with web launguage audiences?

he's a developer and curious about the subject. Since it's a blog post, not a scientific paper, the fact that he did not understand could be a communication failure. I think he's being helpful
Post reply on HN