Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced the cost of building Mastodon at Twitter-scale by 100x

blog.redplanetlabs.com

Post reply on HN
Print view View original

191–200 of 376 posts

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#191

Post by nathanmarz »

mdaniel wrote: post #162
nathanmarz wrote: post #24

Earlier quoted context omitted.

I wouldn't call our instance a load test, as it's a legitimate instance available for anyone to use. It's very much production-grade.

I would not want to speak for raverbashing but I feel the same way: I actually can't tell if the bug is with soapbox or with your instance but clicking on the first link from your post practically locks up my browser due to every single Toot getting swapped out "at twitter scale" If one clicks quick enough to jump to an actual post, it seems relatively static so it's hard to tell if the bots are deleting and recreati…

We did notice on a less powerful machines the browser getting overwhelmed with the rate of new content (even though we're only streaming 10/s instead of the full 3.5k/s actually happening on the backend). I don't know if the poor performance in this context is due to Soapbox, the browser, or just the hardware.

To get a better feeling of Rama's performance on your hardware, I suggest registering an account which will allow you to poke around the whole platform. It takes just a couple seconds to register and we don't send any emails.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#192

Post by dataangel »

I do C++ backend work in a non-web industry and this entire post is Greek to me. Even though this is targeted at developers, you need a better pitch. I get "we did this 100x faster" but the obvious followup question is "how" but then the answer seems to be a ton of flow diagrams with way too many nodes that tell me approximately nothing and some handwaving about something called P-States that are basically defined to be entirely nebulous because they are any kind of data structure.

I'm not saying there's nothing here, but I am adjacent to your core audience and I have no idea whether there is after reading your post. I think you are strongly assuming a shared basis where everybody has worked on the same kind of large scale web app before; I would find it much more useful to have an overview of, "This what you would usually do, here are the problems with it, here is what we do instead" with side by side code comparison of Rama vs what a newbie is likely to hack together with single instance postgres.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#193

Post by chubot »

nathanmarz wrote: post #176
chubot wrote: post #151

Earlier quoted context omitted.

Yeah definitely, these ideas always sound very appealing to me, in theory -- I almost wonder why nobody has built it before e.g. they mention "event sourcing" and "materialized views" in the post -- sounds good But I thought I heard from a few people who were like "we ripped event sourcing" out of our codebase and so forth And yeah your question is an obvious good one, and the Reddit answer of "write through cache" .…

When you step back and consider the incredible amount of manpower and resources that have been put into these applications, it's amazing how buggy these applications are. To put it simply, they're buggy because the underlying infrastructure and techniques used to build them are so complex that the implementation is beyond the realm of human understanding. The way applications are built, and have been built since befo…

Yes, I 100% agree with you. I would like something like this to succeed, and agree the problem is real.

But what are the tradeoffs? There's nothing that comes with 100x benefit with no tradeoffs

(side note: I worked on Google Code for a short while in 2008, concurrent with Github's founding ... I think Github moved a lot faster in a large part because they weren't dealing with distributed systems at first -- they had a Rails app, a database, and RAID disks, and grew it from there. We had BigTable and perf reviews :-P )

Eventual consistency is probably one?

Can I specify that comment editing is correct and ACID, while likes/upvotes are eventually consistent? (No is a fine answer, these problems are hard)

I read through much of the doc, and don't see a mention of the word "consistency" at all, which seems like an oversight for something that is unifying what would be in a database with computation.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#197

Post by ehutch79 »

nathanmarz wrote: post #168
RomanPushkin wrote: post #163

> ...10k lines of code. This is 100x less code than the ~1M lines Twitter I wish I didn't see this comparison, which is not fair at all. Everyone in their right mind understands that the number of features is much less, that's why you have 10k lines. Add large-scale distributed live video support at the top of that, and you won't get any close to 10k lines. It's only one of many many examples. I really wish you compa…

We're comparing just to the original consumer product, which is about the same as Mastodon is today. That's why we said "original consumer product" and not "Twitter's current consumer product". Mastodon actually has more features than the original Twitter consumer product like hashtag follows, global timelines, and more sophisticated filtering/muting capabilities. Some people argue it's not so expensive to build a sc…

Are you sure about that.

With things like twitter, the ui is not the hard part. Things like moderation are the secret sauce. All the corner cases and support for devopsy stuff likely account for a lot. Routing to specific instances for celebrities and such.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#198

Post by joelthelion »

nathanmarz wrote: post #74
skybrian wrote: post #54

It sounds like interesting technology for someone, but I wonder more about scaling down. What does a developer instance running on a laptop look like?

Great question. There's actually two ways to look at this: what does it look like to run Rama in a unit test environment, and what does it look like to run a small-scale single-node Rama application in production? For the former, Rama has a class called "InProcessCluster" that works identically to a real cluster. It enables Rama applications to be tested and experimented with end-to-end. There's an example of this in…

Follow up question : do you see Rama as being a good fit for applications that /don't/ need Twitter scale? These have simpler requirements, but I feel the integration you propose could still have value there.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#199

Post by chiefalchemist »

"We stood on the shoulders of giants..."

X years from now "We reduced the cost of building _____ at Mastodon-scale by 1000x".

It's certainly interesting, certainly an accomplishment, but it's also the nature of the game. The present eating the past, to be eaten by the future. Rinse. Repeat.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#200

Post by adventured »

hosh wrote: post #190
RomanPushkin wrote: post #163

> ...10k lines of code. This is 100x less code than the ~1M lines Twitter I wish I didn't see this comparison, which is not fair at all. Everyone in their right mind understands that the number of features is much less, that's why you have 10k lines. Add large-scale distributed live video support at the top of that, and you won't get any close to 10k lines. It's only one of many many examples. I really wish you compa…

How much of Twitter’s code base is dedicated to things like security, compliance, and moderation? Granted, a decentralized platform would eliminate some of those, just by being decentralized

None of those things get eliminated by decentralization, they get distributed to whatever the point of control / ownership is.

Mastodon still requires security, compliance and moderation. And those requirements are going to keep getting more challenging by the year. It'll end up being another reason nobody will want to host content in a decentralized manner, the burden will become obnoxious.

Post reply on HN