Live data from Hacker News

We reduced the cost of building Mastodon at Twitter-scale by 100x

blog.redplanetlabs.com

321–330 of 376 posts

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#321

Earlier quoted context omitted.

yea but does hn have any client side js?

Yeah, a very small amount so that clicking the upvote button does not need to reload the whole page

Upvoting does work javascript disabled, it upvotes and reloads the page. It approximately keeps your position by jumping the message anchor.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#322
> To demonstrate the scale of our instance, we’re also running 100M bot accounts which continuously post statuses (Mastodon’s analogue of a “tweet”), replies, boosts (“retweet”), and favorites. 3,500 statuses are posted per second, the average number of followers for each post is 403, and the largest account has over 22M followers. As a comparison, Twitter serves 7,000 tweets per second at 700 average fanout (according to the numbers I could find).

Is Twitters 7k tweets per second the average? If so, what’s the peak rate, and have you tested your system under this load?

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#323

Earlier quoted context omitted.

Any JVM language should work. We've built modules with Clojure. We're keeping it closed-source for now.

> We're keeping it closed-source for now. Rama sounds interesting to me for my 'next big project', but I'd not even consider building it on top of a closed core. I think this is a pretty common sentiment in these circles. I understand building an OSS business is not easy either. But perhaps there is some middle of the road that you can walk? - A contractual obligation to open source all (now current) code a couple of…

Yep, not OSS licensed is a nope for us for core dependencies. I am aware many here like to sign away freedom (even future viability) for more productivity now (using some new fangled cloud thingy which might/will be gone in a few months to underpin your entire product), but we are not interested in that.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#324
Why choose Java of all languages. Why not something more modern and less verbose like Go or Rust. Just asking as I have worked enough in Java and then spend a lot of time in GC tunining. Granted the code was not that great and from a diverse team with different skill levels causing all the leaks.. But still

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#325
I think the marketing idea of this is amazing : I would probably never even consider learning and reading about such a framework if I heard of it straight up. But if you are really releasing a usable open source implementation of something performant that actually federates properly, that is a huge selling point that buys you a ton of respect up front.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#326
post #9

I would argue that this is not "a Mastodon instance", since it is not running Mastodon - other than that, very very neat work! I'm excited for that "Source Code" link to be live :)

I think it's smart from a legal perspective, because the team members seem to partially be coming from companies acquired by Twitter. So I guess, if you say "it's a Mastodon-clone", you cannot be accused of taking proprietary ideas from Twitter (this is just a guess, they know better). But technically very interesting and refreshing to see. I really like their approach. It feels they are innovative.

From a legal perspective, it's against Mastodon's trademark policy: "Only use the Mastodon marks to accurately identify those goods or services that are built using the Mastodon software." https://joinmastodon.org/trademark

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#328

Earlier quoted context omitted.

We're comparing just to the original consumer product, which is about the same as Mastodon is today. That's why we said "original consumer product" and not "Twitter's current consumer product". Mastodon actually has more features than the original Twitter consumer product like hashtag follows, global timelines, and more sophisticated filtering/muting capabilities. Some people argue it's not so expensive to build a sc…

I work in gaming, so I cannot speak to your specific experiences. Entity Component Systems are extremely performant, really good science, and shipping in middlewares like Unity. However, in order to ship an ECS game, in my experience, you have to have already made your whole game first in a normal approach, in order to have everything be fully specified sufficiently that you can correctly create an ECS implementation…

> your whole game first in a normal approach

What do you mean by normal approach? OO classes with inheritance?

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#329

Noticeably missing are any details about concurrency control and replication or recovery protocols. A Twitter clone is one thing but any sort of application needing ACID Transactions is a whole other beast.

All data on Rama is replicated automatically with a configurable "replication factor". Data written to Rama is not made visible until it's successfully replicated. The documentation we're releasing next week includes a page going into detail in how this works.

Does "replicated" here mean that the data is persisted to disk?

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#330
post #47

Earlier quoted context omitted.

I consider "DSL" as something that's its own language with it's own lexer and parser, like SQL. The Rama API is just Java – classes, interfaces, methods, etc. Everything you do in Rama, from defining indexes, performing queries, or writing dataflow ETLs, is done in Java.

This is usually referred to as an "embedded DSL" - you have a DSL embedded in a normal programming language using its first class constructs.

Meh. By this definition all libraries expose an "embedded DSL" — their API. I'm honestly not sure this is a useful definition.
Post reply on HN