Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced the cost of building Mastodon at Twitter-scale by 100x

blog.redplanetlabs.com

Post reply on HN
Print view View original

341–350 of 376 posts

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#341

Post by prepor »

dustingetz wrote: post #48

Summarizing, now edited down with some editorializing for clarity: What is it? build web-scale reactive backends with an expressive java dataflow API. Instead of a database you develop your own custom app-specific indexes which are reactive, distributed and durable. It's like event sourcing and materialized views but integrated in a linearly scalable way. > I cannot emphasize enough how much interacting with indexes…

Electric could be an ideal companion to these reactive Queries & Depots to build actual UI, btw

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#342

Post by Aeolun »

throwuxiytayq wrote: post #299
Aeolun wrote: post #288

Earlier quoted context omitted.

I think that’s probably not the case for many new developers that don’t have any exposure to anything not React. Of course ‘react for data’ is entirely misleading, but it may give a decent idea if you don’t have an hour to spend on an explanation.

In other words, it’s the Dark Souls of application backends, but entirely different.

Only if your expectation is to be constantly frustrated, and eventually die, after which you have to do it all again.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#343

Post by mariusor »

But this is not a Mastodon instance.

It's something else that maybe speaks the Mastodon API and/or ActivityPub, but we don't know since it doesn't really federate with anyone.

I commend the effort to try to make happen a non-open fediverse service, but appropriating the Mastodon name is just wrong. You should know better.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#344

Post by dustingetz »

3cats-in-a-coat wrote: post #97
dustingetz wrote: post #91

Earlier quoted context omitted.

you're missing automatic/free linear scaling

Systems that promise "free linear scaling" without qualifiers either withhold or have not analyzed/realized their bottlenecks yet. Say if there is eventual consistency maybe the "eventuality" becomes so long that the service fails at its purpose. Or the communication link bandwidth is exhausted between key business logic (mutation event generating) services, and so on. The only systems that scale linearly are statele…

for example it can scale the way persistent data structures scale, which is to say "O(1) within target operational bounds" despite technically being log-n with high branch factor)

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#345

Post by dustingetz »

leonidasv wrote: post #233
endisneigh wrote: post #215

Earlier quoted context omitted.

I read their post and honestly it’s not really that much different than just materialized views in a regular database plus async jobs to do the long running tasks. It’s a ridiculous amount of fluff to describe that. Not to mention it’s proprietary and only supports the JVM and doesn’t integrate with the tons of tooling designed about RDBMS unless you stream everything to them, defeating the purpose. What really irks…

IIUIC, the most significant difference from a materialized view is that the Rama infrastructure recompute only the changed data by checking the relationship between fields, while a traditional materialized view recomputes the whole table?

isn't Materialized performing symbolic differentiation of SQL queries?

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#346

Post by samsquire »

Thank you for sharing this!

I am actually really impressed. Well done! Good work!

There's lots of interesting lessons and knowledge in the design of this platform.

I also like how you've decided to use Java as your API rather than Clojure.

I hope you're not discouraged by HN's reaction to your hard work. Don't be discouraged!

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#347

Post by prepor »

Based on what I read it's very similar to Kafka Streams + batteries ([semi]automatic workload orchestration, reactive queries, higher-level/slicker/"smarter" API (?))

Could you please compare Rama with Kafka Streams, especially from the point of view, if I would try to reimplement Rama API on top of Kafka Streams? What fundamental difficulties I'd face?

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#348

Post by kingkongjaffa »

doctorpangloss wrote: post #241
nathanmarz wrote: post #168

Earlier quoted context omitted.

We're comparing just to the original consumer product, which is about the same as Mastodon is today. That's why we said "original consumer product" and not "Twitter's current consumer product". Mastodon actually has more features than the original Twitter consumer product like hashtag follows, global timelines, and more sophisticated filtering/muting capabilities. Some people argue it's not so expensive to build a sc…

I work in gaming, so I cannot speak to your specific experiences. Entity Component Systems are extremely performant, really good science, and shipping in middlewares like Unity. However, in order to ship an ECS game, in my experience, you have to have already made your whole game first in a normal approach, in order to have everything be fully specified sufficiently that you can correctly create an ECS implementation…

Where can I learn more about the tech behind cities skylines 2?

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#349

Post by lionkor »

za3faran wrote: post #304
lionkor wrote: post #116

Very interesting, looking forward to reading the docs once they come out. Why Java?

The more apt question I would argue is 'why not Java?' It's the most sane and reliable default for such projects.

It doesnt change that it would be interesting to know why Java was chosen - its definitely not the default for anything anywhere I've worked
Post reply on HN