Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced the cost of building Mastodon at Twitter-scale by 100x

blog.redplanetlabs.com

Post reply on HN
Print view View original

261–270 of 376 posts

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#261

Post by INTPenis »

legi0nary wrote: post #254
nathanmarz wrote: post #120

Earlier quoted context omitted.

We call it a "Mastodon instance" because we implemented the entire Mastodon API ( https://docs.joinmastodon.org/api/ ). This is in addition to also implementing the ActivityPub API which Mastodon also implements for federation.

If you can do this with Bluesky once it federates we might be able to get away from twitter for good.

>We spent nine person-months building our scalable Mastodon instance

They federated this brand new code in 9 months, and bluesky still hasn't released anything regarding federation. Don't keep your hopes up, it would kill their business model to let anyone run part of the network. People-driven networks are just not compatible with commercially driven ones, name one successful example.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#262

Post by nathanmarz »

ThinkBeat wrote: post #208

I am confused. This is meant to be hyped to sell your Rama platform/product/framework? That you have spent 10 years building in secret? During that time you have built a datastore and a Kafke competitor and ? Should not those 10 years be factored into the time it took to develop this technical demo? Is it 100x less code including every LOC in all of Rama? I mean I am sure you picked a use cast that is well suited to…

Your comment is addressed in another thread by other commenters: https://news.ycombinator.com/item?id=37137653

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#263

Post by nathanmarz »

miki123211 wrote: post #229

Is this just me, or does the code in the post feel like they've implemented what should have been a new programming language on top of Java? Their "variables" have names that you have to keep as Java strings and pass to random functions. If you want composable code, you don't declare a function, you call .macro(). For control flow and loops, you don't use if and for, but a weird abstraction of theirs. I feel like thi…

Internally there actually is a new programming language, implemented using Clojure macros (so it's also Clojure). The Java dataflow API is exposing a subset of that language. We did it this way rather than expose this new language directly because most people don't know Clojure and we don't feel it necessary or desirable to require people to have to learn a new language to benefit from this technology.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#264

Post by Aeolun »

nathanmarz wrote: post #252
endisneigh wrote: post #215

Earlier quoted context omitted.

I read their post and honestly it’s not really that much different than just materialized views in a regular database plus async jobs to do the long running tasks. It’s a ridiculous amount of fluff to describe that. Not to mention it’s proprietary and only supports the JVM and doesn’t integrate with the tons of tooling designed about RDBMS unless you stream everything to them, defeating the purpose. What really irks…

We are very open in the post that the core concepts are not new: Individually, none of these concepts are new. I’m sure you’ve seen them all before. You may be tempted to dismiss Rama’s programming model as just a combination of event sourcing and materialized views. But what Rama does is integrate and generalize these concepts to such an extent that you can build entire backends end-to-end without any of the impedan…

> But what Rama does is integrate and generalize these concepts to such an extent that you can build entire backends end-to-end without any of the impedance mismatches or complexity

Every time I hear this the reality turns out to be that building anything with this tech is like building something on top of SAP.

But I’m also just allergic to any post that says ‘look how amazing’ in general, so I’m a bit prejudiced.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#265

Post by newlisp »

nathanmarz wrote: post #103
theogravity wrote: post #78

Earlier quoted context omitted.

The article says they re-wrote Mastodon from scratch (probably the backend piece). I'm guessing in Java.

Yes, it's 100% written in Java.

This looks interesting and a nice weekend read but just skimming through, why is spring needed?

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#266

Post by nathanmarz »

kyle-rb wrote: post #243
nathanmarz wrote: post #212

Earlier quoted context omitted.

The load generator generates boosts/favorites for a subset of posts that are randomly picked to be "popular". However, since the rate of posts is so high even individual posts picked to be "popular" are only getting ~70 reactions. Tracking reactions is considerably easier than timeline fanout though, as a favorite does a small handful of things (updates set of users favoriting a status and sending a notification), wh…

Thanks for the response, I'm still curious about the details of the subindexing and how that scales. I'll be keeping an eye out for the release!

A lot of detail on subindexing and everything else in Rama is coming next week when we release the docs.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#267

Post by asu_thomas »

hosh wrote: post #190
RomanPushkin wrote: post #163

> ...10k lines of code. This is 100x less code than the ~1M lines Twitter I wish I didn't see this comparison, which is not fair at all. Everyone in their right mind understands that the number of features is much less, that's why you have 10k lines. Add large-scale distributed live video support at the top of that, and you won't get any close to 10k lines. It's only one of many many examples. I really wish you compa…

How much of Twitter’s code base is dedicated to things like security, compliance, and moderation? Granted, a decentralized platform would eliminate some of those, just by being decentralized

And advertising. Twitter is an advertising platform.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#268

Post by otikik »

RomanPushkin wrote: post #163

> ...10k lines of code. This is 100x less code than the ~1M lines Twitter I wish I didn't see this comparison, which is not fair at all. Everyone in their right mind understands that the number of features is much less, that's why you have 10k lines. Add large-scale distributed live video support at the top of that, and you won't get any close to 10k lines. It's only one of many many examples. I really wish you compa…

Indeed. Add a single JavaScript dependency… you will get the banana, the gorilla holding the banana, the tree holding the gorilla, and the whole jungle.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#269

Post by FridgeSeal »

nathanmarz wrote: post #184
squeaky-clean wrote: post #173

Earlier quoted context omitted.

In Nathan Marz's (the article author) book, Big Data, he describes this and calls it the Speed Layer. I haven't fully finished the article yet, but the components it's describing seem to be equivalent to what he calls the Batch Layer and the Serving Layer in his book. But I'm kind of getting the impression this works without any speed layer and is expected to be fast enough as-is.

Rama codifies and integrates the concepts I described in my book, with the high level model being: indexes = function(data) and query = function(indexes). These correspond to "depots" (data) , "ETLs" (functions), "PStates" (indexes), and "queries" (functions). Rama is not batch-based. That is, PStates are not materialized by recomputing from scratch. They're incrementally updated either with stream or microbatch proc…

Forgive me if I’m misunderstanding things, but this seems quite similar to what Materialize and ReadySet do, but like “as a library”, because Rama doesn’t use a “separate” layer for the storage stuff. Is that correct-ish?
Post reply on HN