Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced the cost of building Mastodon at Twitter-scale by 100x

blog.redplanetlabs.com

Post reply on HN
Print view View original

141–150 of 376 posts

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#141

Post by dunk010 »

nathanmarz wrote: post #74
skybrian wrote: post #54

It sounds like interesting technology for someone, but I wonder more about scaling down. What does a developer instance running on a laptop look like?

Great question. There's actually two ways to look at this: what does it look like to run Rama in a unit test environment, and what does it look like to run a small-scale single-node Rama application in production? For the former, Rama has a class called "InProcessCluster" that works identically to a real cluster. It enables Rama applications to be tested and experimented with end-to-end. There's an example of this in…

You're killing it with the replies here +++

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#143

Post by nathanmarz »

3cats-in-a-coat wrote: post #129
nathanmarz wrote: post #95

Earlier quoted context omitted.

From the post: Individually, none of these concepts are new. I’m sure you’ve seen them all before. You may be tempted to dismiss Rama’s programming model as just a combination of event sourcing and materialized views. But what Rama does is integrate and generalize these concepts to such an extent that you can build entire backends end-to-end without any of the impedance mismatches or complexity that characterize and…

Thanks, I will read more soon! I'm curious... how do you resolve the "impedance mismatch" between some "canonical" models that business decisions are made, based upon, which need to be synchronous with the depots (and mutually synchronous with other models sharing fragments of the same data), and the eventually consistent read models, which have a more lax constraint on how up to date they are? How do you ensure cons…

The impedance mismatches you're used to from using databases are gone because:

- You can finely tune your indexes to be exactly the optimal shape for your application (data structure). You can see this in our Mastodon implementation with the big variety of data structures we used for all the use cases. - You're generally just using regular Java objects everywhere: appending to depots, during ETL processing, and stored in indexes.

How you coordinate data creation with view updates is a deeper topic, so I'll just summarize one of the basic mechanisms Rama provides for coordinating this. Depot appends can have an "ack level" that determines the conditions before Rama tells you that depot append has completed. The default level is "full ack" which includes all streaming topologies colocated with that depot fully processing that record. With this level, when the depot append completes you know that all associated indexes (PStates) have been updated.

There's also "append ack", which only waits for the depot append to be replicated on the depot, and "no ack", which is fire and forget. These all have their uses depending the specific needs of an application.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#144

Post by mattl »

nathanmarz wrote: post #120
NoraCodes wrote: post #2

I would argue that this is not "a Mastodon instance", since it is not running Mastodon - other than that, very very neat work! I'm excited for that "Source Code" link to be live :)

We call it a "Mastodon instance" because we implemented the entire Mastodon API ( https://docs.joinmastodon.org/api/ ). This is in addition to also implementing the ActivityPub API which Mastodon also implements for federation.

Mastodon-compatible would be better.

Mastodon is the name of a piece of software as well as an API, a website, etc.

Naming this stuff is hard but calling it a Mastodon instance would be more confusing.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#145

Post by dunk010 »

bo1024 wrote: post #82
polishdude20 wrote: post #61

"We spent nine person-months building our scalable Mastodon instance. " Nono, you can't say that when later on you say it's built on top of Rama. You literally spent 10 years building the framework to even make this. And yes, you built this in 10k lines of code but how many lines of code is Rama? This seems disingenuous.

Their point is to show off the power of Rama, I.e. it is possible to build such applications on top with little work.

Exactly, why are so many people missing this point. It's not "we built a narrow, tedious framework for knocking off Twitter clones", it's "We built a platform that turns data processing on its head and look in a couple of months you can clone Twitter just imagine what YOU can do with this."

I see parallels though to Datomic, where they turned the database inside out, co-located the app logic and data and indexes, etc. There are a bunch of great videos on YT about Datomic by Rich Hickey & co, worth a watch and I think shine a light on the approach here, too.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#146

Post by jitl »

This architecture seems very similar to existing offerings in the "in-memory data grid" category, like Apache Ignite and Hazelcast. I'm more familiar with Ignite (I built a toy Notion backend with it over a few afternoons in 2020).

The way Ignite works overall is similar. You make a cluster of JVM processes, your data partitioned and replicated across the cluster, and you upload some JARs of business logic to the cluster to do things. Your business logic can specify locality so it runs on the same nodes as the relevant data, which ideally makes things a lot faster compared to systems where you need to pull all your data across the wire from a DB. Like Rama, Ignite uses a Java API for everything, including serializing and storing plain 'ol java objects.

Ignite's architecture isn't focused on "ETL" into "PStates". Instead it's more about distributed "caches" of data. It does have streaming for ingestion (https://ignite.apache.org/docs/latest/data-streaming), but you can transactionally update the datastore directly (https://ignite.apache.org/docs/latest/key-value-api/transact...). It also has a "continuous query" feature for those reactive queries to retrieve data (https://ignite.apache.org/docs/latest/key-value-api/continuo...).

Rama's data-structure oriented PState index seems easier to work with than building indexes yourself on top of Ignite's KV cache, but Ignite also offers an SQL language, so you can insert your data into the KV cache however, add some custom SQL functions, and then accept more flexible SQL querying of your data compared to the very purpose-built PCache things, but still be able to do lower-level or more performance-oriented logic with data locality.

Anyways, if you like some of this stuff but want to use an existing, already battle-tested open source project, you can look for these "in-memory data grid", "distributed cache", kind of projects. There's a few more out there that have similar JVM cluster computing models.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#147

Post by beders »

polishdude20 wrote: post #61

"We spent nine person-months building our scalable Mastodon instance. " Nono, you can't say that when later on you say it's built on top of Rama. You literally spent 10 years building the framework to even make this. And yes, you built this in 10k lines of code but how many lines of code is Rama? This seems disingenuous.

No it is not disingenuous. They didn't built Rama to build a twitter clone.

And you can't take the "twitter engine" out of twitter and build other apps with it. A lot of it is custom built to fit the twitter data model.

Unlike - it seems - Rama.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#150

Post by jitl »

jedberg wrote: post #55
Pxtl wrote: post #25

I've seen many people describe frameworks like this - you know, first you have the slow back-end event-driven master database that you don't query live against, then you've got eventual-consistency flows against the various data-warehouses and data-stores and partitioned sharded databases in useful query-friendly layouts that you actually read live from... and I never see it clearly explained: how do you read a chang…

The short answer is write-through cache. You write the update directly to the cache closest to the user and into the eventually consistent queue. We did this at reddit. When you make a comment the HTML is rendered and put straight into the cache, and the raw text is put into the queue to go into the database. Same with votes. I suspect they do this client side now, which is now the closest cache to the user, but back…

Rama should bundle a write-through cache! Another in-memory JVM cluster thingamabob (Apache Ignite) used to propose write-through caching as it's primary selling point: https://ignite.apache.org/use-cases/in-memory-cache.html#:~:....

Or, maybe their pitch is that the streaming bits are so fast, you can just await the downstream commit of some write to a depot and it'll be as fast as a normal SQL UPDATE.

Post reply on HN