Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced the cost of building Mastodon at Twitter-scale by 100x

blog.redplanetlabs.com

Post reply on HN
Print view View original

41–50 of 376 posts

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#41

Post by dang »

Kiro wrote: post #31
dang wrote: post #20

Earlier quoted context omitted.

Thank - I've changed the title to be consistent with what the article says.

That's not the title of the article and also not what the article says. I would be really pissed if you editorialized the title of my article like that.

I'm happy to correct it if anyone suggests a better one. The intention is to find a neutral title that accurately reflects what the article itself is saying.

We've learned that when an article's original title generates complaints like https://news.ycombinator.com/item?id=37137317, the thread is likely to get derailed by shallow arguing about the title. It's in both the author's interest and the community's for us to nip that in the bud by (1) putting an accurate and neutral title at the top (preferably using representative language from the article itself), and (2) marking the title complaint offtopic since it no longer applies. These steps nudge the thread toward discussing the article's content rather than merely its title.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#42

Post by softwaredoug »

It’s a massive ask, even if the platform was 100x better, for all developers to give up every programming language and database they’ve ever used to depend on a startups closed source platform for all functionality.

It’s hard enough trusting Google or Amazons cloud offerings won’t change.

It seems that’s what they’re proposing right? What am I missing?

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#44

Post by [deleted] »

Pxtl wrote: post #25

I've seen many people describe frameworks like this - you know, first you have the slow back-end event-driven master database that you don't query live against, then you've got eventual-consistency flows against the various data-warehouses and data-stores and partitioned sharded databases in useful query-friendly layouts that you actually read live from... and I never see it clearly explained: how do you read a chang…

[deleted]

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#45

Post by sixo »

Pxtl wrote: post #25

I've seen many people describe frameworks like this - you know, first you have the slow back-end event-driven master database that you don't query live against, then you've got eventual-consistency flows against the various data-warehouses and data-stores and partitioned sharded databases in useful query-friendly layouts that you actually read live from... and I never see it clearly explained: how do you read a chang…

I imagine you get some UUID back from your write, and effectively "block" until you see it committed to the event stream. The intent of such a system is certainly for the read-after-write latency to be not much longer than a traditional RDBMS. (This is roughly what the RDBMS is doing under the hood anyway.) Probably you can isolate latency-critical paths so they don't get stuck behind big stream processing jobs.

The advantage of the overall architecture is that nearly all application functionality (for something like a social network) can tolerate much higher latency than an RDBMS, so you really want to have architectural building blocks that let you actually use this headroom.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#47

Post by chc4 »

nathanmarz wrote: post #37
_dwt wrote: post #14

Hmmm, "Rama is programmed entirely with a Java API – no custom languages or DSLs" according to the landing page, but this sure looks like an embedded DSL for dataflow graphs to me - Expr and Ops everywhere. Odd angle to take.

I consider "DSL" as something that's its own language with it's own lexer and parser, like SQL. The Rama API is just Java – classes, interfaces, methods, etc. Everything you do in Rama, from defining indexes, performing queries, or writing dataflow ETLs, is done in Java.

This is usually referred to as an "embedded DSL" - you have a DSL embedded in a normal programming language using its first class constructs.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#48

Post by dustingetz »

Summarizing, now edited down with some editorializing for clarity:

What is it? build web-scale reactive backends with an expressive java dataflow API. Instead of a database you develop your own custom app-specific indexes which are reactive, distributed and durable. It's like event sourcing and materialized views but integrated in a linearly scalable way.

> I cannot emphasize enough how much interacting with indexes as regular data structures instead of magical “data models” liberates backend programming

> It allows for true incremental reactivity from the backend up through the frontend. ... enable UI frameworks to be fully incremental instead of doing expensive diffs to find out what changed.

Ok, so in my mind I am positioning this against Materialized / differential dataflow, whose key primitive is a efficient streaming incremental join that works across very large relational tables. Materialized makes SQL reactive, Rama gives you a java dataflow DSL for developing purpose-built reactive database indexes.

How it works? 4 concepts: Depot, ETLs, PState, Query

Depots: "distributed, durable, and replicated logs of data." [Event streams?] "like Kafka except integrated" "All data coming into Rama comes in through depot appends."

ETLs: data arrives via depots, and is ETLed to PStates via "a Java dataflow API for coding topologies that is extremely expressive". "Most of the time spent programming Rama is spent making ETLs."

PStates seem like reactive data structures that are also durable/replicated, these are meant to supersede your database and indexes, letting you build custom purpose-built indexes that contain 100M elements:

> “partitioned states” are how data is indexed in Rama ... Unlike existing databases, which have rigid indexing models (e.g. “key-value”, “relational”, “column-oriented”, “document”, “graph”, etc.), PStates have a flexible indexing model. In fact, they have an indexing model already familiar to every programmer: data structures. A PState is an arbitrary combination of data structures. ... nested data structures can efficiently contain hundreds of millions of elements. For example, a “map of maps” is equivalent to a “document database”, and a “map of subindexed sorted maps” is equivalent to a “column-oriented database”. Any [composition] is valid – e.g. you can have a “map of lists of subindexed maps of lists of subindexed sets”.

Query: once you develop PStates to aggregate relevant data into a custom index of the right ... shape?, query seems sorta like GraphQL selectors over your custom index:

> Queries in Rama take advantage of the data structure orientation of PStates with a “path-based” API that allows you to concisely fetch and aggregate data from a single partition

> “query topologies” ... real-time distributed querying and aggregation over an arbitrary collection of PStates. These are the analogue of “predefined queries” in traditional databases, except programmed via the same Java API as used to program ETLs and far more capable.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#49

Post by nathanmarz »

LeifCarrotson wrote: post #12

> How is it possible that we’ve reduced the cost of building scalable applications by multiple orders of magnitude? > You can begin to understand this by starting with a simple observation: you can describe Mastodon (or Twitter, Reddit, Slack, Gmail, Uber, etc.) in total detail in a matter of hours. It has profiles, follows, timelines, statuses, replies, boosts, hashtags, search, follow suggestions, and so on. It doe…

Building Twitter/Mastodon *not at scale* isn't that hard and certainly doesn't take 200 person-years. Building it *at scale* is a completely different story. Remember the fail-whale? That was years of Twitter struggling to scale their product.

That said, as we described in the post our implementation of Mastodon is less code than Mastodon's official implementation. So not only is Rama orders of magnitude more efficient for building applications at scale, it's also much faster for building first versions of an application.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#50

Post by gfodor »

chc4 wrote: post #47
nathanmarz wrote: post #37

Earlier quoted context omitted.

I consider "DSL" as something that's its own language with it's own lexer and parser, like SQL. The Rama API is just Java – classes, interfaces, methods, etc. Everything you do in Rama, from defining indexes, performing queries, or writing dataflow ETLs, is done in Java.

This is usually referred to as an "embedded DSL" - you have a DSL embedded in a normal programming language using its first class constructs.

Yep the original term DSL was for custom languages, the eventual introduction of using it for these kinds of literate APIs was done later. Using it in the original way unqualified is fine imo.
Post reply on HN