Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced the cost of building Mastodon at Twitter-scale by 100x

blog.redplanetlabs.com

Post reply on HN
Print view View original

301–310 of 376 posts

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#301

Post by andrenotgiant »

leonidasv wrote: post #233
endisneigh wrote: post #215

Earlier quoted context omitted.

I read their post and honestly it’s not really that much different than just materialized views in a regular database plus async jobs to do the long running tasks. It’s a ridiculous amount of fluff to describe that. Not to mention it’s proprietary and only supports the JVM and doesn’t integrate with the tons of tooling designed about RDBMS unless you stream everything to them, defeating the purpose. What really irks…

IIUIC, the most significant difference from a materialized view is that the Rama infrastructure recompute only the changed data by checking the relationship between fields, while a traditional materialized view recomputes the whole table?

incremental view maintenance is the database equivalent of: "recompute only the changed data by checking the relationship between fields,"

Oracle has decent support for incrementally updated materialized views, redshift has some too. Materialize.com is an entire snowflake-like platform built around incrementally maintained materialized views.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#302

Post by tacotacotaco »

doctorpangloss wrote: post #241
nathanmarz wrote: post #168

Earlier quoted context omitted.

We're comparing just to the original consumer product, which is about the same as Mastodon is today. That's why we said "original consumer product" and not "Twitter's current consumer product". Mastodon actually has more features than the original Twitter consumer product like hashtag follows, global timelines, and more sophisticated filtering/muting capabilities. Some people argue it's not so expensive to build a sc…

I work in gaming, so I cannot speak to your specific experiences. Entity Component Systems are extremely performant, really good science, and shipping in middlewares like Unity. However, in order to ship an ECS game, in my experience, you have to have already made your whole game first in a normal approach, in order to have everything be fully specified sufficiently that you can correctly create an ECS implementation…

I only hobby in game dev. I have read some basics of ECS and IMO it seems a lot more intuitive than the “normal approach”. I think this is a case of using familiar tools we are comfortable with, not necessarily better. Unity’s ECS implementation is a hot mess though. I am looking forward to seeing what Bevy delivers as it matures.

I agree that copying an existing product will be easier and is usually cheaper and more performant because you can leverage your competitor’s R&D and lessons learned. I presume this is why some tech companies’ product lines are full of clones.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#303

Post by danbolt »

doctorpangloss wrote: post #241
nathanmarz wrote: post #168

Earlier quoted context omitted.

We're comparing just to the original consumer product, which is about the same as Mastodon is today. That's why we said "original consumer product" and not "Twitter's current consumer product". Mastodon actually has more features than the original Twitter consumer product like hashtag follows, global timelines, and more sophisticated filtering/muting capabilities. Some people argue it's not so expensive to build a sc…

I work in gaming, so I cannot speak to your specific experiences. Entity Component Systems are extremely performant, really good science, and shipping in middlewares like Unity. However, in order to ship an ECS game, in my experience, you have to have already made your whole game first in a normal approach, in order to have everything be fully specified sufficiently that you can correctly create an ECS implementation…

I can’t go into specifics due to NDAs, but I’ve been a developer on a game that was designed and programmed with the EnTT library from the ground up. [1]

I don’t know if I’d suggest ECS for every team as you need the right tooling, culture, and leadership to pull it off. That said, I think the paradigm has unrecognized benefits when it comes to greenfield gameplay programming.

Or, your data structures become a way for sub-teams to communicate and share state without stepping on one another’s toes. A group handling pathfinding and a different one handling scenario/mission logic can both pay attention to the position data without needing to be highly coupled. It becomes easy to “just add a system” or “an extra component” to build on existing functionality.

The Fred Brooks quote about “showing me your tables” comes to mind.

[1] https://www.minecraft.net/en-us/about-legends

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#307

Post by cormacrelf »

sixo wrote: post #244
endisneigh wrote: post #215

Earlier quoted context omitted.

I read their post and honestly it’s not really that much different than just materialized views in a regular database plus async jobs to do the long running tasks. It’s a ridiculous amount of fluff to describe that. Not to mention it’s proprietary and only supports the JVM and doesn’t integrate with the tons of tooling designed about RDBMS unless you stream everything to them, defeating the purpose. What really irks…

The difference is that the materialized-view logic lives naturally in the application code; there's no step where they go out of the DB to do computations and then reinsert. Once SQL materialized views aren't enough, you might do this by replicating your database into Kafka, implementing logic in Flink or something, and reinserting into the same DB/Elasticsearch/etc. Very common architecture. (Writ small, could also…

You can do all of this with https://materialize.com, and you don’t need to write it in Java. Just connect it to a Postgres instance and start creating materialised views using SQL. These views then auto update. So much so, that you can create a view for the top 10 of something, and let it sit there as the list updates. Otherwise just use normal select statements from your views using any Postgres client.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#308

Post by stefcoetzee »

nathanmarz wrote: post #139
fiddlerwoaroof wrote: post #134

Earlier quoted context omitted.

Will there be a first-party Clojure wrapper? Or, would the expectation be that users would use Java interop?

We're not releasing one as we don't have the bandwidth right now to maintain and document another API. That said, making a Clojure wrapper around the Java API should be pretty easy.

And hopefully, simple, too! ;)

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#309

Post by anothernewdude »

RomanPushkin wrote: post #163

> ...10k lines of code. This is 100x less code than the ~1M lines Twitter I wish I didn't see this comparison, which is not fair at all. Everyone in their right mind understands that the number of features is much less, that's why you have 10k lines. Add large-scale distributed live video support at the top of that, and you won't get any close to 10k lines. It's only one of many many examples. I really wish you compa…

It takes more self-control and effort to reduce the number of features to the ones that matter. Twitter having more features is a liability, not a benefit.

> Add large-scale distributed live video support at the top of that,

Why? For the love of all that is good and efficient, why? Why not have a separate platform for that? Or link to a different federated video service? Why does every platform need to do all the things?

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#310

Post by mal-2 »

nathanmarz wrote: post #120
NoraCodes wrote: post #2

I would argue that this is not "a Mastodon instance", since it is not running Mastodon - other than that, very very neat work! I'm excited for that "Source Code" link to be live :)

We call it a "Mastodon instance" because we implemented the entire Mastodon API ( https://docs.joinmastodon.org/api/ ). This is in addition to also implementing the ActivityPub API which Mastodon also implements for federation.

That doesn't match the way people use the term though. Pleroma and Akkoma implement the Mastodon API but wouldn't be called Mastodon instances since they aren't running Mastodon.
Post reply on HN