Who cares. Mastodon was/is destined to fail. Trigger happy mods ban you from a server, then you're banned from a bunch.
skill issue
We reduced the cost of building Mastodon at Twitter-scale by 100x
221–230 of 376 posts
Re: We reduced the cost of building Mastodon at Twitter-scale by 100x
#222The "N bots posting X times/second" isn't a very meaningful statistic. A system's reliability is mostly characterized by its performance under stress.
Re: We reduced the cost of building Mastodon at Twitter-scale by 100x
#223Earlier quoted context omitted.
We're actually not asking anyone to give up anything. First off, it has a simple integration API (which you'll be able to see the details of next week) that allows it to seamlessly integrate with any other backend tool (databases, monitoring systems, queues, etc.). So Rama can be incrementally introduced into any existing architecture. Second, Rama has a pure Java API and is not a bespoke language. So no new language…
I can imagine this being really useful from the ground up. Because it looks like it wants to be the source of truth, with different views on the data. It’s hard to imagine it for a complex legacy application without having lots of added complexity. It wants to be the unifying programming model for the application. It would seem like running with two RDMS sources of truth simultaneously. It’s like the xkcd “there are…
That's xkcd 927.
9, which is 3^2, and 27, which is 3^3. Or 900 is Yoda's age, and 27 which is the 27 club of musicians who committed suicide.
Re: We reduced the cost of building Mastodon at Twitter-scale by 100x
#224Earlier quoted context omitted.
In a typical architecture, the DB stores data, and the backend calls the DB to make updates and compile views. Here, the "views" are defined formally (the P-states), and incrementally, automatically updated when the underlying data changes. Example problem: Get a list of accounts that follow account 1306 "Classic architecture": - Naive approach. Search through all accounts follow lists for "1306". Super slow, scales…
I read their post and honestly it’s not really that much different than just materialized views in a regular database plus async jobs to do the long running tasks. It’s a ridiculous amount of fluff to describe that. Not to mention it’s proprietary and only supports the JVM and doesn’t integrate with the tons of tooling designed about RDBMS unless you stream everything to them, defeating the purpose. What really irks…
This is all armchair for me, but I think they have containers and sharding built in as well, which is the other half of the puzzle when it comes to scaling.
Re: We reduced the cost of building Mastodon at Twitter-scale by 100x
#225Earlier quoted context omitted.
Messaging apps often have a checkmark to indicate the message actually went to the server, and maybe another checkmark to indicate it was received on the other end. Maybe HN needs an icon indicating that your vote went through.
Make the arrows grey to indicate the click registered, make them disappear to indicate the server successfully registered the vote?
Yeah, it's easy enough that I was able to do it in the web inspector in a minute (artificial 1s network delay added): https://s11.gifyu.com/images/ScPMI.gif
Re: We reduced the cost of building Mastodon at Twitter-scale by 100x
#226I do C++ backend work in a non-web industry and this entire post is Greek to me. Even though this is targeted at developers, you need a better pitch. I get "we did this 100x faster" but the obvious followup question is "how" but then the answer seems to be a ton of flow diagrams with way too many nodes that tell me approximately nothing and some handwaving about something called P-States that are basically defined to…
In a typical architecture, the DB stores data, and the backend calls the DB to make updates and compile views. Here, the "views" are defined formally (the P-states), and incrementally, automatically updated when the underlying data changes. Example problem: Get a list of accounts that follow account 1306 "Classic architecture": - Naive approach. Search through all accounts follow lists for "1306". Super slow, scales…
So... at a high level, early React for data? In other words, letting a framework manage update dependency graph tracking, and then cascading updates through its graph in an optimized manner to enhance performance?
Obviously, with tons of implementation difficulties and details, and not actual graph structures, but as a top level analogy.
Re: We reduced the cost of building Mastodon at Twitter-scale by 100x
#227Earlier quoted context omitted.
> Don't even want to wait for an HTTP roundtrip for some of these, e.g. "liking" a post should fill in the heart icon or whatever instantly. HN does this, and on slow days, about half of my upvotes don't go through.
yea but does hn have any client side js?
[deleted]
Re: We reduced the cost of building Mastodon at Twitter-scale by 100x
#228> ...10k lines of code. This is 100x less code than the ~1M lines Twitter I wish I didn't see this comparison, which is not fair at all. Everyone in their right mind understands that the number of features is much less, that's why you have 10k lines. Add large-scale distributed live video support at the top of that, and you won't get any close to 10k lines. It's only one of many many examples. I really wish you compa…
> Add large-scale distributed live video support at the top of that, and you won't get any close to 10k lines.
But Twitter isn't, and was never, about live video support: this is pure feature creep and that's how you get headcount inflation and a company that can be run for 17 years without making profit (AKA terrible business).
> When I worked in Statuspage, we had support of 50-100k requests per second
Having served 150kqps in the past as part of a very small team (3 back-end eng.), this isn't necessarily as big of a deal as you make it sound: it mostly depends on your workload and whether or not you need consistency (or even persistence at all) in your data.
In practice, building scalable system is hard mostly because it's hard to get the management forgot their vanity ideas that go against your (their, actually) system's scalability.
Re: We reduced the cost of building Mastodon at Twitter-scale by 100x
#229Is this just me, or does the code in the post feel like they've implemented what should have been a new programming language on top of Java?
Their "variables" have names that you have to keep as Java strings and pass to random functions. If you want composable code, you don't declare a function, you call .macro(). For control flow and loops, you don't use if and for, but a weird abstraction of theirs.
I feel like this code could have been a lot simpler if it was written in a specialized language (or a mainstream language with a specialized transpiler and/or Macro capabilities.)
I'd quote the old adage about every big program containing a slow and buggy implementation of Common Lisp, but considering that this thing is written in Clojure, the authors have probably heard it before.
Re: We reduced the cost of building Mastodon at Twitter-scale by 100x
#230Earlier quoted context omitted.
I read their post and honestly it’s not really that much different than just materialized views in a regular database plus async jobs to do the long running tasks. It’s a ridiculous amount of fluff to describe that. Not to mention it’s proprietary and only supports the JVM and doesn’t integrate with the tons of tooling designed about RDBMS unless you stream everything to them, defeating the purpose. What really irks…
This is all armchair for me, but I think they have containers and sharding built in as well, which is the other half of the puzzle when it comes to scaling.
Yes, but there are plenty of NewSQL that support views and offer all of that too. Yugabyte, Cockroach, TiDB and that’s just off the top of my head and open source. If we count proprietary then you have Fauna, Cloud Spanner and more I’m sure.