Thank you for sharing this! I am actually really impressed. Well done! Good work! There's lots of interesting lessons and knowledge in the design of this platform. I also like how you've decided to use Java as your API rather than Clojure. I hope you're not discouraged by HN's reaction to your hard work. Don't be discouraged!
We reduced the cost of building Mastodon at Twitter-scale by 100x
361–370 of 376 posts
Re: We reduced the cost of building Mastodon at Twitter-scale by 100x
#362But I won't ever consider investing in it unless it's some form of open-source. It's too much of a risk to have a closed-source core.
Re: We reduced the cost of building Mastodon at Twitter-scale by 100x
#363Earlier quoted context omitted.
In a typical architecture, the DB stores data, and the backend calls the DB to make updates and compile views. Here, the "views" are defined formally (the P-states), and incrementally, automatically updated when the underlying data changes. Example problem: Get a list of accounts that follow account 1306 "Classic architecture": - Naive approach. Search through all accounts follow lists for "1306". Super slow, scales…
I read their post and honestly it’s not really that much different than just materialized views in a regular database plus async jobs to do the long running tasks. It’s a ridiculous amount of fluff to describe that. Not to mention it’s proprietary and only supports the JVM and doesn’t integrate with the tons of tooling designed about RDBMS unless you stream everything to them, defeating the purpose. What really irks…
How about you go and implement a Mastodon server to their level of feature parity, and tell us how much effort and how many lines of code it takes?
I really don't appreciate this kind of fluffy, insubstantial, overly dismissive non-content on HN.
Re: We reduced the cost of building Mastodon at Twitter-scale by 100x
#364Re: We reduced the cost of building Mastodon at Twitter-scale by 100x
#365Earlier quoted context omitted.
Meh. By this definition all libraries expose an "embedded DSL" — their API. I'm honestly not sure this is a useful definition.
Whether you like it or not; internal DSLs became a thing with Ruby back in the day. And these days things like Kotlin also lend themselves pretty well to creating internal DSLs. Java is not ideal for this. Kotlin and Ruby have a few syntax features that make it very easy.
Re: We reduced the cost of building Mastodon at Twitter-scale by 100x
#366Earlier quoted context omitted.
Whether you like it or not; internal DSLs became a thing with Ruby back in the day. And these days things like Kotlin also lend themselves pretty well to creating internal DSLs. Java is not ideal for this. Kotlin and Ruby have a few syntax features that make it very easy.
What's the distinguishing difference between a regular library and an embedded DSL?
If you need an example, Kotlin uses a nice internal DSL for HTML where you write things like
Div {
P {
+"Hello world"
}
}
This is valid Kotlin that happens to construct a dom tree. There is no magic going on here; just usage of a few syntax features of the language. Div and p here are normal functions that take a receiver block as the last parameter that receive an instance of the element they are creating. The + in front of the string is function with a header like this in the implementation of that element. operator fun String.unaryPlus()
The same code in Java would be a lot messier because of the lack of syntactical support for this. You basically end up with a lot of method chaining, semicolumns and their convoluted lamda syntax. The article has a few examples that would look a lot cleaner if you'd rewrite them in Kotlin.Re: We reduced the cost of building Mastodon at Twitter-scale by 100x
#367Earlier quoted context omitted.
Are you sure about that. With things like twitter, the ui is not the hard part. Things like moderation are the secret sauce. All the corner cases and support for devopsy stuff likely account for a lot. Routing to specific instances for celebrities and such.
Nathan worked at Twitter so while he might be wrong, I don't think it's reasonable to assume he's just naive http://nathanmarz.com/blog/leaving-twitter.html .
I originally read it as it's easy to clone twitter.
My response is it's easy enough to build a micro-blogging platform/service. it's all the other shit like moderation, regulatory/legal compliance, making a profit, keeping advertisers happy, etc that's hard.
Re: We reduced the cost of building Mastodon at Twitter-scale by 100x
#368Earlier quoted context omitted.
Yeah, a very small amount so that clicking the upvote button does not need to reload the whole page
Upvoting does work javascript disabled, it upvotes and reloads the page. It approximately keeps your position by jumping the message anchor.
Unfortunately the js used is very minimal. And if the upvote fails, there is no graphical indication that it failed to upvote.
Re: We reduced the cost of building Mastodon at Twitter-scale by 100x
#369I do C++ backend work in a non-web industry and this entire post is Greek to me. Even though this is targeted at developers, you need a better pitch. I get "we did this 100x faster" but the obvious followup question is "how" but then the answer seems to be a ton of flow diagrams with way too many nodes that tell me approximately nothing and some handwaving about something called P-States that are basically defined to…
In a typical architecture, the DB stores data, and the backend calls the DB to make updates and compile views. Here, the "views" are defined formally (the P-states), and incrementally, automatically updated when the underlying data changes. Example problem: Get a list of accounts that follow account 1306 "Classic architecture": - Naive approach. Search through all accounts follow lists for "1306". Super slow, scales…
Have you considered using that?
Re: We reduced the cost of building Mastodon at Twitter-scale by 100x
#370> ...10k lines of code. This is 100x less code than the ~1M lines Twitter I wish I didn't see this comparison, which is not fair at all. Everyone in their right mind understands that the number of features is much less, that's why you have 10k lines. Add large-scale distributed live video support at the top of that, and you won't get any close to 10k lines. It's only one of many many examples. I really wish you compa…
> Add large-scale distributed live video support at the top of that, and you won't get any close to 10k lines. But Twitter isn't, and was never, about live video support: this is pure feature creep and that's how you get headcount inflation and a company that can be run for 17 years without making profit (AKA terrible business). > When I worked in Statuspage, we had support of 50-100k requests per second Having serve…
I would love to see a blog post about your experience!