Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced the cost of building Mastodon at Twitter-scale by 100x

blog.redplanetlabs.com

Post reply on HN
Print view View original

231–240 of 376 posts

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#231

Post by WheelsAtLarge »

RomanPushkin wrote: post #163

> ...10k lines of code. This is 100x less code than the ~1M lines Twitter I wish I didn't see this comparison, which is not fair at all. Everyone in their right mind understands that the number of features is much less, that's why you have 10k lines. Add large-scale distributed live video support at the top of that, and you won't get any close to 10k lines. It's only one of many many examples. I really wish you compa…

We see this type of post regularly. Something like, "How I built a better clone by myself in a month." Well, no, usually it's just a bare skeleton with the least amount of functionality. Not only that, the software is the least of the functionality. The organizational structure around the app is what matters most to keep it going. It's an attention seeking ploy and the whole thing usually disappears real quick.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#232

Post by hot_gril »

codetrotter wrote: post #166
sitzkrieg wrote: post #135

Earlier quoted context omitted.

yea but does hn have any client side js?

Yeah, a very small amount so that clicking the upvote button does not need to reload the whole page

Can confirm, HN relies on client-side JS for voting and collapsing, but view/post/edit/delete don't need it.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#233

Post by leonidasv »

endisneigh wrote: post #215
sdwr wrote: post #207

Earlier quoted context omitted.

In a typical architecture, the DB stores data, and the backend calls the DB to make updates and compile views. Here, the "views" are defined formally (the P-states), and incrementally, automatically updated when the underlying data changes. Example problem: Get a list of accounts that follow account 1306 "Classic architecture": - Naive approach. Search through all accounts follow lists for "1306". Super slow, scales…

I read their post and honestly it’s not really that much different than just materialized views in a regular database plus async jobs to do the long running tasks. It’s a ridiculous amount of fluff to describe that. Not to mention it’s proprietary and only supports the JVM and doesn’t integrate with the tons of tooling designed about RDBMS unless you stream everything to them, defeating the purpose. What really irks…

IIUIC, the most significant difference from a materialized view is that the Rama infrastructure recompute only the changed data by checking the relationship between fields, while a traditional materialized view recomputes the whole table?

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#235

Post by skybrian »

nathanmarz wrote: post #74
skybrian wrote: post #54

It sounds like interesting technology for someone, but I wonder more about scaling down. What does a developer instance running on a laptop look like?

Great question. There's actually two ways to look at this: what does it look like to run Rama in a unit test environment, and what does it look like to run a small-scale single-node Rama application in production? For the former, Rama has a class called "InProcessCluster" that works identically to a real cluster. It enables Rama applications to be tested and experimented with end-to-end. There's an example of this in…

Interesting. How about running the cloud? I'm thinking of the many ways someone who wants to start a blog could install Ghost. [1]

[1] https://ghost.org/docs/install/

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#237

Post by mkl »

fragmede wrote: post #223
softwaredoug wrote: post #219

Earlier quoted context omitted.

I can imagine this being really useful from the ground up. Because it looks like it wants to be the source of truth, with different views on the data. It’s hard to imagine it for a complex legacy application without having lots of added complexity. It wants to be the unifying programming model for the application. It would seem like running with two RDMS sources of truth simultaneously. It’s like the xkcd “there are…

That's xkcd 927. 9, which is 3^2, and 27, which is 3^3. Or 900 is Yoda's age, and 27 which is the 27 club of musicians who committed suicide.

Most members of the 27 Club didn't die by suicide: https://en.wikipedia.org/wiki/27_Club

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#238

Post by nathanmarz »

chubot wrote: post #193
nathanmarz wrote: post #176

Earlier quoted context omitted.

When you step back and consider the incredible amount of manpower and resources that have been put into these applications, it's amazing how buggy these applications are. To put it simply, they're buggy because the underlying infrastructure and techniques used to build them are so complex that the implementation is beyond the realm of human understanding. The way applications are built, and have been built since befo…

Yes, I 100% agree with you. I would like something like this to succeed, and agree the problem is real. But what are the tradeoffs? There's nothing that comes with 100x benefit with no tradeoffs (side note: I worked on Google Code for a short while in 2008, concurrent with Github's founding ... I think Github moved a lot faster in a large part because they weren't dealing with distributed systems at first -- they had…

Rama is a much broader platform than a database, so the consistency semantics you get depend on how you use it. When using Rama, you're not mutating indexes directly like you do with a database, but adding source data that then gets materialized into any number of indexes.

You get read-after-write consistency for any PStates in a streaming ETL colocated with the depot you appended to. This is if you do the depot append with "full acking", which coordinates its response with the completion of colocated streaming ETLs. If you append at a lower level of acking, then you get eventual consistency on those PStates at the benefit of lower latency appends.

Microbatching is always eventually consistent as processing is asynchronous and uncoordinated to depot appends. Microbatching is higher thorughput than streaming and has simpler fault-tolerance semantics.

You'll be able to read a lot more about this when we release the docs next week.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#239

Post by mark_l_watson »

nathanmarz wrote: post #77
mping wrote: post #72

Congrats on the (kinda) launch. I was curious to see what you guys were up to. The blog post is pretty detailed, and with good insights. Reducing modern app development complexity to mixing data structures sounds like a good abstraction. I'm sure you thought really hard about the building blocks of Rama and you know your problems better than most of the hn crowd. Now, the really hard part becomes selling. If companie…

Rama is written in Clojure :)

Nathan, we guessed that it was written in Clojure :-)

Interesting to finally see the announcement.

Re: We reduced the cost of building Mastodon at Twitter-scale by 100x

#240

Post by NoahTheDuke »

nathanmarz wrote: post #217
NoahTheDuke wrote: post #170

Congrats! This looks super cool. Are there any plans for exposing a Clojure API? Given that it's implemented in Clojure, seems like it would be a natural fit. Interop with Java is nice but can be cumbersome compared to the more natural calling conventions and idioms (threading macros instead of `..` builder patterns, etc).

Answered this in another comment: https://news.ycombinator.com/item?id=37138526

Thanks.
Post reply on HN