Yandex open-sources its exabyte-scale big data platform • Hacker News Board

Re: Yandex open-sources its exabyte-scale big data platform

#101

Post by dang » Thu, Mar 23, 2023, 9:28 PM UTC

medo-bear wrote: post #15

garbagecoder wrote: post #13

Earlier quoted context omitted.

[flagged]

nice head in the sand

Hey, could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for.

Also, it looks like you've been using HN primarily for political/national/ideological battle, and that's another (distinct) line at which we ban accounts—regardless of what you're battling for or against. Past explanations here: https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme....

If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

Re: Yandex open-sources its exabyte-scale big data platform

#102

Post by dang » Thu, Mar 23, 2023, 9:29 PM UTC

garbagecoder wrote: post #21

medo-bear wrote: post #15

Earlier quoted context omitted.

nice head in the sand

That's a hilarious response from a Russia apologist. You think telling other people they're naive about the CIA means you can ignore all the bad stuff about Russia but that's not how the world works and you are either paid to have say this or are doing it for free and I'm not sure which is worse. I try not to put myself in the hands of any spy agency. "Other team bad so my team good" is reasoning barely worthy of the…

Please don't respond to a bad comment by breaking the site guidelines yourself. That only makes things worse.

https://news.ycombinator.com/newsguidelines.html

Re: Yandex open-sources its exabyte-scale big data platform

#103

Post by ddorian43 » Fri, Mar 24, 2023, 12:18 PM UTC

gritukan wrote: post #76

g9yuayon wrote: post #58

The github repo has 5 million lines of C++ code (headers included), 1.6 million lines of C code, and even nearly 1 million lines of Scala + Java code. We'd need some serious docs to adopt this technology. The most interesting part of YT is Cypress. I'm particularly interested in how they make their master cluster horizontally scalable.

Historically, the master server of YTsaurus was a single RSM (replicated state machine) that contained all the meta-information about the cluster. This included the tree of the distributed filesystem, transactions, information about users and tables, placement of chunks, and much more. However, this approach proved to be non-scalable as the memory amount and throughput of the master server soon became insufficient. T…

> Therefore, we are currently working on a new technology called Sequoia, which stores information about the Cypress tree shape in horizontally scalable dynamic tables.

Why not just use a database for the metadata? Something that can be sharded and has transactions like YugabyteDB/Yandex-ydb/etc?

Re: Yandex open-sources its exabyte-scale big data platform

#104

Post by gritukan » Sat, Mar 25, 2023, 4:55 AM UTC

ddorian43 wrote: post #103

gritukan wrote: post #76

Earlier quoted context omitted.

Historically, the master server of YTsaurus was a single RSM (replicated state machine) that contained all the meta-information about the cluster. This included the tree of the distributed filesystem, transactions, information about users and tables, placement of chunks, and much more. However, this approach proved to be non-scalable as the memory amount and throughput of the master server soon became insufficient. T…

> Therefore, we are currently working on a new technology called Sequoia, which stores information about the Cypress tree shape in horizontally scalable dynamic tables. Why not just use a database for the metadata? Something that can be sharded and has transactions like YugabyteDB/Yandex-ydb/etc?

Our objects have complex semantics of changes so it seems challenging to implement the whole Cypress over k-v storage.

Storing Cypress nodes in the ad hoc RSMs and information about tree in k-v storage seems a good compromise that is both scalable and allowing to implement any functionality for objects efficiently.