Live data from Hacker News

Yandex open-sources its exabyte-scale big data platform

medium.com

81–90 of 104 posts

Re: Yandex open-sources its exabyte-scale big data platform

#81
post #35

[flagged]

Does anyone know how they used the slurs?

Looking at the OP comment before it was flagged, someone just did `s/slave/n****r/g` on the codebase.

BTW It's pretty ridiculous that American censorship makes it impossible to even call out and criticize racism like in the case of OP.

Re: Yandex open-sources its exabyte-scale big data platform

#82
post #67

How many companies exist that will utilise its full capabilities? My bet would be 50-100

Most people don't utilise the full capabilities of the tools they use.

Sure it would be overkill for a lot of applications, but so is redis, react.js, etc.

Re: Yandex open-sources its exabyte-scale big data platform

#84

It's my understanding that the original Yandex founders and tech left Russia several years ago. Is this new technology? Or is it stuff being extricated from the (now) Kremlin-controlled codebase?

It isn't new, in the article they state that it originated back in 2006. It's been in development since then.

Re: Yandex open-sources its exabyte-scale big data platform

#85

Earlier quoted context omitted.

> a covert invasion of wherever they'd swarm to. > an exiled Russian company. Either you are exhiled or you are invading, how can you be both?

For example, you wholeheartedly support the policy, but are not comfortable living under the sanctions.

My understanding of Exile is that you are rejected by the society you are exiled from, due to your actions or beliefs. Like profound disagreement with the policy.

I have certainly seen the kind of people you talk about, they support (or at least used to) Putin, but don't want their kids to live in Russia. I would call them more like emigrants of convenience.

Re: Yandex open-sources its exabyte-scale big data platform

#86
post #67

How many companies exist that will utilise its full capabilities? My bet would be 50-100

My guess it could be successfully used for relatively small (terabyte-scale) workloads.

It's the same as when people use k8s not utilizing its full capabilities, only to be able to massively scale up when needed.

Re: Yandex open-sources its exabyte-scale big data platform

#87
post #38
post #26

There is an interesting take that Russian part of the Yandex group opensources as much as possible, in order for the overseas companies of the group to leverage technologies without legal or financial ties with Russia. For me this seems very plausible, as for the last year they first did everything to distance from anything related to politics (e. g. they sold their news and their blogging platform to the basically s…

Yandex is incorporated in the Netherlands and the founders live in Israel. I think they will create several spin-off open source companies (like ClickHouse Inc.) outside Russia to continue doing B2B business with the outside world.

It doesn't matter where the company is incorporated -- when Kremlin wants the access to the data Yandex cannot say no. One cannot operate in Russia and not play by Kremlin rules, especially media companies.

Re: Yandex open-sources its exabyte-scale big data platform

#88

Earlier quoted context omitted.

You can take Yandex out of Russia... but you can not take Russia out of Yandex. Admittedly, the same could probably be said about any other $big_search:$nuclear_power pair. It's just that the other companies either can't or have no reason for exile. And so I suspect Yandex leaving it's home turf would essentially be a covert invasion of wherever they'd swarm to. Somehow, money tells me UK would be a likely target. So…

> a covert invasion of wherever they'd swarm to. > an exiled Russian company. Either you are exhiled or you are invading, how can you be both?

Deception. Moving a piece on a game board should have more than one effect. One should preferably make it so that the most obvious effect (exile) is not the one that's actually the most valuable in the long term (taking root in adverse country).

Bonus "inception" points if you can make the adversary believe that you did it because they forced you to (sanctions).

Re: Yandex open-sources its exabyte-scale big data platform

#89
Hello! I work at YT and would like to answer a question that was asked in a flagged thread about the comparison between YT and Hive and Zookeeper.

Both Cypress and Zookeeper are fault-tolerant distributed hierarchical filesystems that can be used for distributed coordination, but Cypress has much richer functionality.

Recall that Zookeeper's data model is just a tree consisting of homogeneous nodes that can be either ephemeral or persistent, along with a set of sessions that control the lifetime of ephemeral nodes. This simple model allows to implement multiple primitives of distributed synchronization, such as leader election, exactly-once queue processing, or two-phase commits. However, it is not always easy to integrate Zookeeper with third-party systems. For example, if you want to elect a leader via Zookeeper and use it to insert data into a database, it is mandatory that the instance remains the leader during the commit into the database, which is not easy to implement without races or some additional assumptions. In YTsaurus, transactions permeate our entire system. You can start a transaction and acquire an exclusive lock at some Cypress node (which is a way to make a leader election), and after that, the transaction becomes the leader lease. You can then modify Cypress, run MapReduce or YQL operations using the transaction as a prerequisite, lock some files and tables in the same transaction, and do many other things. Currently, we are working on the ability to use Cypress locks as prerequisites for dynamic table commits. There are many other features in Cypress that are not implemented in Zookeeper, such as symlinks, automatic expiration of unused nodes, and many others. Moreover, Cypress can be sharded using Portals about which I wrote in a previous comment, so this filesystem is scalable unlike Zookeeper. Even without sharding a single primary master of YTsaurus can hold tens of gigabytes of metadata of Cypress while Zookeeper state size is limited with hundreds of megabytes accoring to etcd vs Zookeeper comparision [1].

One major disadvantage of Cypress compared to Zookeeper is the lack of watches, so all changes tracking should be done via short polling. The good news is that Cypress is well-optimized for read queries with the possibility to read from followers and from multiple threads, so this is not a big problem. In the meantime, we are considering the possibility of adding some kind of watches to Cypress.

The big difference between Cypress and Zookeeper is the replicated state machine implementation. With all due respect to Zookeeper developers, Zookeeper was implemented over 15 years ago when the world of distributed algorithms was different. Today we see that ZAB (the consensus algorithm used in Zookeeper) has some shortcomings in failover speed and stability. There are multiple reports of Zookeeper being unstable under heavy load. In YTsaurus, we use an in-house library called Hydra for RSM implementation. This is our consensus algorithm very similar to RAFT that has proven itself to be both efficient and fault-tolerant. We use Hydra for master servers, clock servers, and tablet cells (RSMs that store data in dynamic tables). I even had an idea to implement a Zookeeper API using Hydra both to simplify migration to YTsaurus and check Hydra performance and correctness via multiple tests implemented for Zookeeper (Jepsen, for instance), but did not have enough time to finish this project.

This comment is already quite long, so I will write about the YT vs Hive comparison in another comment later on.

[1] -- https://etcd.io/docs/v3.3/learning/why/

Post reply on HN