How many companies exist that will utilise its full capabilities? My bet would be 50-100
Why do you think so? How many companies use Apache Hadoop?
Yandex open-sources its exabyte-scale big data platform
71–80 of 104 posts
Re: Yandex open-sources its exabyte-scale big data platform
#72Funny to see 0 of 50 comments on a HN tech article being tech-related...
It’s not technically interesting since a lot of big data solutions already solve this problem so I guess only the geopolitics are left.
I doubt about "a lot". Also, "already solve" does not mean "solve better" or even "good enough".
It would be very interesting to see some in-depth comparisons with already-existing open source technology (like Hadoop, Hive, Iceberg, ZooKeeper) to get a sense of when and where YT could be more effective.
Re: Yandex open-sources its exabyte-scale big data platform
#73Funny to see 0 of 50 comments on a HN tech article being tech-related...
Agreed, but this is presumably also an absolutely massive project hardly anyone here has even used before. So it's not surprising that there are no big tech insights on the day of the release. An `scc` printout might be interesting, but any in-depth analysis is going to take a long time.
Re: Yandex open-sources its exabyte-scale big data platform
#74[flagged]
It was used for filter lists in search engine. I think you understand, that google is using the same exact technique. It's just haven't been leaked
Re: Yandex open-sources its exabyte-scale big data platform
#75Funny to see 0 of 50 comments on a HN tech article being tech-related...
It’s not technically interesting since a lot of big data solutions already solve this problem so I guess only the geopolitics are left.
I am obviously biased, but, yes, technically it is very, very interesting. Distributed transactions, kiparis, YQL are interesting. Another aspect, the only other open alternative is Hadoop, hbase and hive. They don't compete with yt on usability and developer experience aspects. Yt is much more polished, despite historical quirks.
Re: Yandex open-sources its exabyte-scale big data platform
#76The github repo has 5 million lines of C++ code (headers included), 1.6 million lines of C code, and even nearly 1 million lines of Scala + Java code. We'd need some serious docs to adopt this technology. The most interesting part of YT is Cypress. I'm particularly interested in how they make their master cluster horizontally scalable.
Historically, the master server of YTsaurus was a single RSM (replicated state machine) that contained all the meta-information about the cluster. This included the tree of the distributed filesystem, transactions, information about users and tables, placement of chunks, and much more.
However, this approach proved to be non-scalable as the memory amount and throughput of the master server soon became insufficient. To address this issue, we implemented Multicell technology. With Multicell, there are multiple RSMs called secondary masters that store information about chunks of the tables and their placement. The primary master still stores information about the distributed filesystem and transactions but is now single and non-sharded.
After a few years, the masters became overloaded again, and we implemented Portals. With Portals, one can select a subtree of Cypress and place it in one of the secondary masters. This technology is used nowadays, and home directories of some active users are hosted on secondary masters.
However, we anticipate that this approach will also become insufficient in a few years. Therefore, we are currently working on a new technology called Sequoia, which stores information about the Cypress tree shape in horizontally scalable dynamic tables.
It is hard to describe all aspects of master server internals in one comment. Therefore, feel free to join our chat at t.me/ytsaurus for further discussion!
Re: Yandex open-sources its exabyte-scale big data platform
#77Funny to see 0 of 50 comments on a HN tech article being tech-related...
Agreed, but this is presumably also an absolutely massive project hardly anyone here has even used before. So it's not surprising that there are no big tech insights on the day of the release. An `scc` printout might be interesting, but any in-depth analysis is going to take a long time.
Here are some developers from YT which can help you and answer some technical questions. Also YT is a huge and old project. I believe you can find a lot of people which are ex-Yandex who worked with YT.
Re: Yandex open-sources its exabyte-scale big data platform
#78Earlier quoted context omitted.
It's illegal but the rental contract is private, so it can be denied without explanation; good luck suing and proving that you're being discriminated. The discrimination in public ads and places like restaurants is strictly illegal, though essentially not enforced unless you sue, because of the multitude of reasons - racism and lax anti-discrimination policies in particular, and also a million of others (the entire r…
> It's illegal but the rental contract is private, so it can be denied without explanation; good luck suing and proving that you're being discriminated. How is this different than in other places in Europe
When worded like this, it's not that different. It's the little details that add up, depending on the actual country. You're much less likely to be discriminated against in UK or Germany than in places like Bulgaria, Ukraine, or Russia. Due to both the attitude and enforcement. The rental market in Germany seems over-regulated, but my black friend of Ethiopian descent (he's Russian, born and raised) had no problem finding a place to live there, while in Russia he's been overtly or silently rejected by the landlords so often so he had to rent the apartment from myself for a year despite it being far away from his work.
Re: Yandex open-sources its exabyte-scale big data platform
#79Re: Yandex open-sources its exabyte-scale big data platform
#80Is it an alternative to Snowflake?
We provide YQL for running large-scale OLAP SQL queries. In this regard YT can be compared to Snowflake. However, we target on prem deployment in the first place, while Snowflake runs in aws/gcp/azure, and queries are performed over data that sits in S3.