Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TiDB – cloud-native, distributed SQL database written in Go

github.com

Post reply on HN
Print view View original

41–50 of 67 posts

Re: TiDB – cloud-native, distributed SQL database written in Go

#41

Post by std_reply »

convolvatron wrote: post #7
srameshc wrote: post #5

I love these new distributed DBs. CockroachDB is one of them. Still I think a managed Postgres/MySQL is a better choice. My primary concern is how challenging will it be if you have to eventually move your data out to a RDBMS for cost or other reasons. Does anyone have any experience ? I am not talking enterprise scale but data about size of 50 - 100GB scale.

why do you think that would be harder? assume for the moment that the reader here is going to run at the effective rate of a single node and not that we're going to try to parallelize that. Assuming we have transaction isolation, that reader is going to get a consistent snapshot. a distributed database is potentially more complicated to operate, and optimize, and because its new and potentially has more sharp edges m…

I would argue the opposite, distributed databases are much easier to operate at large scale. Truly online distributed DDL, at least in TIDB, strong consistency etc.

People who bang on about Postgres replication have rarely setup replication in Postgres themselves and that too in the 100a of Pb scale.

MySQL replication works well and can be scaled more easily (relative to Postgres) but has its own problems. eg., DDL is still a nightmare, lag is a real problem, usually masked by async replication. But then eventual consistency makes the application developers life more complicated.

Re: TiDB – cloud-native, distributed SQL database written in Go

#42

Post by std_reply »

fipar wrote: post #37
nuphonenuacc wrote: post #25

What is that first paragraph? "This implementation proves the CAP theorem as wrong"? Doesn't tell me what your drawbacks are. This is mongodb all over again. Will only ever consider if the customer really wants it. (My guess performance) (Also, bold move to write a db without fully being able to manage memory).

I can't find that first paragraph, and it seems that readme file was last modified 3 months ago, though you're using quotes, so perhaps you are paraphrasing? The first paragraph I find under Key Features is: Distributed Transactions: TiDB uses a two-phase commit protocol to ensure ACID compliance, providing strong consistency. Transactions span multiple nodes, and TiDB's distributed nature ensures data correctness ev…

Go is used for the SQL layer, it’s a modern optimizer that can do distributed joins etc. ie. Issue parallel reads to the storage nodes.

Additionally, it can push down the DAG to the TiKV storage nodes, written in Rust, to reduce movement of data and work closer to the physical data.

Re: TiDB – cloud-native, distributed SQL database written in Go

#43

Post by benatkin »

eikenberry wrote: post #13

VC backed database company with a CLA on their "open source" project. Red flag. Is there a community fork yet?

With it under the Apache 2 license, misuse of a CLA to get free contributions from the community that aren't shared back to the community isn't currently a problem.

Re: TiDB – cloud-native, distributed SQL database written in Go

#44

Post by benatkin »

eikenberry wrote: post #29
hbbio wrote: post #17

Earlier quoted context omitted.

How would you fund the open source development of a database? TiDB is Apache licensed, that should be enough no? Because if we ban all projects from VC-backed startups, you're going to clear your tech stack pretty quickly.

> How would you fund the open source development of a database? I think the best way would be something similar to the Linux Foundation. Companies in need of a certain type of database would pool resources to develop and maintain it. PostgreSQL was a community developed database that was funded in something approaching this way as most of the developers were either students or worked for someone who paid them while t…

> the CLA gives the owning company the right to change that at any time

It isn't copyleft, so even without a CLA, they could make a proprietary or open core version of the product, as can anyone else.

Re: TiDB – cloud-native, distributed SQL database written in Go

#45

Post by eikenberry »

std_reply wrote: post #39
eikenberry wrote: post #29

Earlier quoted context omitted.

> How would you fund the open source development of a database? I think the best way would be something similar to the Linux Foundation. Companies in need of a certain type of database would pool resources to develop and maintain it. PostgreSQL was a community developed database that was funded in something approaching this way as most of the developers were either students or worked for someone who paid them while t…

TiKV the Core Storage scalable component is a CNCF graduated projected. PingCAP cannot change the license even if it wanted to.

Thanks for the info! I wasn't aware of this and didn't see any mention of this in their github docs. It means that at least you can use that part of the project without worry. They should promote this aspect of the project more prominently. It should be of great interest to anyone thinking about adopting the technology.

Do you have any links to the Core Storage CNCF project? I couldn't find it on the CNCF website under the Database and Cloud Native Storage sections?

Re: TiDB – cloud-native, distributed SQL database written in Go

#46

Post by peterson_lock »

std_reply wrote: post #32
xmichael909 wrote: post #6

Earlier quoted context omitted.

Same as before, excellent when you little little data and limited transactions. Once things get big and you lose sync, everything goes to shit.

This is quite an ignorant comment. TiDB routinely handles 100s of TB of data. Go watch LinkedIn’s presentation on why they choose TiDB as their strategic db going forward.

Do you have a link?

Edit: https://www.youtube.com/watch?v=DzZ-I8WL2jM

Re: TiDB – cloud-native, distributed SQL database written in Go

#48

Post by PeterCorless »

arccy wrote: post #18
dangoodmanUT wrote: post #11

Earlier quoted context omitted.

how do you have doubts when nearly every significant company is running one, and distributed DBs are consistently proven to be the most correct (e.g. FDB, TigerBeetle)?

"every significant company"? i'd say most of them just use postgres or mysql and scale fine.

If you look at DB-engines.com/ranking and look at all of the collective interest in all of the databases listed, you will see that the aggregate "score" of all databases combined 7105.84. Postgres is indeed popular; but it is only ranked 4th on the list, with its own score of 648.96. MySQL currently is still 50% larger in terms of interest, with a score of 998.15.

Which means interest in Postgres (specifically) is only 9.13% of overall interest in databases; MySQL another 14.04%. Combined 23.27%.

Is that a significant percentage of interest? Yes. Many others are a fraction of 1% of mindshare in the market.

Yet the reason there are 423 systems ranked in DB-Engines is because no one size fits all data, or data query patterns, or workloads, or SLAs, or use cases.

PostgreSQL and MySQL are, at the end of the day, oriented towards OLTP workloads. While you can stretch them to be used for OLAP, these are "unnatural acts." They were both designed in days long ago for far smaller datasets than typical for modern-day petabyte-scale, real-time (streaming) ingestion, cloud-native deployments. While many engineering teams have cobbled together PostgreSQL and MySQL frankenservers designed for petabyte-scale workloads, YMMV for your data ingest, and for p99s and QPS.

The dynamic at play here is that there are some projects that lend themselves to "general services" databases, where MySQL or PostgreSQL or anything else to hand is useful for them. And then there are specialized databases designed for purpose for certain types of workloads, data models, query patterns, use cases, and so on.

So long as "chaos" fights against "law" in the universe, you will see this desire to have "one" database standard rule them all, versus a Cambrian explosion of options for users and use cases.

Re: TiDB – cloud-native, distributed SQL database written in Go

#49

Post by PeterCorless »

sgarland wrote: post #35
PeterCorless wrote: post #27

Earlier quoted context omitted.

It all depends on what kind of queries you're running. I came from the OLTP market, where you're generally doing single-row operations. Basic CRUD. Single table work on denormalized data. Now go to OLAP, and a single query might be doing multiple table joins. It might be scouring billions of records. It might need to do aggregations. Suddenly "millions of ops" might be reduced to 100 QPS. If you're lucky. And yes, th…

Not sure why you think OLTP doesn’t also do complex joins. In a properly normalized schema, you’ll likely have many. But yes, OLAP is of course its own beast, and most DBs are suited for one or the other.

I think it's a matter of use case. Doing ad hoc data exploration on an OLTP system generally sucks the wind out of the performance. Even if you have some type of workload prioritization, isolation, and limitation, allowing data scientists and business analysts freely wandering through your production OLTP database sounds like a Bad Time.

The organization might say "Okay. Maybe you should do your ad hoc exploration on an OLAP system. Preferably our data warehouse where you can let your report run for hours and we won't see a production brownout while it's running."

So complexity of ad hoc joins in the warehouse generally can get more complex.

Re: TiDB – cloud-native, distributed SQL database written in Go

#50

Post by rpaik »

eikenberry wrote: post #13

VC backed database company with a CLA on their "open source" project. Red flag. Is there a community fork yet?

Is your main issue with the VC funding, the CLA, or the combination of both. Doesn’t Kubernetes (hosted by the Linux Foundation) also have a CLA?
Post reply on HN