Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres is eating the database world

medium.com

Post reply on HN
Print view View original

101–110 of 147 posts

Re: Postgres is eating the database world

#101

Post by AtlasBarfed »

lmm wrote: post #31

Postgres is still single-node-first, and while Citus exists I'm skeptical that it can ever become as easy to administer as a true HA-first datastore. For me the reason to use something like Cassandra or Kafka was never "big data" per se, it was having true master-master fault tolerance out of the box in a way that worked with everything.

If you are going to multi-node Postgres, you need to start planning for Cassandra/Dynamo.

That is a BIG lift. Joins don't really practically scale at the Cassandra/Dynamo scale, because basically every row in the result set is subject to CAP uncertainty. "Big Data SQL" like Hive/Impala/Snowflake/Presto etc are more like approximations at true scale.

Relational DBMS is sort of storage-focused in the design and evolution: you figure out the tables you need to store the data in a sensible way. They you add views and indexes to optimize for view/retrieval.

Dyanmo/Cassandra is different, you start from the views/retrieval. That's why it is bad to start with these models for an application because you have not fully explored all your specific data structuring and access patterns/loads yet.

By the time Postgres hits the single node limits, you should know what your highest volume reads/writes are and how to structure a cassandra/dynamo table to specifically handle those read/writes.

Re: Postgres is eating the database world

#102

Post by AtlasBarfed »

CyberDildonics wrote: post #95
ksec wrote: post #66

>As DuckDB’s manifesto “Big Data is Dead” suggests, the era of big data is over. I have been stating this since at least 2020 if not earlier. We are expecting DDR6 and PCI-E 7.0 Spec to be finalised by 2025. You could expect them to be on market by no later than 2027. Although I believe we have reach the SSD IOPS limits without some special SSD with Z-NAND. I assume ( I could be wrong ) this makes SSD bandwidth on Se…

I'm not sure what your point is here, it seems like you are just listing off announced hardware.

The threshold for Cassandra / Dynamo scaling is increasing is probably the only point. "Big data is dead" is pretty stupid to say, typical clickbait marketing by a database that will probably be chucked away by something else trendy in another year.

But at a certain point, a 10,000 core 5 petabyte single megamachine starts to practically encounter CAP from the internal scale alone. It already ... kind of ... does.

And no matter how big your node scales, if you need to globally replicate data ... you have to globally replicate it over a network, and you need Cassandra (DynamoDB global replication is shady last I looked at it, I have no idea how row-level timestamps can merge-resolve conflicting rows updated in separate global regions)

Re: Postgres is eating the database world

#103

Post by remram »

fulafel wrote: post #47

Someone who picked their tools with good tech judgement 25 years ago can be using the same today (eg PG, Python, Linux) without corporate control of them, it's pretty great.

25 years ago... PostgreSQL 6, Python 1.5, Linux 2.2? I don't know if picking those tools was good judgement then...

Re: Postgres is eating the database world

#104

Post by fulafel »

vsnf wrote: post #72
fulafel wrote: post #47

Someone who picked their tools with good tech judgement 25 years ago can be using the same today (eg PG, Python, Linux) without corporate control of them, it's pretty great.

That feels a bit like hindsight talking. Linux perhaps, but were Python and Postgres really the obvious good judgement choices 25 years ago? Every other choice was poor judgement?

MySQL didn't have transactions (for many years after that), so PG over MySQL would have been the case for "good technical judgement" though maybe not the majority choice. For Python, maybe I'm biased - but you could have went for PHP in the argument if you swing that way and it still works.

Re: Postgres is eating the database world

#105

Post by zachmu »

We're writing a postgres-compatible database that doesn't use any postgres code:

https://github.com/dolthub/doltgresql/

We're doing this because our main product (Dolt) is MySQL-compatible, but a lot of people prefer postgres. Like, they really strongly prefer postgres. When figuring out how to support them, we basically had three options:

1) Foreign data wrapper. This doesn't work well because you can't use non-native stored procedure calls, which are used heavily throughout our product (e.g. CALL DOLT_COMMIT('-m', 'changes'), CALL DOLT_BRANCH('newBranch')). We would have had to invent a new UX surface area for the product just to support Postgres.

2) Fork postgres, write our own storage layer and parser extensions, etc. Definitely doable, but it would mean porting our existing Go codebase to C, and not being able to share code with Dolt as development continues. Or else rewriting Dolt in C, throwing out the last 5 years of work. Or doing something very complicated and difficult to use a golang library from C code.

3) Emulation. Keep Dolt's Go codebase and query engine and build a Postgres layer on top of it to support the syntax, wire protocol, types, functions, etc.

Ultimately we went with the emulation approach as the least bad option, but it's an uphill climb to get to enough postgres support to be worth using. Our main effort right now is getting all of postgres's types working.

Re: Postgres is eating the database world

#106

Post by renegade-otter »

sgarland wrote: post #92
throwawaaarrgh wrote: post #51

It's not a best practice, it's a fad. 99% of people who recommend or use Postgres barely know how to use it. Another trendy database will come along and you'll stop seeing all these posts about it. Happens every decade. I'll link back to this post in a few years with "I told you so".

> 99% of people who recommend or use Postgres barely know how to use it. You're not wrong here, although you could just as easily say "99% of people who recommend $DB barely know how to use it." Databases remain a mysterious black box to entirely too many people, despite the three largest (SQLite, Postgres, MySQL) being open source, and having extensive documentation. I've come to the conclusion that most devs don't…

I feel this on a soul level. I wrote about it: https://renegadeotter.com/2023/11/12/your-database-skills-ar...

Re: Postgres is eating the database world

#107

Post by Izkata »

sgarland wrote: post #90
Izkata wrote: post #17

Earlier quoted context omitted.

Every time stuff like this comes up I wonder how much the people having issues would be willing to share - because every time I've fought with the postgres query planner, it eventually turned out what I wanted to do had massively worse performance* because of something I didn't take into account that postgres did. And each time, once I learned what that thing was, I was able to fix it the right way and get the query…

> CLUSTER And here we see the benefit of clustered indices, á la MySQL. Assuming, of course, your PK is k-sortable.

It may use the same name, but reading https://dev.mysql.com/doc/refman/8.0/en/innodb-index-types.h... this doesn't really look like the same thing and wouldn't help here. InnoDB's clustered index (usually on the primary key) is used for fast row lookups by that primary key. It only has an advantage if the primary key is the order you want, but otherwise would have the exact same problem postgres's query planner was protecting me against, but without any way to fix the problem.

Re: Postgres is eating the database world

#108

Post by The_Colonel »

hans_castorp wrote: post #79
egnehots wrote: post #54

Postgres is far from perfect: - The codebase is old and huge, accruing some heavy technical debt, making it a less than ideal foundation for iterating quickly on a new paradigm like AI and vector databases. - Some ancient design decisions have aged poorly, such as its one connection per process model, which is not as efficient as distributing async tasks over thread pools. If not mitigated through an external connect…

> Some ancient design decisions have aged poorly, such as its one connection per process model Oracle uses the same model by default on Linux. Since 19 (or maybe earlier) it is configurable though, but the default is still one process per connection if I'm not mistaken.

It's actually quite impressive they were willing & able to make such a drastic change in such an old and conservative codebase.

Re: Postgres is eating the database world

#109

Post by ksec »

CyberDildonics wrote: post #95
ksec wrote: post #66

>As DuckDB’s manifesto “Big Data is Dead” suggests, the era of big data is over. I have been stating this since at least 2020 if not earlier. We are expecting DDR6 and PCI-E 7.0 Spec to be finalised by 2025. You could expect them to be on market by no later than 2027. Although I believe we have reach the SSD IOPS limits without some special SSD with Z-NAND. I assume ( I could be wrong ) this makes SSD bandwidth on Se…

I'm not sure what your point is here, it seems like you are just listing off announced hardware.

The point is Big Data or Hard to Scale aren't as much of a thing with Hardware technology moving forward.

Re: Postgres is eating the database world

#110

Post by The_Colonel »

dagw wrote: post #93
Shish2k wrote: post #88

Earlier quoted context omitted.

In my personal experience of being around back then, postgres and python were still considered "technically better", but such a massive pain in the ass to install (especially on cheap shared hosting where it was often actually impossible to install) that only the most masochistic people would even try. I myself wrote a fastcgi implementation in PHP which would allow a web server which only supported php to call pytho…

no other language has even attempted to compete with PHP PHP has been very good at reinventing itself and being its own biggest competitor. Zend outcompeted PHP3, PHP5 outcompeted PHP4, PHP7 outcompeted PHP5 and so on. Compare a site written in modern idiomatic PHP8 using something like Laravel to a site written in classic PHP3 and they might as well be two different languages.

IIRC the big break was betweem PHP 4 and PHP 5, they had very different object model. Otherwise the it's not that different.
Post reply on HN