Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PostgreSQL is enough

gist.github.com

Post reply on HN
Print view View original

291–300 of 323 posts

Re: PostgreSQL is enough

#291

Post by DanielHB »

bun_terminator wrote: post #226
DanielHB wrote: post #223

Earlier quoted context omitted.

Besides what others are saying here, data integrity is a lot easier with a relational database if your data is heavily relational. I have managed systems with About data types, what limitations are you referring to? jsonb is well supported in many dbs and throwing random large binary blobs in the middle with your normal data is a bad idea with or without a relational database. I can see the appeal to replace noSQL so…

I think my wtf moment was that you can't have real variable length strings. And I'm pretty pampered by c++ highly nuanced numeric types with varying sizes and signed/unsigned.

fair enough, relational DBs are optimised for querying so you often see tradeoffs like this.

Re: PostgreSQL is enough

#292

Post by vp8989 »

rcaught wrote: post #47
chuckhend wrote: post #31

Earlier quoted context omitted.

You get exactly once when you consume with pgmq and run your queue operations inside transactions in your postgres database. I can't think of an easy way to get some equivalent on SQS without building something like an outbox.

SQS FIFO has exactly-once processing

That's not what the parent post was referring to. If SQS (or your ability to talk to it) is down and your database isn't, what do you do?

Re: PostgreSQL is enough

#293

Post by javcasas »

tangjurine wrote: post #100
randomdata wrote: post #92

Earlier quoted context omitted.

Postgres complicates the application in several ways. In particular, Postgres suffers from the n+1 problem, while SQLite does not. That requires a significant amount of added complexity in the application to hack around. Why over engineer the application before it has proven itself as something anyone even wants to use? Let's face it, the large majority of software written gets thrown away soon after it is created. I…

> Postgres suffers from the n+1 problem, while SQLite does not. ?

Meaning insertion sort does not suffer from the problems that made us construct quicksort because we optimized the insertion a lot (at a tradeoff that we will not specify).

Re: PostgreSQL is enough

#294

Post by mharig »

spothedog1 wrote: post #206
cnity wrote: post #142

Earlier quoted context omitted.

This desire can sometimes be so strong that people insist on truly wacky decisions. I have before demonstrated that Postgres performs perfectly well (and in fact exceeds) compared with a niche graph database, and heard some very strange reasons for why this approach should be avoided. A lot of the time you hear that it's engineers who chase shiny technology, but I've seen first hand what can happen when it's leadersh…

Can you expand on Postgres vs Graph Databases?

IIRC, the biggest (AFAIK) graph DB in the world, TAO from Facebook, is based on an underlying MySQL. There must be a good reason why FB prefers a SQL DB over a dedicated graph DB.

Re: PostgreSQL is enough

#295

Post by sgarland »

randomdata wrote: post #284
dventimi wrote: post #283

Earlier quoted context omitted.

Compare and contrast the query execution stages of PostgreSQL and SQLite. How exactly do they work? Please be as precise as possible. Try to avoid imprecise terms like "simple", "suffering", "system layer", and "hack."

For what purpose? I can find no source of value in your request.

To prove that you have an inkling about the subject you have wandered into.

Engineers and Scientists do that; they don’t hide behind airy and vague language – that is the realm of conmen.

You stated that you cannot implement a tree-like structure with tables, so I set about proving you wrong, and posted a gist that does so.

Until you can back up your claims with data, your words are meaningless, and no one here is going to take you seriously.

Re: PostgreSQL is enough

#296

Post by sfn42 »

deathanatos wrote: post #79
superb-owl wrote: post #2

I often go down rabbit holes like this, trying to collapse and simplify the application stack. But inevitably, as an application grows in complexity, you start to realize _why_ there's a stack, rather than just a single technology to rule them all. Trying to cram everything into Postgres (or lambdas, or S3, or firebase, or whatever other tech you're trying to consolidate on) starts to get really uncomfortable. That s…

I have certain experience with some technologies, e.g., SQS and Postgres. Say I'm on your team, and you're an application developer, and you need a queue. If you're taking the "we're small, this queue is small, just do it in PG for now and see if we ever grow out of that" — that's fine. "Let's use SQS, it's a well-established thing for this and we're already in AWS" — that's fine, I know SQS too. I've seen both of th…

I worked in a team that did this. It was mostly staffed by juniors, and the team leader wasn't very interested in the technical aspects, they just went to a page, checked that the new feature worked alright and gave the green light.

So over the years these juniors have repeatedly chosen different tech for their applications. Now the team maintains like 15-20 different apps and among them there's react, Vue, angular, svelte, jQuery, nextjs and more for frontends alone. Most use Episerver/Optimizely for backend but of course some genius wanted to try Sanity so now that's in the mix as well.

And it all reads like juniors built it. One app has an integration with a public api, they built a fairly large integration app with an integration db. This app is like 20k lines of code, much of which is dead code, and it gets data from the public api twice a day whereas the actual app using the data updates once a day and saves the result in its own Episerver db. So the entire thing results in more api traffic rather than less, the app itself could have just queried the api directly.

But they don't want me to do that, they just want me to fix the redundant integration thing when it breaks instead. Glad I'm not on that team any more.

Re: PostgreSQL is enough

#297

Post by hagen1778 »

ThePhysicist wrote: post #51

How about metrics? Would it be possible to do this in Postgres as well? I'm talking e.g. about simple counters that are not directly vital to business logic. I'm currently using Redis for those things as I'm thinking that Postgres seems to be the wrong choice for this kind of data, would anyone disagree?

> Would it be possible to do this in Postgres as well?

Of course! The question is only in your requirements. Keeping a simple counter with limited cardinality should work just great. But nowadays monitoring is much more serious than that. For monitoring k8s clusters the average ingestion rate of metrics per second varies from 100K to 2Mil. I don't know if, resource-wise, it would be a right decision to use Postgres for storing this.

So when requirements are high, and they are for real-time infrastructure and applications monitoring, it is better to consider something like ClickHouse (for people familiar with Postgres) or VictoriaMetrics (for people familiar with Prometheus).

Re: PostgreSQL is enough

#298

Post by Rudism »

I worked for years on a massive PostGraphile project where 99% of the backend application code was written as stored procedures and table triggers, and let me tell you, once the complexity of the system moves beyond basic CRUD operations this kind of system is an absolute chore to work with. The code base is extremely difficult to organize and manage, code completion and compile-time error safety is virtually non-existent, deploying new versions will probably be non-intuitive and confusing for everyone, training new programmers on the code is chaos... it is absolutely something I would never recommend to anyone under any circumstance.

That being said, I do fully agree that things like Redis, ElasticSearch, RabbitMQ, and Kafka are probably unnecessary in a large number of projects that decide to add them to the stack if those projects are also already using PostgreSQL as a database.

Re: PostgreSQL is enough

#299

Post by asah »

ndriscoll wrote: post #246
asah wrote: post #235

Earlier quoted context omitted.

minor nit: 9K TPS for Visa are business transactions - TBD how many database transactions are generated... (still, modern postgresql can easily scale to 10,000s (plural) of TPS on a single big server, especially if you setup read replicas for reporting)

Yeah, I don't mean to say Visa can run global payment processing on a single postgres install; I'm sure they do a ton of stuff with each transaction (e.g. for fraud detection). But for system design, it gives an order of magnitude for how many human actions a global system might need to deal with, which you can use to estimate how much a wildly successful system might need to handle based on what processing is needed…

yyy we're in violent agreement!

interesting re reddit, that's really tiny! but again, I'm even more curious about how many underlying TPS this turns into, net of rules firing, notifications and of course bots that read and analyze this comment, etc. Still, this isn't a scaling issue because all of this stuff can be done async on read replicas, which means approx unlimited scale in a single-database-under-management (e.g. here's this particular comment ID, wait for it)

Slack experiences 300K write QPS: https://slack.engineering/scaling-datastores-at-slack-with-v...

Re: PostgreSQL is enough

#300

Post by [deleted] »

sgarland wrote: post #295
randomdata wrote: post #284

Earlier quoted context omitted.

For what purpose? I can find no source of value in your request.

To prove that you have an inkling about the subject you have wandered into. Engineers and Scientists do that; they don’t hide behind airy and vague language – that is the realm of conmen. You stated that you cannot implement a tree-like structure with tables, so I set about proving you wrong, and posted a gist that does so. Until you can back up your claims with data, your words are meaningless, and no one here is go…

[deleted]
Post reply on HN