Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to use Postgres for everything

github.com

Post reply on HN
Print view View original

171–180 of 181 posts

Re: How to use Postgres for everything

#171

Post by whateveracct »

underdeserver wrote: post #11
whateveracct wrote: post #7

Just don't use a single Postgres DB for everything as you scale up to 100+ engineers. You'll inevitably get database-as-the-API. Now if you have the actual technical leadership [1] to scale your systems by drawing logical and physical boundaries so that each unit has its own Postgres? Yeah Postgres for everything is solid. [1] Surprisingly rare I've found. Lots of "successful" CTOs who don't do this hard part.

Drop the scare quotes from "successful". These guys shipped products. The migration to multiple databases, syncing user information etc. is a milestone, not a necessity at every step.

nah at the scale I'm talking about, if you've shipped a produc but your engineers are increasingly dissatisfied with the effort it takes to ship (and are becoming gun shy because you haven't decoupled things so their outage blast radius is huge), then you are currently failing as a technical leader. And if those things are true and you stick to "one Postgres monolithic DB for everything" you are cargo culting too hard.

You can't just keep "doing things that don't scale" forever. If you have 100+ engineers [1], you aren't a startup anymore no matter your roots.

[1] and remember, my comment that is the ultimate parent of this conversation is about not continuing to do this at 100+. I said nothing about scaling up to that point.

Re: How to use Postgres for everything

#172

Post by Jeffpk11 »

ksynwa wrote: post #103

Anyone has advice for backup up a postgres database that is run for personal use? I currently have one that is used by miniflux. I wrote a script for backing it up but it is gonna fail me some day. ``` #!/usr/bin/env sh set -e PGUSER=miniflux PGPASSWORD=... pg_dump -F t -h 127.0.0.1 miniflux | gzip > /backups/miniflux_db_temp.tar.gz mv /backups/miniflux_db{_temp,}.tar.gz ```

[dead]

Re: How to use Postgres for everything

#173

Post by anon3459 »

lokimedes wrote: post #77
anon3459 wrote: post #27

Earlier quoted context omitted.

Haaaa same here for apache age. Can you elaborate a little please

The original sponsor of the project just withdrew all resources, the state of the existing codebase is far from mature, the client I tried (python) was really shaky and the lidt goes on. As for the precarious life of graph database companies. DGraph also went though being sold recently, and OrientDB that I also liked, as acquired by SAP, only to be abandoned. Neo4j has stood its time, but the licensing doesn’t fit ou…

What did you decide on in the end ?

Re: How to use Postgres for everything

#174

Post by com »

whateveracct wrote: post #141
com wrote: post #90

Earlier quoted context omitted.

And the best thing is that until it’s a problem you can focus on product/market fit and delighting your customers. Overengineering is a plague amongst SWEs, and almost as dangerous as failing to sell the product in the market.

I think cargo cutting stuff like "Postgres for everything" is also a plague. I agree you don't want to overengineer, but if you're a leader and your policy is basically "engineers can't ever be trusted to do hard things" then you're a bad leader. Sorry if this comes off as brusque, but I've seen good engineers want to use Elasticache for the right reasons and seen "leaders" tell them no because "caching is one of the…

I worked in a Postgres for everything business, and the IPO and later value creation for customers and investors (including me) had been stratospheric.

Not dealing with loads of technology choices in the early days is a boon.

Once we hit more than $100m in rev it made sense to allow a bit more optimisation for purpose - but only when we had cash flow to pay for it. Otherwise all these fancy-shmancy choices are just dressed-up tech debt.

Re: How to use Postgres for everything

#175

Post by seanhunter »

refset wrote: post #153
seanhunter wrote: post #138

Earlier quoted context omitted.

I’m not 100% sure what you mean. Systems I have used that do this don’t generally store each time series in a different table. Normally there’s just one big table for intraday time series and one for daily, with columns being like ts, as_of, series_id, value, metadata or something like that. It scales just fine depending of course on the usual stuff - load pattern etc. If you want really high scalability you should b…

I meant 'scale' mostly in the sense of 'complexity' (sorry!). If you only have a small number of tables you need/want this versioning for then the DIY approach is workable, but if you want to apply this across an entire schema then things can get complicated fast.

I have only seen bitemporality being useful in two contexts. 1 is a timeseries store which as I say usually 2 tables is all you need. Secondly is an EAV (entity/attribute/value store) which is just one table. So again entirely manageable. I’ve seen it work just fine with 10s of thousands of logical timeseries and billions of ticks or entities up to the millions without much of an issue. You definitely don’t need a special database and if anything the special database would probably scale worse than normal databases of the kind I’ve mentioned.

Re: How to use Postgres for everything

#176

Post by refset »

seanhunter wrote: post #175
refset wrote: post #153

Earlier quoted context omitted.

I meant 'scale' mostly in the sense of 'complexity' (sorry!). If you only have a small number of tables you need/want this versioning for then the DIY approach is workable, but if you want to apply this across an entire schema then things can get complicated fast.

I have only seen bitemporality being useful in two contexts. 1 is a timeseries store which as I say usually 2 tables is all you need. Secondly is an EAV (entity/attribute/value store) which is just one table. So again entirely manageable. I’ve seen it work just fine with 10s of thousands of logical timeseries and billions of ticks or entities up to the millions without much of an issue. You definitely don’t need a sp…

An EAV table is usually a symptom of a wider set of issues with schema management, and in contrast, most people I've spoken to who have implemented their own EAV tables on top of a regular SQL database have ended up regretting it because the approach is too hard to scale and maintain. In your experience, was the EAV model limited to a subset of the overall schema?

I agree a special database shouldn't be necessary at all, and instead, convenient syntax for immutable DML and temporal support should be built into Postgres already. But short of a miracle it will probably take a new ('special') database in order for Postgres to evolve in response. Therefore, in the meantime, we believe there's a gap in the market for organisations who value the 'safety' (foolproof complexity reduction) that native bitemporality in a database can offer above the raw query performance offered by existing update-in-place databases: https://xtdb.com/blog/but-bitemporality-always-introduces-co...

Re: How to use Postgres for everything

#177

Post by whateveracct »

com wrote: post #174
whateveracct wrote: post #141

Earlier quoted context omitted.

I think cargo cutting stuff like "Postgres for everything" is also a plague. I agree you don't want to overengineer, but if you're a leader and your policy is basically "engineers can't ever be trusted to do hard things" then you're a bad leader. Sorry if this comes off as brusque, but I've seen good engineers want to use Elasticache for the right reasons and seen "leaders" tell them no because "caching is one of the…

I worked in a Postgres for everything business, and the IPO and later value creation for customers and investors (including me) had been stratospheric. Not dealing with loads of technology choices in the early days is a boon. Once we hit more than $100m in rev it made sense to allow a bit more optimisation for purpose - but only when we had cash flow to pay for it. Otherwise all these fancy-shmancy choices are just d…

> Once we hit more than $100m in rev

Yeah my original comment was about experiences working at places with this kind of monetary success (if not more) and stubbornly not evolving. Low trust eng leadership philosophies will do that though.

Re: How to use Postgres for everything

#178

Post by halfcat »

RedShift1 wrote: post #22
jensneuse wrote: post #16

I absolutely love Postgres, but please allow me to say that you absolutely don't want to expose an API generated from a database to people outside of your team. This limits you a lot in changing the way you store your data. I wrote about this topic before and haven't changed my opinion much. You don't want to have that tight coupling: https://wundergraph.com/blog/six-year-graphql-recap#generate...

What exactly is the problem with tight coupling? You're going to insert an entire layer that basically translates format A to format B, just so you can later change a column name in the database and not have to change it in the API or something?

> * What exactly is the problem with tight coupling?*

As Joel Spolsky put it: ”the cost of software is the cost of its coupling”.

More specifically the cost of making changes when “if I change this thing I have to change that thing”. But if there’s no attention paid to coupling, then it’s not just the two things you gave to change, but “if I change this thing I have to change those 40 things”.

Re: How to use Postgres for everything

#179

Post by rcombatwombat »

pdimitar wrote: post #150
osigurdson wrote: post #149

Earlier quoted context omitted.

Perhaps I misunderstand, but if you have 50 parallel agents, why not just have each pull messages, process them and ACK when complete? The part I don't understand is the pre-fetch. Note that NATS is much more flexible than Kafka however, so more likely to fit more uses cases (even just for streaming). For the first question, I'd definitely recommend using ClickHouse for 10B - 1T points.

I mean the following (let me simplify it). You pull stuff from Kafka in batches of 4 and immediately send each message to a parallel worker, first parsing them in-thread and sequentially and these are the results of the parsing: 1. ok 2. error 3. ok 4. ok I cannot not-ACK message#2 because that means message#1 is not ACK-ed as well. Does NATS solve this? F.ex. can I get a reference to each message in my parallel work…

Yes, one of the many differences (advantages) of NATS JetStream over Kafka: with NATS you can ack explicitly each message individually, and even better if you set your stream to be in 'work-queue' mode it will also automatically (and atomically) delete the ack'd message from the stream (i.e. like a 'proper' queue) another difference with Kafka where you can't delete individual messages in the middle of a stream (only trim the tail end).

You can also 'negative ack' messages, specify a back-off period before the message is re-delivered (because NATS automatically re-delivers un-acked (or nacked) messages) when you can't temporarily process it, or 'term' a message (don't try to re-deliver it, e.g. because the payload is bad), or even 'ask for more time before needing to ack the message (if you are temporarily too slow at processing the message).

Re: How to use Postgres for everything

#180

Post by pdimitar »

rcombatwombat wrote: post #179
pdimitar wrote: post #150

Earlier quoted context omitted.

I mean the following (let me simplify it). You pull stuff from Kafka in batches of 4 and immediately send each message to a parallel worker, first parsing them in-thread and sequentially and these are the results of the parsing: 1. ok 2. error 3. ok 4. ok I cannot not-ACK message#2 because that means message#1 is not ACK-ed as well. Does NATS solve this? F.ex. can I get a reference to each message in my parallel work…

Yes, one of the many differences (advantages) of NATS JetStream over Kafka: with NATS you can ack explicitly each message individually, and even better if you set your stream to be in 'work-queue' mode it will also automatically (and atomically) delete the ack'd message from the stream (i.e. like a 'proper' queue) another difference with Kafka where you can't delete individual messages in the middle of a stream (only…

Ohhh, this is an awesomely informative and concrete message! Extremely useful, thank you!

I like everything about this: the ability to NACK individual messages, the specifying of a backoff period, _and_ to just discard a message f.ex. if you really cannot do anything about it. Super nice. I am grateful.

Post reply on HN