[flagged]
PostgreSQL for Everything
91–100 of 286 posts
Re: PostgreSQL for Everything
#92Use case matters.
I use a SQL databases as needed. I've used Postgres, Sqlite, Duckdb, Json files with AWS Athena, Oracle enterprise for ERP systems (a multitude of schemas and objects with interoperability), and others.
I'm currently, deploying Duckdb with AWS S3 Tables (Iceberg) to see how it fits for a use case I have.
IT is great and always changing. Keep trying new things.
Cheers
Re: PostgreSQL for Everything
#93This kind of post (Postgres! It's all you need!) is getting pretty tiresome. Postgres does not even come close to a full replacement for Elastic, and that's just the first bullet. Looking down the list it is pretty easy to go: Yes, postgres can be used instead of that for extremely basic use cases, but it all goes out the window you actually need any of the power of these other tools.
The power of the other tools mostly shines in large scales. For most applications, though, performance of postgres more than suffices.
I tried to use rabbitmq for a small app, installed it, configured it and then it didn't work. Spent a day jumping through hoops getting it right.
Dumped it and used postgres, in half an hour. Worked like a charm.
Re: PostgreSQL for Everything
#94This isn't just theory either, for example: Revolut is a bank that does all its event persistence and streaming on top of postgres. No traditional message queues/brokers in their stack. https://medium.com/revolut/recording-more-events-but-where-w...
As SRE dealing with this at current company, a benefit of using well known software like Kafka is a lot of problems you will run into have solutions/guidance already available vs you having to explore solutions which a lot of time end with “Kafka could easily do this. “
Re: PostgreSQL for Everything
#95Earlier quoted context omitted.
In the early days, PostgreSQL was so much more awkward to deal with. Crash-prone at first, and then there was the whole business around having to drop the db during upgrades. It didn't really match MySQL operationally until around 2002. During the dot com era it was common to develop and launch on MySQL with the intention of migrating to something else if they became successful (though your typical LAMP stack develop…
I would agree with that, although I personally found it preferable in 2001-2002, since it matched the feature set of SQL Server in all the important ways that MySQL didn't. I think it may have been late 2000s before it was clearly preferred for new projects and even later before it was the thing you had to explain why you weren't using it. edit to add: MySQL did have operational advantages even later than mid 2000s s…
Oh yeah, we were early adopters of MySQL's master/master replication at an online retailer. Certainly an "interesting" experience, and I remember becoming very well acquainted with the vagaries of the binlog!
(I remember it being heavily touted in the first edition of the O'Reilly "High Performance MySQL" book. The second edition was about twice the length, with most of the additional pagecount going into detailed explanations of why you should actually be very careful and do lots of testing before deciding to rely on master/master!)
Actually, I remember that one of the drivers away from MySQL before Oracle came along was the 4.x and 5.x period, when there were various performance regressions. And even when those were sorted out, the introduction of InnoDB made people realise that MySQL's apparent speed advantage was really just down to MyISAM lacking referential integrity and transactions.
Certainly, there were people clinging on to MySQL 3.23 for read-heavy data warehousing applications for a very long time.
Re: PostgreSQL for Everything
#96Can Partitioning be used to move data to S3 Storage, for long term archiving?
No, and with timescaledb is a feature of their cloud platform. For on prem you can use something like Arc: https://github.com/Basekick-Labs/arc
Thank you.
I have decided to use clickhouse with that config because of missing S3 for logs and metrics for long term store.
s3
https://S3-EndPoint/{{ audit_bucket_name }}/clickhouse/
{{ clickhouse_audit_s3_access_key }}
{{ clickhouse_audit_s3_secret_key }}
default
audit_s3
Re: PostgreSQL for Everything
#97This kind of post (Postgres! It's all you need!) is getting pretty tiresome. Postgres does not even come close to a full replacement for Elastic, and that's just the first bullet. Looking down the list it is pretty easy to go: Yes, postgres can be used instead of that for extremely basic use cases, but it all goes out the window you actually need any of the power of these other tools.
The power of the other tools mostly shines in large scales. For most applications, though, performance of postgres more than suffices. I tried to use rabbitmq for a small app, installed it, configured it and then it didn't work. Spent a day jumping through hoops getting it right. Dumped it and used postgres, in half an hour. Worked like a charm.
Sure, best not to overcomplicate early if you don't need it.
PG is great and I work with it daily, but it's also not a problem to think about scale early and at least have a notional plan for what to and how to know when scale is becoming an issue in your system as you're designing it. Even PG is overkill and sqlite is more than enough for some of my projects.
There are a lot of specialized tools available, but you definitely don't need to put every one in your toolbox. Experience and observation help you make those edits -- and of course there's almost always room for improvement, but "good enough" definitely exists (until it doesn't anymore :D).
Re: PostgreSQL for Everything
#98How do you implement HA in postgres, i found MySQL HA stack pretty straight forward with Innodb cluster, MySQL router and Shell.
https://patroni.readthedocs.io/ But I sure wish it was 'core' and we didn't have to worry about it potentially going away, becoming de-supported..
Yes, especially something that touches DB. Router and shell are stupidly simple and you get auto failover in some 30 minutes. That is something make me stick to mysql.
Re: PostgreSQL for Everything
#99Earlier quoted context omitted.
The main issue with SQLite is the very poor type system, after testing it for an app I was shocked.
Are you saying that STRICT was insufficient for you? What more did you want beyond one of: INT, INTEGER, REAL, TEXT, BLOB, ANY? https://sqlite.org/stricttables.html
Want SQL-92 standard DATE/TIME/TIMESTAMP.
Re: PostgreSQL for Everything
#100I tend to agree with quite a few points in the article, but some topics warrant some careful scrutiny. * As a message queue: Only if your required features are very basic, like if you need cluster communication and run your own coordination protocol on top. * High Volume Time Series: TimeScale works, but composes badly with other workloads on the same DB server ( from an operational perspective at scale ) * Vector Da…
Also to note: (Not a fault of PGVector again just a limit of our algorithmic knowledge) PGVector does HSNW or IVFlat indices ... (there is nothing better persistent) however it breaks down with high latency at LARGE amounts of vectors ( 100MIO+ ) that seems like a high ceiling, but when designing production RAG systems, you tend to do per chunk embeddings, or even visual patch embeddings... e.g one page of a document…
I would keep a per-document summary, then dive down only into the filtered set. This is more production grade than selecting from billions of chunks.