101–110 of 126 posts
Will this work alongside Citus? If yes, what benefits/disadvantages would it have? I presume at some point of time workloads do need a cluster.
With what we're working in v0.2, we re-architecture some of our dependencies there, and will be able to scale out and support Citus.
As others have mentioned the results are from an extension, not postgres. Unfortunately using an extension that's not "part" of postgres (like pg_stat_statements) is not trivial for most people since there are both technical and legal issues.
It should be fairly trivial to get started
As others have mentioned the results are from an extension, not postgres. Unfortunately using an extension that's not "part" of postgres (like pg_stat_statements) is not trivial for most people since there are both technical and legal issues.
What are th legal issues?
Even if the extension has a liberal licence eg Apache or MIT nothing guarantees that the licence won't change in the future.
As others have mentioned the results are from an extension, not postgres. Unfortunately using an extension that's not "part" of postgres (like pg_stat_statements) is not trivial for most people since there are both technical and legal issues.
What are th legal issues?
Others, like Citus, are licensed under the AGPL, which is basically an even stronger version of the GPL that requires anyone hosting a service to provide their modifications to the source code. While this doesn't strictly rule out use in e.g. RDS it does basically say that Amazon would be required to open source any changes they make to get it working in RDS, which is a burden they don't seem interested in taking on, and which might reveal details about how their services work under the hood which they are generally fairly cagey about doing.
The AGPL is also viewed even more suspiciously than the GPL by corporate lawyers who fear it may lead to virally forcing the company to reveal all of its source code. This fear is probably unfounded, but all copyleft licenses are primarily about expanding end-user freedom and so are treated as undesirable in most businesses that make money from selling software.
Earlier quoted context omitted.
> This is a solved problem, and it seems the technical folks over there lacked the skills to make it work. Having indexes is just the tip of the iceberg. Composite indexes, partitioning, sharding, caching, etc, can lower reads to a few seconds on disk. Or just use BigQuery and it is works, it is cheaper to run (by 10x to 100x) and can be done by a junior dev rather than a PhD in Database configuration. I prefer simpl…
I have 50 columns in a table with 10B records in a partitioned table. People can search using any combination. There are a huge number of possible composite indexes, so we don't have any, but we have every column indexed. Most queries take a minute or two, and we also experienced a sudden drop in performance even for queries that used to work OK. Not sure what to do next to improve query speed.
For this workload, having a columnstore version of your table will help.
DM us: https://join.slack.com/t/mooncakelabs/shared_invite/zt-2sepj.... We can help.
Earlier quoted context omitted.
> This is a solved problem, and it seems the technical folks over there lacked the skills to make it work. Having indexes is just the tip of the iceberg. Composite indexes, partitioning, sharding, caching, etc, can lower reads to a few seconds on disk. Or just use BigQuery and it is works, it is cheaper to run (by 10x to 100x) and can be done by a junior dev rather than a PhD in Database configuration. I prefer simpl…
I have 50 columns in a table with 10B records in a partitioned table. People can search using any combination. There are a huge number of possible composite indexes, so we don't have any, but we have every column indexed. Most queries take a minute or two, and we also experienced a sudden drop in performance even for queries that used to work OK. Not sure what to do next to improve query speed.
Sai from PeerDB/ClickHouse here. Nice to see the progress on this project! I wanted to leave a few notes: For analytics on transactional data, it looks like you'd still need to use logical replication ( https://github.com/Mooncake-Labs/pg_mooncake/issues/90 ). Logical replication is somewhat similar to an ETL/CDC experience, though it's more Postgres-native. Managing logical replication at a production grade isn't tr…
Hi, Zhou From Mooncake labs here. Love your work on PeerDB and it's inspiring the evolvement of pg_mooncake (logical replication will be the killing feature for V2) The core idea of mooncake is to built upon open columnar format + substitutable vectorized engine, while natively integrate with Postgres: 1. For small devs, we allow the whole stack to be embedded as a Postgres extension for ease of use 2. For enterprise…
1, makes sense.
On 2, I understand your thinking around purpose-built — but you're retrofitting an analytical database into a transactional database without fully supporting all the features (both in terms of functionality and performance) of either. It's really hard to be truly purpose-built this way. As a result, users might not get the best of both worlds.
PeerDB is different. We keep Postgres and ClickHouse separate and just move data reliably between them. Users get to query Postgres and ClickHouse in isolation and make the best of each of them.
Anyway, keep up the good work! Just wanted to share some challenges we've seen before when building an analytics extension (Citus), particularly around chasing both Postgres compatibility and performance.
Earlier quoted context omitted.
That and RDS are kind of a rip-off if you are trying to have a sandbox out there, short of running your own instance. For that, I found Digital Ocean to be very reasonable.
DO’s pricing has increased a lot. I was surprised, I couldn’t start a $5 vm anymore, it’s worth checking again.
Earlier quoted context omitted.
As someone dealing with billions of records on it, BigQuery is far from cheap; G will not charge you much for storage as they will charge you for queries and data transfer. AFAIK, the cheapest Postgres server on GCP is very expensive compared to the usual Postgres installation (price/performance).
Yep. I used it once for a personal project involving GHTorrent [0], and each query was about $20. That may be peanuts for a business, but for someone playing around on their own, it's incredibly prohibitive. [0]: https://github.com/ghtorrent/ghtorrent.org/blob/master/gclou...
Earlier quoted context omitted.
> Did you use plain Postgres tables or a columnar extension like Timescale or pg_mooncake? Plain Postgres tables hosted on Google Cloud SQL on a decent machine. > Nobody in their right mind would argue that Postgres without columnar storage is good for analytics. Which is what I am saying. Standard means without extensions. > Personally I'm a huge fan of dedicated databases like ClickHouse. Where do you run it? What…
Your comment is confusing to me because it appears to have nothing to do with the article, which is not about using Postgres' heap tables. In fact, you're arguing for a purpose-built storage engine (BigQuery), which is what pg_mooncake also is, except the latter is integrated with Postgres. At my company we self-host ClickHouse, but if we started out today we'd definitely adopt ClickHouse Cloud. One big advantage of…