Does the elastic license fall under open source as defined by OSI?
No it does not.
Moving a billion Postgres rows on a $100 budget
41–50 of 71 posts
Re: Moving a billion Postgres rows on a $100 budget
#42Earlier quoted context omitted.
Or wouldn't COPY from CSV be much faster?
CSV based export/import would involve writing to disk, sending over network then reading back from file. Select is just one copy over network, no intermediate CSV.
You can stream CSV without writing it to a disk.
Re: Moving a billion Postgres rows on a $100 budget
#43Re: Moving a billion Postgres rows on a $100 budget
#44Re: Moving a billion Postgres rows on a $100 budget
#45Re: Moving a billion Postgres rows on a $100 budget
#46Earlier quoted context omitted.
That sentence is more in the context of the blog of moving a billion rows across data-stores. We will edit it to make that more clear. Thanks for the feedback.
data stores or data lakes with data rivers?
gotta stick to the rivers and lakes that you're used to
Re: Moving a billion Postgres rows on a $100 budget
#47I've already thought of the follow-up to this article, 'Querying a billion rows on a $XX budget.' Let me give you my answer directly: switch from Snowflake to StarRocks. It's an open-source project under the Linux Foundation, with speed that's more than adequate, especially for queries involving multiple tables. If you're interested, you might want to check it out, https://medium.com/starrocks-engineering/how-to-redu…
Starrocks sounds too good to be true, what are the cons?
Hard to set up in Kubernetes. Seems to want you to know too many specifics for how to shard your db ahead of time. Don't remember how it handles schema evolution. I settled for Databend because it just worked easy and fast with nothing more than table definitions.
Re: Moving a billion Postgres rows on a $100 budget
#48Re: Moving a billion Postgres rows on a $100 budget
#49> Moving a Billion Postgres Rows on a $100 Budget
In a magical universe where your time is free.
Re: Moving a billion Postgres rows on a $100 budget
#50Postgres seriously needs a columnstore backed table instead of just a rowstore. MSSQL has this and it is magic. SingleStore has it, and it is wonderful. I'm willing to give a bounty of $1000 to whoever adds that into main postgres tree. Snowflake is great as a warehouse. it's latency is shit when it comes to fast lookups and aggregates. If you can tolerate >1s api calls, that is fine. It takes forever to insert a few…
Columnar store PostgreSQL extension exists, here are two but I think I’m missing at least another one:
https://github.com/citusdata/cstore_fdw
https://github.com/hydradatabase/hydra
You can also connect other stores using the foreign data wrappers, like parquet files stored on an object store, duckdb, clickhouse… though the joins aren’t optimised as PostgreSQL would do full scan on the external table when joining.