Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving a billion Postgres rows on a $100 budget

blog.peerdb.io

Post reply on HN
Print view View original

11–20 of 71 posts

Re: Moving a billion Postgres rows on a $100 budget

#11

Post by __s »

happytiger wrote: post #4
twelfthnight wrote: post #3

If someone is price conscious, why move from postgres to snowflake?

That was my first thought as well — like who is the audience for this? My first thought was that you could stay on Postgres and save that $100 by using the secret power of Open Source. Well said.

Seems like the title would be more accurate specifying postgres to snowflake

In the end, there's companies paying to use snowflake, & despite what one may believe they aren't price oblivious. Having their application in postgres is a cost optimization, but then still relying on snowflake for data warehouse integrations

Re: Moving a billion Postgres rows on a $100 budget

#12

Post by saisrirampur »

happytiger wrote: post #4
twelfthnight wrote: post #3

If someone is price conscious, why move from postgres to snowflake?

That was my first thought as well — like who is the audience for this? My first thought was that you could stay on Postgres and save that $100 by using the secret power of Open Source. Well said.

Based on my experience working with Postgres users since a decade (ex-Citus/Microsoft), I don't think Postgres is there yet to support every possible workload - ex: medium to larger scale (ex:1TB+) workloads in Real Time analytics, Data Warehousing, Search etc. Sure at smaller scales, it is very versatile to support any workload. That is why it is super common for companies to complement Postgres with other data stores. Don't mean to say that Postgres will not get there, I think it will! But I see it to be more in the long term.

Re: Moving a billion Postgres rows on a $100 budget

#13

Post by hipadev23 »

> Moving 1 billion rows is no easy task

This isn't an accurate premise. Modern OLAP databases make dealing with billions to trillions of rows manageable, including on a single server. Exporting "select * from table" from an OLTP such as Postgres or MySQL into an OLAP is trivial and quite fast, and if 100M rows/sec on commodity servers isn't fast enough, there's always performance tuning [1].

[1] https://altinity.com/blog/loading-100b-rows-in-minutes-in-al...

Re: Moving a billion Postgres rows on a $100 budget

#14

Post by tharakam »

I'm confused. Everything sounds very expensive to me.

The last table which compares it with the other vendors is surprising. Even Stich Data (cheapest) costs $1 to move 240K records: (1B / 4,166.67 = 240K). Is this real?

So, their solution costs $1 to process 13.6M records. Sounds like this is not very share-worthy.

What I'm missing here?

Re: Moving a billion Postgres rows on a $100 budget

#16

Post by saisrirampur »

hipadev23 wrote: post #13

> Moving 1 billion rows is no easy task This isn't an accurate premise. Modern OLAP databases make dealing with billions to trillions of rows manageable, including on a single server. Exporting "select * from table" from an OLTP such as Postgres or MySQL into an OLAP is trivial and quite fast, and if 100M rows/sec on commodity servers isn't fast enough, there's always performance tuning [1]. [1] https://altinity.com/…

That sentence is more in the context of the blog of moving a billion rows across data-stores. We will edit it to make that more clear. Thanks for the feedback.

Re: Moving a billion Postgres rows on a $100 budget

#17

Post by jmholla »

This was a disappointing article. It was expecting it to explore validating the integrity and consistency of the data, but that just seems to be handwaved away by a short section saying PeerDB handles it. This is especially disappointing since the article calls that out as one of the cruxes that makes this so difficult.

Re: Moving a billion Postgres rows on a $100 budget

#18

Post by hipadev23 »

saisrirampur wrote: post #16
hipadev23 wrote: post #13

> Moving 1 billion rows is no easy task This isn't an accurate premise. Modern OLAP databases make dealing with billions to trillions of rows manageable, including on a single server. Exporting "select * from table" from an OLTP such as Postgres or MySQL into an OLAP is trivial and quite fast, and if 100M rows/sec on commodity servers isn't fast enough, there's always performance tuning [1]. [1] https://altinity.com/…

That sentence is more in the context of the blog of moving a billion rows across data-stores. We will edit it to make that more clear. Thanks for the feedback.

I can't speak for Snowflake, but on Clickhouse it's quite literally [1]:

    insert into new_table select * from postgresql('postgres:5432', 'db', 'table', 'user', 'pass');
I assume it's similarly easy on Snowflake, Databricks, SingleStore, and the rest.

[1] https://clickhouse.com/docs/en/sql-reference/table-functions...

Re: Moving a billion Postgres rows on a $100 budget

#19

Post by quickslowdown »

rplnt wrote: post #7

(edit: mostly offtopic observation follows) I only knew Snowflake the id selection algorithm, so was a bit confused, but googling "snowflake db" showed me this blurb and now I'm even more confused. > Snowflake enables organizations to learn, build, and connect with their data-driven peers. Collaborate, build data apps & power diverse workloads in the ...

Snowflake & Azure Synapse are competing products, if that helps.

Re: Moving a billion Postgres rows on a $100 budget

#20

Post by xenator »

saisrirampur wrote: post #12
happytiger wrote: post #4

Earlier quoted context omitted.

That was my first thought as well — like who is the audience for this? My first thought was that you could stay on Postgres and save that $100 by using the secret power of Open Source. Well said.

Based on my experience working with Postgres users since a decade (ex-Citus/Microsoft), I don't think Postgres is there yet to support every possible workload - ex: medium to larger scale (ex:1TB+) workloads in Real Time analytics, Data Warehousing, Search etc. Sure at smaller scales, it is very versatile to support any workload. That is why it is super common for companies to complement Postgres with other data stor…

We operate with 80 Tb of data ATM. It is laying in several nodes and meta nodes (this is our own terminology). All Postgres.

Recently we need to move data from one DB to another, about 600M records. It is not biggest chank of the data, but we need it on different server because we use FTS a lot. And don't want to interrupt other operations on previous server. It took 3 days and costs 0.

Post reply on HN