Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A list of PostgreSQL libraries, tools and resources

github.com

Post reply on HN
Print view View original

1–10 of 30 posts

Re: A list of PostgreSQL libraries, tools and resources

#3

Post by pilif »

With regards to psql2csv: The default psql can already do this very nicely. Just use \copy.

\copy (select whatever from whatever) to 'yourlocalfile.csv' with (format 'csv')

and if you want column headers, add a "header true" inside of the with clause.

Generally, \copy works just like COPY[1] but it does so from the remote server to the local machine, whereas file names given to COPY are relative to the server.

Yes. A dedicated tool might feel easier initially, but once you know how \copy works, you can always get a CSV file from whatever database you're connected to and no matter what machine you're on.

[1]: http://www.postgresql.org/docs/current/static/sql-copy.html

Re: A list of PostgreSQL libraries, tools and resources

#4

Post by dang »

davidw wrote: post #2

Can't it just be a 'list' of Postgres stuff? Do we really expect that it's generated by an algorithm if someone doesn't write 'curated'? If it's a good list, I'm going to find it interesting no matter how it was created.

We took 'awesome' and 'curated' out the title, as per usual.

Re: A list of PostgreSQL libraries, tools and resources

#5

Post by Cieplak »

What is the best way to ETL data from one Postgres cluster to another (by ETL I mean things like data sanitization and denormalization)?

a) I think most people end up doing some kind of batch synchronization, but I'm interested in streaming solutions.

b) A lot of folks use trigger based replication, but triggers have to be on the primary/master node, and not just on the replicas.

c) Another common solution is to force the database client to write to a message broker, but that opens the door to data discrepancies and synchronization issues.

d) In theory I think the best way is to do something like bottledwater-pg or pg_kafka [1] [2] [3], but I'm not sure how battle hardened these are. I think logical replication of the WAL is the right approach, but there is still not much tooling around this.

[1] https://github.com/confluentinc/bottledwater-pg

[2] https://github.com/xstevens/pg_kafka

[3] https://github.com/xstevens/decoderbufs

PS: There are a bunch of interesting MySQL solutions out there, such as Zendesk's Maxwell:

https://github.com/zendesk/maxwell

Re: A list of PostgreSQL libraries, tools and resources

#6

Post by [deleted] »

Cieplak wrote: post #5

What is the best way to ETL data from one Postgres cluster to another (by ETL I mean things like data sanitization and denormalization)? a) I think most people end up doing some kind of batch synchronization, but I'm interested in streaming solutions. b) A lot of folks use trigger based replication, but triggers have to be on the primary/master node, and not just on the replicas. c) Another common solution is to forc…

[deleted]

Re: A list of PostgreSQL libraries, tools and resources

#8

Post by davidw »

dang wrote: post #4
davidw wrote: post #2

Can't it just be a 'list' of Postgres stuff? Do we really expect that it's generated by an algorithm if someone doesn't write 'curated'? If it's a good list, I'm going to find it interesting no matter how it was created.

We took 'awesome' and 'curated' out the title, as per usual.

Thanks for your awesome curation of this site :-)

Re: A list of PostgreSQL libraries, tools and resources

#9

Post by hvo »

dang wrote: post #4
davidw wrote: post #2

Can't it just be a 'list' of Postgres stuff? Do we really expect that it's generated by an algorithm if someone doesn't write 'curated'? If it's a good list, I'm going to find it interesting no matter how it was created.

We took 'awesome' and 'curated' out the title, as per usual.

Yes. Thanks for your great work on this site.Keep up the good work.
Post reply on HN