Live data from Hacker News

How to move from Amazon RDS to a dedicated PostgreSQL server

layer0.authentise.com

51–60 of 73 posts

Re: How to move from Amazon RDS to a dedicated PostgreSQL server

#51
post #33

>To my surprise, PostgreSQL v9.5 on Ubuntu 14.04 does not default to unicode schema. PostgreSQL does (if unicode character set was specified when PostgreSQL database cluster is initialized first time); it's Ubuntu OS which creates PostgreSQL database cluster with ASCII character set (encoding) after PostgreSQL's installation. > Here is the snippet that I had to use: ... Instead of resorting to those hacks, follow the…

actually that is not true for ubuntu 16.04 there it will use the locale setting:

newly setup:

                                      List of databases
       Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   
    -----------+----------+----------+-------------+-------------+-----------------------
     postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
     template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
               |          |          |             |             | postgres=CTc/postgres
     template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
               |          |          |             |             | postgres=CTc/postgres
    (3 rows)

Re: How to move from Amazon RDS to a dedicated PostgreSQL server

#52
It would be interesting to hear the motivation of why they did this. My own experience is moving the opposite direction (from master-slave self-managed to RDS master-replica).

I wrote up my experiences and the reasons for the switch here: https://www.theguardian.com/info/developer-blog/2016/feb/04/... .

Re: How to move from Amazon RDS to a dedicated PostgreSQL server

#53
post #12

From the article (emphasis mine): > [pg_dump ...] The main disadvantage of this method is that it will not provide high reliability . Pretty sure the author means "availability" not "reliabilitiy". pg_dump is completely reliable, arguably more so than ANY other backup mechanism as it creates logical machine independent backups. Best intro tip regarding pg_dump: use -Fc (custom format)

Yes, and when you are using pg_restore use the -j flag! (restore in parallel).

Re: How to move from Amazon RDS to a dedicated PostgreSQL server

#54
post #31
post #22

OK a bit of topic, but just curious we are considering rolling Postgres hosted service that would support much large instances up 50TB much higher IOPS 1,000,000+ lower latency (infiniband vs ethernet SAN) pricing would be in line with AWS RDS +/- optional ElasticSearch integration optional change notification pub/sub Would you guys be interested/think such service is a good idea? Would appreciate to learn what you g…

When it comes to data, trust is key. Think about it this way: OCZ has on paper SSDs that are simultaneously both faster and cheaper than, say, Intel. But I wouldn't put a OCZ drive into the lowest budged ricer gaming PC I could imagine (let's just say that have a bit of a reputation, to put it lightly), and I should be rightfully fired if I suggested putting one in a server. Yes, your service might be massively faste…

The latest OCZ Trion series at least honor O_DSYNC correctly. You get shitty write performance, but it is the same for Samsung 950 Pro. The read performance is another story for OCZ though :(

Anyway enterprise SSD's have so much overall better performance that consumer grade SSD's is something I consider a joke. A joke even for consumer applications, web-surfing and email-clients. Remember that a lot of applications out there use SQLite, and poor DB performance means poor application performance.

I highly recommend reading this: http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test...

Re: How to move from Amazon RDS to a dedicated PostgreSQL server

#55
post #33

>To my surprise, PostgreSQL v9.5 on Ubuntu 14.04 does not default to unicode schema. PostgreSQL does (if unicode character set was specified when PostgreSQL database cluster is initialized first time); it's Ubuntu OS which creates PostgreSQL database cluster with ASCII character set (encoding) after PostgreSQL's installation. > Here is the snippet that I had to use: ... Instead of resorting to those hacks, follow the…

> Instead of resorting to those hacks, follow the PostgreSQL documentation[1] to do it the right way. The simplest way is to initialize your PostgreSQL database cluster as:

This is actually poor advice. The right way on Ubuntu is to use the packages provided by PostgreSQL Development Group (https://wiki.postgresql.org/wiki/Apt), and in that setup you use pg_createcluster instead of initdb.

The snippet in the original article was so hacky because it fixed an existing cluster instead of destroying it and creating a new one in UTF-8.

Re: How to move from Amazon RDS to a dedicated PostgreSQL server

#56
post #44

Earlier quoted context omitted.

> Using the distro-supplied packages for PG > (or any database server) Or almost any of your application environment, really.

Distro supplied packages are a handy way of receiving timely security fixes without keeping too close an eye on recent CVEs + application changelogs.

For regular packages, sure. But for highly-active, well-maintained codebases like postgres – who package their software for your OS, upstream is going to be quicker with security fixes than distro.

Re: How to move from Amazon RDS to a dedicated PostgreSQL server

#57
post #33

>To my surprise, PostgreSQL v9.5 on Ubuntu 14.04 does not default to unicode schema. PostgreSQL does (if unicode character set was specified when PostgreSQL database cluster is initialized first time); it's Ubuntu OS which creates PostgreSQL database cluster with ASCII character set (encoding) after PostgreSQL's installation. > Here is the snippet that I had to use: ... Instead of resorting to those hacks, follow the…

> Instead of resorting to those hacks, follow the PostgreSQL documentation[1] to do it the right way. The simplest way is to initialize your PostgreSQL database cluster as: This is actually poor advice. The right way on Ubuntu is to use the packages provided by PostgreSQL Development Group ( https://wiki.postgresql.org/wiki/Apt ), and in that setup you use pg_createcluster instead of initdb. The snippet in the origin…

> This is actually poor advice. The right way on Ubuntu is to use the packages provided by PostgreSQL Development Group (https://wiki.postgresql.org/wiki/Apt), and in that setup you use pg_createcluster instead of initdb.

initdb is part of standard PostgreSQL utilities/tools[1] for initializing your PostgreSQL server that's why I suggested using it. On the other hand, I don't see any reference to pg_createcluster command in the official PostgreSQL documentation. So I don't know why you think using a non-standard PostgreSQL tool (in place of a standard one i.e. initdb) is the right way of initializing a PostgreSQL database cluster.

[1] https://www.postgresql.org/docs/9.5/static/reference-server....

Re: How to move from Amazon RDS to a dedicated PostgreSQL server

#58
post #51
post #33

>To my surprise, PostgreSQL v9.5 on Ubuntu 14.04 does not default to unicode schema. PostgreSQL does (if unicode character set was specified when PostgreSQL database cluster is initialized first time); it's Ubuntu OS which creates PostgreSQL database cluster with ASCII character set (encoding) after PostgreSQL's installation. > Here is the snippet that I had to use: ... Instead of resorting to those hacks, follow the…

actually that is not true for ubuntu 16.04 there it will use the locale setting: newly setup: List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges -----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres + | | | | | postgres=CTc/postgres…

It's good news that they've changed the default PostgreSQL character set (encoding) from ASCII to UTF8 in Ubuntu 16.04. It'll definitely save a good amount of time of new PostgreSQL users who need UTF8 encoding as default.

Re: How to move from Amazon RDS to a dedicated PostgreSQL server

#59
post #22

OK a bit of topic, but just curious we are considering rolling Postgres hosted service that would support much large instances up 50TB much higher IOPS 1,000,000+ lower latency (infiniband vs ethernet SAN) pricing would be in line with AWS RDS +/- optional ElasticSearch integration optional change notification pub/sub Would you guys be interested/think such service is a good idea? Would appreciate to learn what you g…

My priorities are: Reliability > Size ~= Price > Performance

Reliability of a database is always no.1, without it, nothing else matters. So the customer must have enough access to the machine to setup replication and failover themselves (eg. using something like repmgr). Or have damn good guarantees that you're doing it well for them. ie. multiple replicas, verifiable backups, offsite backups, etc.

Secondly is size. This is a major sticking point for me when looking at hosted postgres solutions - they're mostly for relatively small databases Closely in third is price. Even when you do find a service that provides a decent size like when RDS eventually went up to 6Tb (still not quite enough for me), the price they charge for it (and the ram) over buying dedicated iron is exorbitant. Given that 6-8Tb drives are mainstream now, it's a real head scratcher why they'll charge $750-1500/mth for a 6Tb drive that costs $250 outright. And then charge you for the IOPS on top of that... Even factoring in multiple redundant drives/replicas, SSD vs HDD, it's still very unfavourable. I can install 5 machines (at least) in a cluster for the price of 1 on RDS; giving me not just price savings, but my no.1 need - reliability.

Finally performance. The number of high IOPS workloads out there is rather small. You're far more likely to have a high storage need with small IOPS, than high IOPS, smaller storage. Everyone overestimates the amount of traffic they'll get, and generally if you have an IOPS problem, it's better off fixed elsewhere in the app. Usually some bad code or ORM is thrashing the DB.

Take a simple login DB as a thought experiment, just 10,000 IOPS is upto 864 million users logging in per day. Whereas the storage for that number of users, at a very generous 1Kb per user, is 864Gb. Size is way more important than IOPS. And if you've got that number of users, you've also got your own datacentre :)

You might also run into the problem that the kind of organisations that need high storage + high iops are also the kind that wouldn't use the cloud for it in a million years (banks, high freq. traders, etc).

In summary, I think the main thing that would attract me is high capacity storage for a decent price. Reliability is a given, but it must be easy to manage. Ideally I'd like a mixed storage system, with the ability to arrange the tablespaces across the drives as I need. 50Tb is great, but I need the ability to put the high IOPS tables on an SSD, then spread the infrequently accessed tables onto large HDDs.

Re: How to move from Amazon RDS to a dedicated PostgreSQL server

#60
post #32

Earlier quoted context omitted.

Thanks for the background info, very helpful. How has on-prem install gone for you guys ? There was an article on HN yesterday suggesting it was a royal PITA (for ops and support), would be interesting to hear about your experience. Also FYI, your homepage has a 2015 copyright notice on it. No big deal, but thought you might like to know since it could put off particularly "nit picky" types of customer :)

> Also FYI, your homepage has a 2015 copyright notice on it. No big deal, but thought you might like to know since it could put off particularly "nit picky" types of customer :) If that's when the work was created, then that's correct. Copyright notices aren't there to tell you what year it is today. They are there to tell you when the work was created. If they change it to 2016 when the work was really created in 20…

Exactly. I've actually had a manager tell me to update a copyright notice from 2016 to 2015. These are the real nitpickers.
Post reply on HN