101–108 of 108 posts
It seems that after lots of hyped NoSQL systems companies are still using MySQL or Postgresql as simple storage backend with lots of custom crutches over it, like it's 2007. So all these cassandaras and riaks failed expectations?
Earlier quoted context omitted.
I'm not sure what anything you said has to do with their choice of database.
Then with your own reasoning, i'm also not sure that is worth a comment of yours. I clearly indicate their recent changing of db software twice to avoid issues (experts could solve) is an indication one in my eyes is not very talented.
Earlier quoted context omitted.
That requires the master and slaves to run different versions for a while. And that is not possible with stock postgresql, is it? Regarding your second point, I meant copying the data directory as in a 'cp' command. Or rsync if you will. The functions you mentioned are only useful when doing a dump, isn't it? And recovering from a upgrade problem using a dump is way slower than just starting the previous version in t…
> That requires the master and slaves to run different versions for a while. And that is not possible with stock postgresql, is it? Yes. That's not possible. But if I announce the downtime, bring master and slave down, migrate the slave and run our test-suite, migrate the master, run the test suite again and bring the site back up, then I know whether the migration worked. If the migration on the slave fails, well, t…
Earlier quoted context omitted.
The problem with this is that if anything fails, you can potentially corrupt your data and have no backup plan. To make that option safe, you would have to copy your data directory first, and you need to be offline for that. So you have to add the time it takes to make that copy.
This is why I ensure that the slaves are up to date, then disconnect them, pg_upgrade the master and resync the slaves (which is required anyways). If something goes wrong, I would fail over to the slave. Also: You don't need to be offline to copy the data directory. Check `pg_start_backup` or `pg_basebackup` (which calls the former)
Earlier quoted context omitted.
I think the trap is simple POCS. Do a simple POC where you ram 1 million rows in a MongoDB and then make a webapp to do some basic analytics. Look, it works and get responses within a second. Cool! Then real data comes in at 12 billion rows and your analytics take 3 hours to run. So you try to do the sharding thing, and realize that it works for a while, except now every analytics query needs to hit every shard....
Absolutely true, and the scale keeps getting larger > 12 billion rows, and comes up with even harder challenges. Worked on a 1.3 trillion row (for 1 table, others varying below this) database this year for predictive analytics, and it was mind boggling the hoops to jump through to try to get that thing to run anything in a manageable time frame. Any POC level show of a database would be about meaningless at that scal…
Earlier quoted context omitted.
Absolutely true, and the scale keeps getting larger > 12 billion rows, and comes up with even harder challenges. Worked on a 1.3 trillion row (for 1 table, others varying below this) database this year for predictive analytics, and it was mind boggling the hoops to jump through to try to get that thing to run anything in a manageable time frame. Any POC level show of a database would be about meaningless at that scal…
but do you need to access all 1.3 trillion rows to do predictive analytics? why not just use a very small sample of the data?
Ideally, you'd actually use more than that, however diminishing returns leaves you with that as what we were going with.
Earlier quoted context omitted.
Absolutely true, and the scale keeps getting larger > 12 billion rows, and comes up with even harder challenges. Worked on a 1.3 trillion row (for 1 table, others varying below this) database this year for predictive analytics, and it was mind boggling the hoops to jump through to try to get that thing to run anything in a manageable time frame. Any POC level show of a database would be about meaningless at that scal…
Which database did you use and what would you recommend?
Keeping 1.3 trillion in memory is pretty expensive, so we were trying to cut some costs by using that to funnel data in and out.
Not a bad solution overall.
Earlier quoted context omitted.
Then with your own reasoning, i'm also not sure that is worth a comment of yours. I clearly indicate their recent changing of db software twice to avoid issues (experts could solve) is an indication one in my eyes is not very talented.
Actually no, you didn't clearly say that at all. How do the experts solve the fact that Postgres' rewrites entire indexes on row updates?
And why do you make the assumption, i'm a postgresql expert to answer that question.
Though others did, so if you would have bothered to read the other thousands of comments on the matter. You would not have needed to ask this question,
PostgreSQL Heap-Only-Tuples (HOT) from: http://use-the-index-luke.com/blog/2016-07-29/on-ubers-choic...