Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres scaling advice

cybertec-postgresql.com

Post reply on HN
Print view View original

151–160 of 207 posts

Re: Postgres scaling advice

#151

Post by hectormalot »

jayd16 wrote: post #98
ccmcarey wrote: post #34

Earlier quoted context omitted.

I think it's easier to run a small k8s cluster than it is to attempt to recreate a lot of the functionality provided manually, especially if you're running in a cloud where the control plane is handled for you. It provides unified secrets management, automatic service discovery and traffic routing, controllable deployments, resource quotas, incredibly easy monitoring (with something like a prometheus operator). Being…

Agreed. What is it that people are doing (or not doing) where a simple managed k8s cluster is more work than the minimum way to do this? Are teams not even setting up automated builds and just sshing into a box?

For me that’s Heroku. I just push my app (RoR) and done. I’ve actually moved it once to k8s for a few months and decided to move it back after I understood how easy heroku made the opa side of the business.

Note: it’s a side project, 50 business users, $5k annual revenue, 4h per month as target for the time spent on customer support, admin and maintenance. So it’s both easy to pay heroku and important for me to not spend too much time on Ops.

Re: Postgres scaling advice

#152

Post by jeffbee »

btilly wrote: post #148
jeffbee wrote: post #117

Earlier quoted context omitted.

Indexes are not free, they take up space and they make mutations more costly. Also, building the index may not even be possible while your application is running, because postgresql and other RDBMS have inadequate facilities for throttling index construction such that it doesn't harm the online workload. You might have to build indexes at midnight on Sundays, or even take your whole system offline. It can be a nightm…

I've seen https://www.postgresql.org/docs/12/sql-createindex.html#SQL-... work well in practice on busy transactional databases. I'd be interested in knowing about cases where it doesn't work well. My experience on systems without indexes differs strongly from yours. Yes, they can work well. But if you have multiple use cases for how your data is being queried, they push you into keeping multiple copies of your data.…

Just from personal experience, if they can build gmail on top of a database (bigtable) having neither indexes nor consistency, then probably it will also be suitable for the purposes of my far smaller, much less demanding products.

On the other hand I've seen, and am currently suffering through, products that have desperate performance problems with trivial amounts (tens of GB) of data in relational databases with indexes aplenty.

Re: Postgres scaling advice

#153

Post by jarym »

atonse wrote: post #65
wiredfool wrote: post #58

Earlier quoted context omitted.

And nearly 2 million for a 3 year reserved instance.

This reminded me of a story from 15 years ago. I once worked for a company that was writing a proposal for a US Homeland Security IT system. This was 2006. I wasn't involved in it but my office-mate was. He randomly turned his chair around and said "hey, can you go on Dell.com and see if you can build the most expensive server imaginable" – so I did and I ended up at around $350k. I don't remember what it was, but at…

They got to the end of their financial year and had $$$ left to spend... happens all the time.

Re: Postgres scaling advice

#154

Post by mkilling »

claytonjy wrote: post #50
jandrewrogers wrote: post #42

The assertion that PostgreSQL can handle dozens of TB of data needs to be qualified, as this is definitely not the case in some surprising and unexpected cases that are rarely talked about. PostgreSQL's statistics collection, which is used by the query planner, doesn't scale with storage size . For some ordinary data distributions at scale, the statistical model won't reflect any kind of reality and therefore can pro…

This is intriguing; could you give an example of a distribution Postgres struggles with at scale, and an application that produces such a distribution?

We’ve recently been surprised by wrong estimates made by the planner resulting in inefficient query plans.

It seems that power law distributions over multiple tables aren’t handled very well: https://stackoverflow.com/questions/65861780/is-there-a-way-...

Re: Postgres scaling advice

#155

Post by yowlingcat »

corty wrote: post #60
zdragnar wrote: post #43

Earlier quoted context omitted.

In big enough organizations, it is very easy to lose track of who owns what, especially when it is those little ad-hoc internal tools. Manually managing the infrastructure for them is a recipe for them to become permanently enshrined in the wasteland of "services we think we use, but do not maintain because we don't remember who needed it or put it up or how to configure it". K8s isn't the only answer, but if you are…

Having a Dockerfile that copies a few binary blobs into an age-old distro image isn't an improvement, it's a huge liability. And most of that stuff that no one knows anything about anymore is like that. Same as with an old VM or PM. I'd rather have that old crap as a physical machine. Why? Because the hardware lifetime "naturally" limits the lifetime of such applications. If the hardware dies, it forces a decision to…

> the hardware lifetime "naturally" limits the lifetime of such applications

Oh, my sweet summer child. Of all the things that "naturally" limits the lifetime of such applications, that is not it. Consider the case of the mainframes running the US banking system, for example.

Re: Postgres scaling advice

#156

Post by Turbots »

majewsky wrote: post #111
pas wrote: post #96

Earlier quoted context omitted.

1.0 was released in 2015. There are stable LTS vendors for it. It's pretty established. And much saner than cobbling together Ansible/Puppet/Chef playbooks for everything.

Saying that 2021's Kubernetes is established because 1.0 was released in 2015 is like saying that 1991's Linux is stable because Unix had existed for 20 years at that point. Kubernetes 1.0 and 1.20 share the same name, design principles and a certain amount of API compatibility, but it's impossible to take a nontrivial application running on 1.20 and just `kubectl apply` it on 1.0. Too much has changed. Kubernetes is…

I know of hundreds of customers already running Kubernetes at scale (not just some clusters but hundreds) for applications in production.

Companies running 15000+ containers in production is not rare.

Kubernetes is quite stable and established in many "old skool" banks, insurance companies, governments, etc already.

There are no more software products that remain stable with same API over 20 years. Everything goes faster and faster, you need to adopt or die ;-)

Re: Postgres scaling advice

#157

Post by Turbots »

mumblemumble wrote: post #31
jjice wrote: post #13

In the opinion of a last semester CS student who has never written an application from scratch that needed more than a SQLite DB (so take me with a half grain of salt), it seems like premature optimization, while always talked about, is very common. I see people talking about using Kubernetes for internal applications and I just can't figure out why. If it's a hobby project and you want to learn Kubernetes, that's a…

It's resumé-driven development, and it's also entertainment-driven development. Bringing in new technologies gives you a chance to play with a new toy. That's an effective way to make your job more interesting when the thing you're supposed to be working on is boring. Which, in business applications, is more often than not the case.

Kubernetes is in no way "resume driven" anymore.

It is being used by the largest companies in the world at massive scale, in the public and private cloud.

Re: Postgres scaling advice

#158

Post by throwdbaaway »

wejick wrote: post #41
jjice wrote: post #13

In the opinion of a last semester CS student who has never written an application from scratch that needed more than a SQLite DB (so take me with a half grain of salt), it seems like premature optimization, while always talked about, is very common. I see people talking about using Kubernetes for internal applications and I just can't figure out why. If it's a hobby project and you want to learn Kubernetes, that's a…

I know one of the biggest Ecommerce shop in Asia were using 1 big DB with multiple read only slave in monolithic architecture for more than 5 years. However not only driven by DB performance, but also on organizing hundreds of engineers they adapted microservice architecture. Then they slowly migrating to per domain specific DB, it is just classic microservice migration story. While single DB may bring us pretty long…

Upvoted, but I am not sure tokopedia is even in the top 10 in Asia.

Also, the fascination with GMV tends to make it looks like high scalability is required. In another HN discussion, someone mentioned about running the database for an ecommerce that had 1 billions GMV a few years back. Assuming a conservative $5 per order, that translates to about 6 orders per second on average.

Re: Postgres scaling advice

#159

Post by bastawhiz »

evanelias wrote: post #110
bmcahren wrote: post #84

Earlier quoted context omitted.

Shards increase the number of failure modes and increase the complexity of those failure modes. For most businesses, the recommendation holds true... keep it simple, don't shard until you need . I find it somewhat concerning that MongoDB has a better architecture for upgrades than Postgres. You add a replica to the cluster running the new major version and then switch that replica over as your primary once you've rep…

> Shards increase the number of failure modes and increase the complexity of those failure modes. I would only agree with this during the initial implementation of sharding. Once deployed and stable, I have not found this to be the case, at all. I say this as someone who has directly architected a sharded database layer that scaled to over a trillion rows, and later worked on core automation and operations for sharde…

> In both cases, each company's non-sharded databases were FAR more operationally problematic than the sharded ones. The sharded database tiers behave in common ways with relatively uniform workloads, and the non-sharded databases were each special snowflakes using different obscure features of the database.

That's because sharded tables restrict what features you can use (e.g., no JOINs). If you constrained the features on the non-sharded databases, you'd achieve the same net result.

Sharding _necessarily_ only solves one problem: queries operate against only a subset of data. While what you're saying is true (sharding avoids certain problems) it also restricts your ability to perform other operations (more complicated queries or reports are ~impossible). It is not without its tradeoffs.

Re: Postgres scaling advice

#160

Post by takeda »

nforest wrote: post #108
pg_bot wrote: post #26

Earlier quoted context omitted.

What you're describing is called resumé driven development. It happens every few years when people want to cash in on trends/buzzwords that people believe will be disruptive to all industries but are just tools to have in the toolbox for most. New tools pop up all the time that fit this mould. Over the past ten years I can think of Hadoop (Big data), MongoDB (NoSQL), Kubernetes, "Serverless" computing, and TensorFlow…

For artificial intelligence, I think it's more often marketing driven development. It's easier to seem disruptive if you claim to have AI in your product. Easier to get funding and have people talk about your company. I feel like it comes more often from business executives than technical people.

You reminded me of the Phillips toothbrush with AI. Marketing like this makes the term AI worthless.
Post reply on HN