Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Omnigres: Postgres as a Platform

github.com

Post reply on HN
Print view View original

41–50 of 77 posts

Re: Omnigres: Postgres as a Platform

#41

Post by MrBuddyCasino »

rtpg wrote: post #14

This sort of stuff (and a lot of other popular things) highlights something that I've felt for years: we are still so far away from a good ops story as an industry. Why would someone use this? Because they only want to deal with "one thing" in production. Dealing with multiple stacks is annoying. People show stuff like kubernetes as a way to handle multiple stacks. That stuff is also complicated. There is still a hug…

> There is still a huge opportunity for a methodology (accessible to non-operators) to get systems in production easily, where we wouldn't feel the need to force it all into these kinds of models.

Nomad is a simpler way than K8S to run not just containers, but plain old binaries. Can’t beat K8S network effect. Technical merit or economical efficiency are secondary factors in tech trends.

Re: Omnigres: Postgres as a Platform

#42

Post by lovasoa »

jordanthoms wrote: post #38

This is an interesting project, but I'd be unlikely to use it, for a couple of reasons: - In my experience the performance of the stateful DB server has been the biggest bottleneck when scaling - it's much easier to scale the stateless application servers which sit between end user requests and your DB in a traditional architecture. So usually I'm wanting to move as much work as possible away from the DB in order to…

These are good points, but I'd like to nuance some of them. Disclaimer: I work on an SQL web application builder [1] that shares a lot of the philosophy behind Omnigres.

- about scaling: you have to get very far before saturating a single postgres server. A lot of applications certainly do get to that point, but most don't. And once you get there, scaling postgres is definitely more work than scaling a stateless service, but it also gives you a lot more in terms of performance, reliability and further scalability.

- about C being scary: as a rust afficionado, I am not going to contradict you. But postgres itself is already C, and @yrashk is not just any C developer. Notably, he contributes to postgres itself.

- about managing complexity: postgREST, Omnigres, hasura, SQLPage and other tools that simplify building directly on top of the database never require exclusive access to the database. You can always put some of the complexity outside if you need to, when you need to.

[1] https://sql.ophir.dev

Re: Omnigres: Postgres as a Platform

#43

Post by deep_sea03 »

Oh wow, this is great. Congratulations Yurii.

So i Tried omnigress to build a simple api that does few db calls a month back to mock real world scenario. And damn, the numbers were quite thought provoking. Comparing those numbers with a similar api built using fastapi and asyncpg, there was if i remember correctly at least 4x to 5x throughput increase.

Even though the project is in quite nascent stage, the idea is promising. This tackles the concept or should i rephrase, pain of having multiple platform to run a service and the needed expertise that is necessary.

If the scaling part and dev convenience can be fully sorted, and also the security aspect like many have highlighted, this could very well have the potential to change the game.

Re: Omnigres: Postgres as a Platform

#45

Post by factormeta »

lovasoa wrote: post #42
jordanthoms wrote: post #38

This is an interesting project, but I'd be unlikely to use it, for a couple of reasons: - In my experience the performance of the stateful DB server has been the biggest bottleneck when scaling - it's much easier to scale the stateless application servers which sit between end user requests and your DB in a traditional architecture. So usually I'm wanting to move as much work as possible away from the DB in order to…

These are good points, but I'd like to nuance some of them. Disclaimer: I work on an SQL web application builder [1] that shares a lot of the philosophy behind Omnigres. - about scaling: you have to get very far before saturating a single postgres server. A lot of applications certainly do get to that point, but most don't. And once you get there, scaling postgres is definitely more work than scaling a stateless serv…

>- about scaling: you have to get very far before saturating a single postgres server. A lot of applications certainly do get to that point, but most don't. And once you get there, scaling postgres is definitely more work than scaling a stateless service, but it also gives you a lot more in terms of performance, reliability and further scalability.

Thank you for saying this. Mostly when postgres as a platform is brought up, the horizontally scaling ppl will often mention the parent thread. I have being developing since Apple computer had floppy disk, and there is rarely many situations where I need to saturating a single postgres server.

And even if one did get to the situation where that happens, with introduction of hydra, or other postgers columnar db, we can just put that in. Most user will never get to a point where they need to saturating a single postgres server. And also keep in mind when processing large row data, writting stuff in middleware is just not as efficent or fast as in postgres when it has native access to data and data manipulation.

Re: Omnigres: Postgres as a Platform

#46

Post by vira28 »

deep_sea03 wrote: post #43

Oh wow, this is great. Congratulations Yurii. So i Tried omnigress to build a simple api that does few db calls a month back to mock real world scenario. And damn, the numbers were quite thought provoking. Comparing those numbers with a similar api built using fastapi and asyncpg, there was if i remember correctly at least 4x to 5x throughput increase. Even though the project is in quite nascent stage, the idea is pr…

Those numbers sound similar to what the author benchmarked in the past. https://yrashk.com/blog/2023/02/16/what-happens-if-you-put-h...

Re: Omnigres: Postgres as a Platform

#48

Post by ezekiel68 »

At some point during the dot-com era, Microsoft toyed with adding an HTTP server to its SQL Server product to enable similar development paradigms. My recollection is that this was a disaster on several fronts, not the least of which was that successfully breaching the HTTP server gave an attacker direct access to the database. This (and other considerations) make a good case for the kind of separation-of-concerns which would avoid entwining API service with DB access.

I read the README, but it focuses on the practical aspects of getting the solution to work, not its philosophical justification.

edit: When I say "disaster" for the MS SQL Server attempt, I don't believe anyone ever experienced a breach, etc. But, as I recall, there were no large customers (MS caters to the larger corporate crowd) with IT deparments willing to risk exposing their DB servers to the "DMZ zone" of public Internet access (or only 1 layer past is, such as behind a proxy). So from that standpoint the disaster was providing a solution nobody (with money and corporate experience) wanted. I guess for hobbies or Silicon Valley MVPs, the concept might have some legitimate appeal.

Re: Omnigres: Postgres as a Platform

#49

Post by jordanthoms »

lovasoa wrote: post #42
jordanthoms wrote: post #38

This is an interesting project, but I'd be unlikely to use it, for a couple of reasons: - In my experience the performance of the stateful DB server has been the biggest bottleneck when scaling - it's much easier to scale the stateless application servers which sit between end user requests and your DB in a traditional architecture. So usually I'm wanting to move as much work as possible away from the DB in order to…

These are good points, but I'd like to nuance some of them. Disclaimer: I work on an SQL web application builder [1] that shares a lot of the philosophy behind Omnigres. - about scaling: you have to get very far before saturating a single postgres server. A lot of applications certainly do get to that point, but most don't. And once you get there, scaling postgres is definitely more work than scaling a stateless serv…

These are also good points you've raised.

On scaling, yeah a single postgres server can handle a lot. For us we were well past the million user mark before running into serious issues. However, a lot of how we were able to keep postgres working for us as we grew was by shifting work from postgres to our stateless services like I alluded to before - e.g. making our SQL queries as simple to execute as possible even if it means more work for the client to piece the parts back together.

If everything had been running inside the database we wouldn't have had that option and we'd probably have hit scaling limits much earlier - I guess we could have split off the traffic to the highest traffic endpoints and have those handled by a separate service calling the PG db, but then you get into issues with keeping the authentication etc consistent.

Re security - yep, PG is already using C to parse untrusted inputs from the network, which is also scary, but it's (hopefully) well reviewed and mature code - and even so, I wouldn't want to expose PG's usual wire protocol port to the internet, so it's hard to imagine exposing HTTP from postgres to the wild west.

Ultimately it probably is just a question of the sort of project it's being used for - if it's for something that's not going to get need to get to larger scales, handle a lot of complexity over time, or pass security reviews and your main goal is simplicity, then maybe an approach like this is a good option. I've just found that things tend to start off looking small and simple and then turn out to be anything but, so I'd rather run `rails new` and point it at a standard PG server - which would be just as simple and productive when you are starting out, and can keep scaling as your customer base and team size grows up to the size of Shopify, Github, or Kami (shameless plug).

Post reply on HN