Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

pganalyze.com

Post reply on HN
Print view View original

151–159 of 159 posts

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#151

Post by hylaride »

ttfkam wrote: post #135
immibis wrote: post #67

Earlier quoted context omitted.

I noticed this with bandwidth. AWS price for bandwidth: $90.00/TB after 0.1TB/month. Price everywhere else (low cost VPSes): $1.50/TB after 1-5TB/month. Price some places (dedicated servers): $0.00/TB up to ~100TB/month, $1.50/TB after. You pay 60 times the price for the privilege of being on AWS. Bandwidth is just their most egregious price difference. The servers are more expensive too. The storage is more expensiv…

Which is why most folks use CloudFront like AWS recommends, which is typically free for services to route through. Prices for CloudFront egress are competitive with what you described for other vendors. I don't know anyone paying $90/TB out to internet on AWS. https://aws.amazon.com/cloudfront/pricing/?nc=sn&loc=3 Were you trying to route S3 directly out to the internet?

Not everybody is serving public http traffic, though.

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#152

Post by ttfkam »

immibis wrote: post #149
ttfkam wrote: post #135

Earlier quoted context omitted.

Which is why most folks use CloudFront like AWS recommends, which is typically free for services to route through. Prices for CloudFront egress are competitive with what you described for other vendors. I don't know anyone paying $90/TB out to internet on AWS. https://aws.amazon.com/cloudfront/pricing/?nc=sn&loc=3 Were you trying to route S3 directly out to the internet?

The Cloudfront pricing page you cited is just as bad, except the free tier for Cloudfront traffic is 1TB/month rather than 0.1TB/month. The pricing is just as bad above that. It says "Free for origin fetches" but upon further research, seems like that just means you won't pay twice by also having to pay for your traffic to get from its origin to Cloudfront.

You still haven't cited where you got $90/TB from.

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#153

Post by detaro »

ttfkam wrote: post #152
immibis wrote: post #149

Earlier quoted context omitted.

The Cloudfront pricing page you cited is just as bad, except the free tier for Cloudfront traffic is 1TB/month rather than 0.1TB/month. The pricing is just as bad above that. It says "Free for origin fetches" but upon further research, seems like that just means you won't pay twice by also having to pay for your traffic to get from its origin to Cloudfront.

You still haven't cited where you got $90/TB from.

For example EC2 traffic (from US and EU locations, other get more expensive).

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#154

Post by slashdev »

perching_aix wrote: post #110
slashdev wrote: post #79

Earlier quoted context omitted.

Yeah, so restart does not. Which means you can count on it about as much as a server of your own, if you could not repair the server. I know a database company that uses instance storage as the primary storage. It’s common.

Wasn't aware, interesting. I did consider it in the past as well, but the reliability aspect made me consider this as a moonshot rather than anything practical. Kind of weirdly validating to know there are (supposedly) database providers using it. That said, a good bit of our environments are scheduled, so it still wouldn't be an option there without hacks (e.g. doing a compressed blockwise dump before shutting down…

I would take a logical dump on shutdown and the restore that rather. You’d need a backup and restore process anyway, so that path has to exist either way. It’s kind of like a full vacuum, so has some benefits too.

The NDA prevents me from saying which database company, but it’s a major provider of cloud managed databases across clouds.

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#155

Post by anonymars »

the8472 wrote: post #44
codegeek wrote: post #29

Earlier quoted context omitted.

You probably already know this but I will say it anyway. These cloud services like AWS are not succeeding in enterprise because they have outdated hardware. They succeed because in enterprise, CIOs and CTOs want something that is known, has a brand and everyone else uses it. It's like the old adage of "No one got fired for using IBM". Now it is "No one gets fired for hosting with AWS no matter how ridiculous the cost…

> No one gets fired for hosting with AWS But consider the counterfactual: Non-realized customers because AWS certified solutions architect(tm) software couldn't deliver the price/perf they would have needed. At $work this is a very real problem because a software system was built on api gateway, lambdas, sqs and a whole bunch of other moving pieces (serverless! scalable! easy compliance!) that combined resulted in wa…

Reminds me of https://idlewords.com/talks/website_obesity.htm

> Let me give you a concrete example. I recently heard from a competitor, let’s call them ACME Bookmarking Co., who are looking to leave the bookmarking game and sell their website.

While ACME has much more traffic than I do, I learned they only have half the daily active users. This was reassuring, because the hard part of scaling a bookmarking site is dealing with people saving stuff.

We both had the same number of employees. They have an intern working on the project part time, while I dither around and travel the world giving talks. Say half a full-time employee for each of us.

We have similar revenue per active user. I gross $12,000 a month, they gross $5,000.

But where the projects differ radically is cost. ACME hosts their service on AWS, and at one point they were paying $23,000 (!!) in monthly fees. Through titanic effort, they have been able to reduce that to $9,000 a month.

I pay just over a thousand dollars a month for hosting, using my own equipment. That figure includes the amortized cost of my hardware, and sodas from the vending machine at the colo.

So while I consider bookmarking a profitable business, to them it's a $4,000/month money pit. I'm living large off the same income stream that is driving them to sell their user data to marketers and get the hell out of the game.

The point is that assumptions about complexity will anchor your expectations, and limit what you're willing to try. If you think a 'real' website has to live in the cloud and run across a dozen machines, a whole range of otherwise viable projects will seem unprofitable.

Similarly, if you think you need a many-layered CMS and extensive custom javascript for an online publishing venture, the range of things you will try becomes very constricted.

Rather than trying to make your overbuilt projects look simple, ask yourself if they can't just be simple.

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#156

Post by ttfkam »

detaro wrote: post #153
ttfkam wrote: post #152

Earlier quoted context omitted.

You still haven't cited where you got $90/TB from.

For example EC2 traffic (from US and EU locations, other get more expensive).

Okay, I stand corrected. Now my question to you is how are you generating all these terabytes over the internet? And would data transfer costs really be a significant portion of your bill at that scale regardless of cloud provider? Not video transcoding resources, AI capacity, GPU capacity, etc.?

You're talking about the equivalent of completely saturating a gigabit Internet connection for over two hours per terabyte. A high def video stream from Netflix is only about 5 megabits/sec. 4K is 15 megabits.

67+ simultaneous 4K streams for two hours. Or 200+ simultaneous high def streams. Or a metric fuck-ton of web resources.

And you think you're going to find a provider that will allow you to send those kinds of volumes for $10/TB or less AND have relatively few outages AND stick around because their business model is sound?

By all means, point out who these unicorns are. Sign me up!

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#157

Post by ksec »

seunosewa wrote: post #18
skeptrune wrote: post #11

How close is this to the way MySQL does it with InnoDB? It appears to be about the same.

Yep. It's a low hanging fruit they should've picked years ago. They will eventually figure out using b-trees for tables too.

Despite having relatively fewer improvement on the MySQL side, especially the OSS version and not the enterprise version, there are still many areas Postgres have not exceeded MySQL, and doesn't look like will be doing so in next 3 - 4 years.

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#158

Post by sroussey »

cwbriscoe wrote: post #146
sroussey wrote: post #140

Earlier quoted context omitted.

On commercial databases from Oracle and Microsoft, you can cluster the DB on any index. Really would love for Postresql to do the same.

You can but new rows will not be clustered until you re-cluster. CLUSTER table_name USING index_name;

I remember that for MySQL MyISAM (syntax was different) in the olden days. I thought Oracle let you do it though. My bad.

MySQL Innodb and other have real clustered primary key indexes with all the pluses and minuses that entails.

Re: Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O

#159

Post by immibis »

ttfkam wrote: post #156
detaro wrote: post #153

Earlier quoted context omitted.

For example EC2 traffic (from US and EU locations, other get more expensive).

Okay, I stand corrected. Now my question to you is how are you generating all these terabytes over the internet? And would data transfer costs really be a significant portion of your bill at that scale regardless of cloud provider? Not video transcoding resources, AI capacity, GPU capacity, etc.? You're talking about the equivalent of completely saturating a gigabit Internet connection for over two hours per terabyte…

On Hetzner I have no problem saturating my gigabit connection with static files if I want to.
Post reply on HN