The fall of datacenter software • Hacker News Board

Re: The fall of datacenter software

#31

Post by rmah » Thu, Aug 25, 2022, 7:47 PM UTC

crmd wrote: post #26

jeffbee wrote: post #22

Earlier quoted context omitted.

All progress happens at the limit. You may just be running a lot of crud that loafs along, but out there somewhere there is an operator of something that sends a million packets per second per core and they'd rather get 2 million. You don't use your SSD but there is some organization that finds 1 million IOPS to be a little lower than they'd like. You do benefit from these advances. Your searches get faster, your vid…

How do you find these limit users? My startup makes low cost block and file storage on AWS that does 2M IOPS per volume with 200 uSec latency, and we’re surprised to find most customers are perfectly happy using gp3 with 3k IOPs and 1-2 ms latency.

How can you drive by mentioning this without mentioning your company name?!?!? Our firm uses smallish sizes of AWS EFS storage and we're highly annoyed by the throughput limitations.

Re: The fall of datacenter software

#32

Post by 8note » Thu, Aug 25, 2022, 9:15 PM UTC

bombcar wrote: post #28

crmd wrote: post #26

Earlier quoted context omitted.

How do you find these limit users? My startup makes low cost block and file storage on AWS that does 2M IOPS per volume with 200 uSec latency, and we’re surprised to find most customers are perfectly happy using gp3 with 3k IOPs and 1-2 ms latency.

I have a suspicion that many of them are not running on AWS, since they'd like to remove as much cloud/network latency as possible.

My parents company is an example - they need that kind of performance in remote northern canada where connecting to AWS would be a couple bytes every couple seconds with a while of a latency.

The server goes travelling instead, along with the people to use it

Re: The fall of datacenter software

#33

Post by 8note » Thu, Aug 25, 2022, 9:38 PM UTC

crmd wrote: post #26

jeffbee wrote: post #22

Earlier quoted context omitted.

All progress happens at the limit. You may just be running a lot of crud that loafs along, but out there somewhere there is an operator of something that sends a million packets per second per core and they'd rather get 2 million. You don't use your SSD but there is some organization that finds 1 million IOPS to be a little lower than they'd like. You do benefit from these advances. Your searches get faster, your vid…

How do you find these limit users? My startup makes low cost block and file storage on AWS that does 2M IOPS per volume with 200 uSec latency, and we’re surprised to find most customers are perfectly happy using gp3 with 3k IOPs and 1-2 ms latency.

The limit work I've done is in geophysics, and AWS is a non-starter for data collection and analysis in the bush.

If you can wait a few weeks to upload all the data to AWS, why are you going to care about moving the processing speed from a month to half a month?

The high speed stuff is in dedicated hardware that can travel to where it's needed, whether that's in a base station or with the surveyors

Re: The fall of datacenter software

#34

Post by dilyevsky » Fri, Aug 26, 2022, 2:06 AM UTC

crmd wrote: post #26

jeffbee wrote: post #22

Earlier quoted context omitted.

All progress happens at the limit. You may just be running a lot of crud that loafs along, but out there somewhere there is an operator of something that sends a million packets per second per core and they'd rather get 2 million. You don't use your SSD but there is some organization that finds 1 million IOPS to be a little lower than they'd like. You do benefit from these advances. Your searches get faster, your vid…

How do you find these limit users? My startup makes low cost block and file storage on AWS that does 2M IOPS per volume with 200 uSec latency, and we’re surprised to find most customers are perfectly happy using gp3 with 3k IOPs and 1-2 ms latency.

I’d be interested in the company name too (feel free to dm if it’s stealth or something). I’d add that gp3 and other cloud pds implement ec under the hood and as such have very high durability and zero maintenance issues overhead (since they’re managed). And yeah you trade dogshit performance for that privilege