Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

IO Devices and Latency

planetscale.com

Post reply on HN
Print view View original

121–130 of 159 posts

Re: IO Devices and Latency

#121

Post by jiggawatts »

CSDude wrote: post #113

For years, I just didn't get why replicated databases always stick with EBS and deal with its latency. Like, replication is already there, why not be brave and just go with local disks? At my previous orgs, where we ran Elasticsearch for temporary logs/metrics storage, I proposed we do exactly that since we didn't even have major reliability requirements. But I couldn't convince them back then, we ended up with even…

I looked into this with an idea of running SQL Server Availability Groups on the Azure Las_v3 series VMs, which have terabytes of local SSD.

The main issue was that after a stop-start event, the disks are wiped. SQL Server can’t automatically handle this, even if the rest of the cluster is fine and there are available replicas. It won’t auto repair the node that got reset. The scripting and testing required to work around this would be unsupportable in production for all but the bravest and most competent orgs.

Re: IO Devices and Latency

#122

Post by _1tem »

If this is true, then how do "serverless" database providers like Neon advertise "low latency" access? They use object storage like S3, which I imagine is an order of magnitude worse than networked storage for latency.

edit: apparently they build a kafkaesque layer of caching. No thank you, I'll just keep my data on locally attached NVMe.

Re: IO Devices and Latency

#123

Post by badmintonbaseba »

sedatk wrote: post #72
bob1029 wrote: post #67

Earlier quoted context omitted.

The entire point is to avoid the network hop. Application SQLite NVMe has orders of magnitude less latency than Application Postgres Client Network Postgres Server NVMe > You should be avoiding serial database queries as much as possible in the first place. I don't get to decide this. The business does.

"...has orders of magnitude less latency than..." [citation needed]. Local network access shouldn't be much different than local IPC.

You don't have IPC for sqlite, do you?

Re: IO Devices and Latency

#124

Post by bob1029 »

chatmasta wrote: post #106
bob1029 wrote: post #68

Earlier quoted context omitted.

> If only one thread of writing is required, then SQLite works absolutely great. The whole point of getting your commands down to microsecond execution time is so that you can get away with just one thread of writing. Entire financial exchanges operate on this premise.

Entire financial exchanges are not running single threaded writes to their persistent data store. If they are, and you have a link, I’d love to be proven wrong.

https://www.infoq.com/presentations/LMAX/

Re: IO Devices and Latency

#125

Post by nolist_policy »

badmintonbaseba wrote: post #123
sedatk wrote: post #72

Earlier quoted context omitted.

"...has orders of magnitude less latency than..." [citation needed]. Local network access shouldn't be much different than local IPC.

You don't have IPC for sqlite, do you?

You do of you access the same database from miltiple processes.

Re: IO Devices and Latency

#126

Post by spratzt »

chatmasta wrote: post #105
conradev wrote: post #62

Earlier quoted context omitted.

Until you hit the single-writer limitation in SQLite, you do not need to spend more CPU cycles on Postgres

That’s a limitation you’ll hit pretty quickly unless you’ve specifically planned your architecture to be mostly read-only SQLite or one SQLite per session.

You certainly won’t hit it with most corporate OLAP processing, which is nearly all read-only SQlite. Writes are generally batched and processed outside ‘normal’ business hours, where the limitations of SQlite writing are irrelevant.

Re: IO Devices and Latency

#129

Post by anymouse123456 »

bddicken wrote: post #3

Author of the blog here. I had a great time writing this. By far the most complex article I've ever put together, with literally thousands of lines of js to build out these interactive visuals. I hope everyone enjoys.

I love this kind of datavis.

We are generally bad at internalizing comparisons at these scales. The visualizations make a huge difference in building more detailed intuitions.

Really nice work, thank you!

Re: IO Devices and Latency

#130

Post by carderne »

I'm always curious about latency for all these newdb offerings like PlanetScale/Neon/Supabase.

It seems like they don't emphasise strongly enough _make sure you colocate your server in the same cloud/az/region/dc as our db. I suspect a large fraction of their users don't realise this, and have loads of server-db traffic happening very slowly over the public internet. It won't take many slow db reads (get session, get a thing, get one more) to trash your server's response latency.

Post reply on HN