Every fast write moves work somewhere else
1–10 of 25 posts
Re: Every fast write moves work somewhere else
#2When I saw the title the first thing I thought of was schema on read versus schema on right when in data platforms.
You can make writes faster and part of that is by not dealing with schema resolution but you do push the work somewhere else there too.
I guess the same principles apply on many different levels, from when you write to the file system up to how you deal with conflicted data types during data ingestion.
Re: Every fast write moves work somewhere else
#3When I saw the title the first thing I thought of was schema on read versus schema on right when in data platforms. You can make writes faster and part of that is by not dealing with schema resolution but you do push the work somewhere else there too. I guess the same principles apply on many different levels, from when you write to the file system up to how you deal with conflicted data types during data ingestion.
The first thing I thought when seeing the title was "writing" with LLMs - writing quickly can easily just move the work onto your readers!
Re: Every fast write moves work somewhere else
#42 angles I think DB designers don’t often think about:
1. Durability extends to the client. Replicated db might ack a write to client, but what if that ack gets lost on the way back over network? If client talks to the DB over simple HTTP, the write might first look like a failure. Can the client retry?
2. Human perception times are biological and don’t change much. But everything in the tech stack has gotten so so much faster since the 80s. Throughput matters, sure, but latency (relatively speaking), is much less of a constraint now than it was.
Re: Every fast write moves work somewhere else
#5When I saw the title the first thing I thought of was schema on read versus schema on right when in data platforms. You can make writes faster and part of that is by not dealing with schema resolution but you do push the work somewhere else there too. I guess the same principles apply on many different levels, from when you write to the file system up to how you deal with conflicted data types during data ingestion.
The first thing I thought when seeing the title was "writing" with LLMs - writing quickly can easily just move the work onto your readers!
I like this a lot, it’s true for more than DBs!
Re: Every fast write moves work somewhere else
#6Today I learned there’s a name for the general version of this idea.
https://en.wikipedia.org/wiki/Waterbed_theory
At a certain point in a solution everything you do to optimize (“push”) in one area will cause a negative effect in a different area (“bulge”).
But this is a nice concrete example.
Re: Every fast write moves work somewhere else
#7Today I learned there’s a name for the general version of this idea. https://en.wikipedia.org/wiki/Waterbed_theory At a certain point in a solution everything you do to optimize (“push”) in one area will cause a negative effect in a different area (“bulge”). But this is a nice concrete example.
v interesting. Thanks for sharing. I kept seeing this phenomenon of "no free lunches". TIL about the waterbed theory.
Re: Every fast write moves work somewhere else
#82 angles I think DB designers don’t often think about: 1. Durability extends to the client. Replicated db might ack a write to client, but what if that ack gets lost on the way back over network? If client talks to the DB over simple HTTP, the write might first look like a failure. Can the client retry? 2. Human perception times are biological and don’t change much. But everything in the tech stack has gotten so so m…
1) you ack the ack, then ack the ack ack ;)
2) depends, i have a database that farks up a pretty complex distributed system when clients write from another az, latency really can be an issue for some workloads
Re: Every fast write moves work somewhere else
#9Today I learned there’s a name for the general version of this idea. https://en.wikipedia.org/wiki/Waterbed_theory At a certain point in a solution everything you do to optimize (“push”) in one area will cause a negative effect in a different area (“bulge”). But this is a nice concrete example.
It’s called constraints and tradeoffs yeah
Re: Every fast write moves work somewhere else
#10Another fun failure mode in the same vein: if you are emulating an NVMe device and declare that a write operation is successful once bytes are in memory, you will quickly find yourself buffering arbitrarily large amounts of data as "persist to disk" (SSD speed) falls behind "acknowledge writes" (RAM speed). If you do not add backpressure to your system intentionally, it will be added for you – and you may not like where it's placed!
(This also happens at the SSD level: burst writes can be very fast as data is buffered in the SSD's own RAM, then performance steady-states at the true write speed once that's saturated)