11–20 of 33 posts
Writing to disk for every write is required, otherwise you're not durable. Sure it's faster to never write to disk, then you reboot and you've lost data. /dev/null is a webscale database that is even faster!
read the whole article. WAL is the transaction log and the author tested correctness after a crash.
> A 100-bit bloom filter holding 100,000 keys is saturated instantly > This is the kind of bug you only find by building the thing and measuring it. No? I mean, maybe if you're vibecoding it's the only way, but in the prehistoric days you could reason about what code would do before you ran it.
I'm sure you've never made a silly mistake where you passed the wrong integer parameter to a function, stared at your screen, and failed to notice it. Or, forgot the order of arguments to calloc().
If you're saying that profiling is for those too lazy to reason about their code, you're distorting the whole lesson: profiling is more powerful than guessing.
> A 100-bit bloom filter holding 100,000 keys is saturated instantly > This is the kind of bug you only find by building the thing and measuring it. No? I mean, maybe if you're vibecoding it's the only way, but in the prehistoric days you could reason about what code would do before you ran it.
Do you think the author is somehow capable of writing the entire codebase, but not able to reason about code??? I'm sure you've never made a silly mistake where you passed the wrong integer parameter to a function, stared at your screen, and failed to notice it. Or, forgot the order of arguments to calloc(). If you're saying that profiling is for those too lazy to reason about their code, you're distorting the whole…
Writing to disk for every write is required, otherwise you're not durable. Sure it's faster to never write to disk, then you reboot and you've lost data. /dev/null is a webscale database that is even faster!
read the whole article. WAL is the transaction log and the author tested correctness after a crash.
It seems to me that neither the old nor the new version of the code is really "durable" as I would understand the word. The old version made a write syscall per batch, but doesn't say it also did an fsync per batch. The new version writes data to an mmap'ed file, and calls fsync in the background.
So both versions are "durable" in the sense that written data is preserved even if the process gets killed, because it's in the OS page cache. But in both versions, a write can be completed before the data actually makes it to disk, so a power failure will lose acknowledged writes.
Writing to disk for every write is required, otherwise you're not durable. Sure it's faster to never write to disk, then you reboot and you've lost data. /dev/null is a webscale database that is even faster!
https://github.com/facebook/rocksdb/wiki/WAL-Performance#non...
> A 100-bit bloom filter holding 100,000 keys is saturated instantly > This is the kind of bug you only find by building the thing and measuring it. No? I mean, maybe if you're vibecoding it's the only way, but in the prehistoric days you could reason about what code would do before you ran it.
> A 100-bit bloom filter holding 100,000 keys is saturated instantly > This is the kind of bug you only find by building the thing and measuring it. No? I mean, maybe if you're vibecoding it's the only way, but in the prehistoric days you could reason about what code would do before you ran it.
Do you think the author is somehow capable of writing the entire codebase, but not able to reason about code??? I'm sure you've never made a silly mistake where you passed the wrong integer parameter to a function, stared at your screen, and failed to notice it. Or, forgot the order of arguments to calloc(). If you're saying that profiling is for those too lazy to reason about their code, you're distorting the whole…
Me: so you have an in-memory cache, right?
Them: yes!
Me: what is the TTL?
Them: Oh, it's not set, oops. Here, let's set it to 1 minute. Hey look, the performance went way up!
Me: okay, great. When you say 1 minute, do you mean 60 seconds?
Them: uh...wait...uh....oh, the unit is seconds. Wait, why is the performance so good with a 1 second TTL?
Me: What's your load test?
Them: We crank 1M TPS fetching the same 30 items over and over.
Me: ....
I totally agree about the power of profiling but profiling without understanding would not have helped this team.
> A 100-bit bloom filter holding 100,000 keys is saturated instantly > This is the kind of bug you only find by building the thing and measuring it. No? I mean, maybe if you're vibecoding it's the only way, but in the prehistoric days you could reason about what code would do before you ran it.
Mistakes are always easy to recognize in retrospect, so hopefully this comment isnt too unfair, but one thing that caught me about this, is that logically it makes no sense. You would never use a bloom filter for just 10 entries. If you have only 10 entries it is almost certainly faster to skip the bloom filter. So i feel like that is the part that should have instantly stood out. [Obviously, i've made my own silly m…
bool getSchemaSizes(size_t * expectedBatchSize, size_t * expectedEntriesPerBlock) { ... }
size_t expectedEntriesPerBlock, expectedBatchSize;
getSchemaSizes(&expectedEntriesPerBlock, &expectedBatchSize)
initBloomFilter(expectedEntriesPerBlock)
Earlier quoted context omitted.
Mistakes are always easy to recognize in retrospect, so hopefully this comment isnt too unfair, but one thing that caught me about this, is that logically it makes no sense. You would never use a bloom filter for just 10 entries. If you have only 10 entries it is almost certainly faster to skip the bloom filter. So i feel like that is the part that should have instantly stood out. [Obviously, i've made my own silly m…
Sure, it logically makes no sense. But while learning a new subject, have you never made a silly mistake like: bool getSchemaSizes(size_t * expectedBatchSize, size_t * expectedEntriesPerBlock) { ... } size_t expectedEntriesPerBlock, expectedBatchSize; getSchemaSizes(&expectedEntriesPerBlock, &expectedBatchSize) initBloomFilter(expectedEntriesPerBlock)
> A 100-bit bloom filter holding 100,000 keys is saturated instantly > This is the kind of bug you only find by building the thing and measuring it. No? I mean, maybe if you're vibecoding it's the only way, but in the prehistoric days you could reason about what code would do before you ran it.
Do you think the author is somehow capable of writing the entire codebase, but not able to reason about code??? I'm sure you've never made a silly mistake where you passed the wrong integer parameter to a function, stared at your screen, and failed to notice it. Or, forgot the order of arguments to calloc(). If you're saying that profiling is for those too lazy to reason about their code, you're distorting the whole…