11–20 of 73 posts
Earlier quoted context omitted.
TLS certs are freeeeee
Judging from this very release, where he implemented support for page-level checksums and encryption for LMBD, I assume the author knows a thing or two about encryption. He probably then deemed it unnecessary for this specific website.
Earlier quoted context omitted.
TLS certs are freeeeee
Judging from this very release, where he implemented support for page-level checksums and encryption for LMBD, I assume the author knows a thing or two about encryption. He probably then deemed it unnecessary for this specific website.
Earlier quoted context omitted.
Judging from this very release, where he implemented support for page-level checksums and encryption for LMBD, I assume the author knows a thing or two about encryption. He probably then deemed it unnecessary for this specific website.
Cryptography engineers are not excluded from being lazy sysadmins.
Anyway, of course in case you feel the website is a risk, you should refrain from using it. Safety comes first.
Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class. I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author als…
Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class. I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author als…
It is really reliable except write performance in my experience.
Author of it writes very spicy stuff and sounds pretty rude.
I would recommend doing a prototype with real data scale and testing if it meets your requirements. The write performance can be really atrocious and It doesn't have a high performance potential because it is based on memmap.
Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class. I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author als…
And yes, I ensured there were no outstanding long lived readers, verified with mdb_stat -r. My workload used one transaction per read/write anyway (never needed larger atomicity). Once the db got into the bad state, running my program on it would almost immediately run into the issue again, so I really think the db is in a bad state such that most writes would cause it to hang, not related to how I do transactions. This workload would pretty consistently hit the issue once the db got to several hundred gb.
Issue #10236 on the OpenLDAP bug tracker might be the root cause, who knows. It's been marked CONFIRMED for years without a fix, while other similar issues are created.
This is extremely annoying. It seems workload dependent (other workloads I've run create absolutely massive lmdb dbs without this issue) and once it happens your only recourse is to make a new db and copy the contents over (thankfully reads still work fine on these borked dbs).
Other than that, though, it's great. Never in any case had actual data corruption, and reads and writes are extremely fast (until this issue happens)
Edit: fun fact, since shopify may have created Bolt in response to this bug, and then Bolt was the root cause of the 73-hour Roblox downtime in 2021, this bug may indirectly have caused one of the worst outages ever!
Do people have good experiences with LMDB, in terms of reliability? I've never used it in production, but I've read through the code and design documents for a database implementation class. I remember some strange code (such as pushing return values 4k above the stack, with a comment like "this works as long as the caller doesn't use more than 4k of stack space before accessing the return value"), and the author als…
I had a situation where the web service's writes were slowing down to an unbearable crawl because the number of entries in the database were reaching tens of billions of entries. Thankfully, the users never experienced the slowness. The website stayed nice and fast, even though the background updates were extraordinarily slow. The issue was fixed by sharding the databases.
HTTP ?? Com’on man
it is just a link to documentation
You can seal memfds too, which means that the "read-only" mode is easy to implement: just map your memfd for write, apply F_SEAL_FUTURE_WRITE, and share the memfd to anyone you want to have read-only access.
By doing your own O_DIRECT IO instead of relying on the kernel's defaults, you get a lot more control. You choose how much readahead to do; you choose your read-cluster size. You choose your cache eviction strategy. You choose when to write back.
BTW: O_DIRECT can also be done asynchronously using aio or io_uring. There's no such thing as an asynchronous page fault. And IO errors? Would you rather deal with EIO or SIGBUS?
Why would you want the kernel to do these things for you? It'll do a worse job: it has less information than you do and has to use blunt heuristics that work sort-of-good-enough for the whole world, not just your program.
And it's not any faster either. O_DIRECT is DMA. A page cache fill is also DMA. It's the same operation, spelled differently.
I've never understood the fascination some people have with mmap. Memory-mapped file IO is just a RAM cache combined with a hidden system call (a page fault) to fill the cache. You can do the same thing yourself by using O_DIRECT to fill regular anonymous memory. If you're feeling social, you can fill a mapped and shared memfd. You can seal memfds too, which means that the "read-only" mode is easy to implement: just…