Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Disks Lie: Building a WAL that actually survives

blog.canoozie.net

Post reply on HN
Print view View original

41–50 of 53 posts

Re: Disks Lie: Building a WAL that actually survives

#41

Post by dspillett »

rogerrogerr wrote: post #7
jtregunna wrote: post #5

Earlier quoted context omitted.

In what sense? The phrasing is just a generalization, production-grade anything needs consideration of the needs and goals of the project.

“ isn’t just , it’s ” is an AI smell.

It is, but partly because it is a common form in the training data. LLM output seems to use the form more than people, presumably either due to some bias in the training data (or the way it is tokenised) or due to other common token sequences leading into it (remember: it isn't an official acronym but Glorified Predictive Text is an accurate description). While it is a smell, it certainly isn't a reliable marker, there needs to be more evidence than that.

Re: Disks Lie: Building a WAL that actually survives

#42

Post by amelius »

wmf wrote: post #36
amelius wrote: post #32

Earlier quoted context omitted.

Then how do you know the writes are done after the second sync?

AFAIK multiple syncs can't happen at the same time so the second sync implicitly waits for the first one to complete.

If it was that simple, then why doesn't sync just do 2x sync internally?

Re: Disks Lie: Building a WAL that actually survives

#43

Post by henning »

This looks AI-generated, including the linked code. That explains why the .zig-cache directory and the binary is checked into Git, why there's redundant commenting, and why the README has that bold, bullet point and headers style that is typical of AI.

If you can't be bothered to write it, I can't be bothered to read it.

Re: Disks Lie: Building a WAL that actually survives

#44

Post by twoodfin »

henning wrote: post #43

This looks AI-generated, including the linked code. That explains why the .zig-cache directory and the binary is checked into Git, why there's redundant commenting, and why the README has that bold, bullet point and headers style that is typical of AI. If you can't be bothered to write it, I can't be bothered to read it.

The front page this weekend has been full of this stuff. If there’s a hint of clickbait about the title, it’s almost a forgone conclusion you’ll see all the other LLM tics, too.

These do not make the writing better! They obscure whatever the insight is behind LinkedIn-engagement tricks and turns of phrase that obfuscate rather than clarify.

I’ll keep flagging and see if the community ends up agreeing with me, but this is making more and more of my hn experience disappointing instead of delightful.

Re: Disks Lie: Building a WAL that actually survives

#45

Post by wakawaka28 »

amelius wrote: post #42
wmf wrote: post #36

Earlier quoted context omitted.

AFAIK multiple syncs can't happen at the same time so the second sync implicitly waits for the first one to complete.

If it was that simple, then why doesn't sync just do 2x sync internally?

If I had to guess, it is just extra work to do it twice, and you may not need to wait for it for some use cases. The better way would be to add a flag or alternative function to make the sync a blocking operation in the first place.

Re: Disks Lie: Building a WAL that actually survives

#46

Post by wmf »

amelius wrote: post #42
wmf wrote: post #36

Earlier quoted context omitted.

AFAIK multiple syncs can't happen at the same time so the second sync implicitly waits for the first one to complete.

If it was that simple, then why doesn't sync just do 2x sync internally?

Why is creat() missing the e? Why does an FTP server connect back to the client?

Re: Disks Lie: Building a WAL that actually survives

#47

Post by scottlamb »

n_u wrote: post #40
scottlamb wrote: post #34

Earlier quoted context omitted.

> Wait, so the OS can re-order the fsync() to happen before the write request it is supposed to be syncing? Is there a citation or link to some code for that? It seems too ridiculous to be real. This is an io_uring-specific thing. It doesn't guarantee any ordering between operations submitted at the same time, unless you explicitly ask it to with the `IOSQE_IO_LINK` they mentioned. Otherwise it's as if you called wri…

I guess I'm a bit confused why the author recommends using this flag and fsync. Related: I would think that grouping your writes and then fsyncing rather than fsyncing every time would be more efficient but it looks like a previous commenter did some testing and that isn't always the case https://news.ycombinator.com/item?id=15535814

I'm not sure there's any good reason. Other commenters mentioned AI tells. I wouldn't consider this article a trustworthy or primary source.

Re: Disks Lie: Building a WAL that actually survives

#48

Post by n_u »

scottlamb wrote: post #47
n_u wrote: post #40

Earlier quoted context omitted.

I guess I'm a bit confused why the author recommends using this flag and fsync. Related: I would think that grouping your writes and then fsyncing rather than fsyncing every time would be more efficient but it looks like a previous commenter did some testing and that isn't always the case https://news.ycombinator.com/item?id=15535814

I'm not sure there's any good reason. Other commenters mentioned AI tells. I wouldn't consider this article a trustworthy or primary source.

Yeah that seems reasonable. The article seems to mix fsync and O_DSYNC without discussing their relationship which seems more like AI and less like a human who understands it.

It also seems if you were using io_uring and used O_DSYNC you wouldn't need to use IOSQE_IO_LINK right?

Even if you were doing primary and secondary log file writes, they are to different files so it doesn't matter if they race.

Re: Disks Lie: Building a WAL that actually survives

#49

Post by [deleted] »

scottlamb wrote: post #34
n_u wrote: post #27

> Submit the write to the primary file > Link fsync to that write (IOSQE_IO_LINK) > The fsync's completion queue entry only arrives after the write completes > Repeat for secondary file Wait, so the OS can re-order the fsync() to happen before the write request it is supposed to be syncing? Is there a citation or link to some code for that? It seems too ridiculous to be real. > O_DSYNC: Synchronous writes. Don't retu…

> Wait, so the OS can re-order the fsync() to happen before the write request it is supposed to be syncing? Is there a citation or link to some code for that? It seems too ridiculous to be real. This is an io_uring-specific thing. It doesn't guarantee any ordering between operations submitted at the same time, unless you explicitly ask it to with the `IOSQE_IO_LINK` they mentioned. Otherwise it's as if you called wri…

[deleted]

Re: Disks Lie: Building a WAL that actually survives

#50

Post by scottlamb »

n_u wrote: post #48
scottlamb wrote: post #47

Earlier quoted context omitted.

I'm not sure there's any good reason. Other commenters mentioned AI tells. I wouldn't consider this article a trustworthy or primary source.

Yeah that seems reasonable. The article seems to mix fsync and O_DSYNC without discussing their relationship which seems more like AI and less like a human who understands it. It also seems if you were using io_uring and used O_DSYNC you wouldn't need to use IOSQE_IO_LINK right? Even if you were doing primary and secondary log file writes, they are to different files so it doesn't matter if they race.

> It also seems if you were using io_uring and used O_DSYNC you wouldn't need to use IOSQE_IO_LINK right? Even if you were doing primary and secondary log file writes, they are to different files so it doesn't matter if they race.

I think there are a lot of reasons to use this flag besides a write()+f(data)sync() sequence:

* If you're putting something in a write-ahead log then applying it to the primary storage, you want it to be fully committed to the write-ahead log before you start changing the primary storage, so if there's a crash halfway through the primary storage change you can use the log to get to a consistent state (via undo or redo).

* If you're trying to atomically replace a file via the rename-a-temporary-file-into-place trick, you can submit the whole operation to the ring at once, but you'd want to use `IOSQE_IO_LINK` to ensure the temporary file is fully written/synced before the rename happens.

btw, a clarification about my earlier comment: `O_SYNC` (no `D`) should be equivalent to calling `fsync` after every write. `O_DSYNC` should be equivalent to calling the weaker `fdatasync` after every write. The difference is the metadata stored in the inode.

Post reply on HN