Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't defer Close() on writable files (2017)

joeshaw.org

Post reply on HN
Print view View original

11–20 of 311 posts

Re: Don't defer Close() on writable files (2017)

#11

Post by randomdata »

adrianmsmith wrote: post #4

Surely this would all go away if Go had an exception handling mechanism like most mainstream languages do? You'd just concentrate on the "happy path", you'd close the file, there'd be nothing to forget or write blog posts about because the exception would be propagated, without needing to write any lines of code.

Go does indeed has an exception handling system like most other languages. Errors and exceptions are very different things, though.

Of course, nothing stops you from building your own file handing package that overloads exception handlers to deal with errors. If it gains traction then it would prove the stdlib should consider a v2 API.

But that you already haven’t done so is telling…

Re: Don't defer Close() on writable files (2017)

#12

Post by tux3 »

silon42 wrote: post #8
tux3 wrote: post #3

Don't call close() twice if you're not using pidfds, this is racy. The fd could be reused in between the two calls. You'll risk closing random things a frustratingly small fraction of the time, creating very hard bugs for yourself.

Surely File.close will clear the fd on success...

You're right, looks like it does!

Re: Don't defer Close() on writable files (2017)

#13

Post by dist-epoch »

masklinn wrote: post #7

There are more wrinkles with this: - if you are creating a file, to ensure full synchronisation you also need to fsync the parent directory, otherwise the file can be fsynced but the update to the directory lost - if sync fails, you can not assume anything about the file, whether on-disk or in memory , critically one understanding which got dubbed "fsyncgate" and lead to many RDBMS having to be updated is that you ca…

So if I do fopen/fwrite/fsync/fclose, that is not enough? That is crazy, I think 90% of apps don't fsync the parent directory. Also, how many levels of parents do you need to fsync?

Re: Don't defer Close() on writable files (2017)

#14

Post by trashburger »

Arguably, one should call `flush()` on the file first. Resource deallocation must always succeed; otherwise a lot of invariants break. This is why Zig's close method[0] ignores errors (with the exception of `EBADF`).

[0]: https://github.com/ziglang/zig/blob/fb0028a0d7b43a2a5dd05f07...

Re: Don't defer Close() on writable files (2017)

#16

Post by masklinn »

kzrdude wrote: post #6
K0nserv wrote: post #2

Rust has the same problem. Files are closed in `Drop` when the value goes out of scope, but all errors are silently ignored. To solve this there's `sync_all`[0]. Generally, relying on defer in Go or Drop in Rust for anything that can fail seems like an anti-pattern to me. 0: https://doc.rust-lang.org/std/fs/struct.File.html#method.syn...

An ownership consuming close(self) would make sense, but has not been added, there must be some good reason for that?

That nobody has gone through the effort of collating its requirements and writing an RFC after https://github.com/rust-lang/rfcs/pull/770 was closed (back in 2015).

I assume a big issue is that this is full of edge cases up the ass, and the value is somewhat limited in the sense that if you know you want durable writes you'll sync() and know you're fucked if you get an error, but close() does not guarantee a sync to disk, as the linux man page indicates:

> A successful close does not guarantee that the data has been successfully saved to disk, as the kernel uses the buffer cache to defer writes.

So you'd need a "close", and a "close_sync", and possibly also a "close_datasync" (if you're ok with discarding metadata). And one could argue at that point `close` has essentially no value beyond hopefully getting rid of the fd / handle, and drop already does a fine job of that.

Re: Don't defer Close() on writable files (2017)

#17

Post by masklinn »

dist-epoch wrote: post #13
masklinn wrote: post #7

There are more wrinkles with this: - if you are creating a file, to ensure full synchronisation you also need to fsync the parent directory, otherwise the file can be fsynced but the update to the directory lost - if sync fails, you can not assume anything about the file, whether on-disk or in memory , critically one understanding which got dubbed "fsyncgate" and lead to many RDBMS having to be updated is that you ca…

So if I do fopen/fwrite/fsync/fclose, that is not enough? That is crazy, I think 90% of apps don't fsync the parent directory. Also, how many levels of parents do you need to fsync?

> So if I do fopen/fwrite/fsync/fclose, that is not enough?

That is my understanding.

> Also, how many levels of parents do you need to fsync?

Only one, at least if you didn't create the parent directory (if you did then you might have to fsync its parent, recursively). The fsync on the parent directory ensures the dir entry for your new file is flushed to disk.

Re: Don't defer Close() on writable files (2017)

#18

Post by peterashford »

randomdata wrote: post #11
adrianmsmith wrote: post #4

Surely this would all go away if Go had an exception handling mechanism like most mainstream languages do? You'd just concentrate on the "happy path", you'd close the file, there'd be nothing to forget or write blog posts about because the exception would be propagated, without needing to write any lines of code.

Go does indeed has an exception handling system like most other languages. Errors and exceptions are very different things, though. Of course, nothing stops you from building your own file handing package that overloads exception handlers to deal with errors. If it gains traction then it would prove the stdlib should consider a v2 API. But that you already haven’t done so is telling…

You mean panics? They don't work across go routines so they're limited and hardly exception handling like most other languages

Re: Don't defer Close() on writable files (2017)

#19

Post by saurik »

devjab wrote: post #10
adrianmsmith wrote: post #4

Surely this would all go away if Go had an exception handling mechanism like most mainstream languages do? You'd just concentrate on the "happy path", you'd close the file, there'd be nothing to forget or write blog posts about because the exception would be propagated, without needing to write any lines of code.

Explicit error handling is a choice and implicit error handling through exceptions is not necessarily a feature. Both have advantages and disadvantages, I’d say the more “modern” approach actually the opposite to what you state here, and is in my opinion the way to Go (pun int intended), though it’s also how Haskell does it. You’ll find the same philosophy in Rust, Zig, Swift and others which all build on the previou…

I feel like you might be claiming that Haskell does something like Go does, but it actually doesn't: it supports monads, and so uses a monad to hide the error semantics entirely, providing exception-like syntax with automatic propagation.

(I might misunderstand your use of "though", though? It could be that you were just noting in passing how Haskell disagrees with all of these supposedly-"modern" languages, and instead leaned into the sane happy path semantics, thanks to monads.)

(edit: to be clear, though... I do not think exceptions solve this. I wrote a comment elsewhere on this thread about the semantics issue, but a few other people also wrote similar things while I was trying to type my overly-verbose reply ;P.)

Re: Don't defer Close() on writable files (2017)

#20

Post by randomdata »

peterashford wrote: post #18
randomdata wrote: post #11

Earlier quoted context omitted.

Go does indeed has an exception handling system like most other languages. Errors and exceptions are very different things, though. Of course, nothing stops you from building your own file handing package that overloads exception handlers to deal with errors. If it gains traction then it would prove the stdlib should consider a v2 API. But that you already haven’t done so is telling…

You mean panics? They don't work across go routines so they're limited and hardly exception handling like most other languages

Which languages see exception handlers work across go routines?

Anyway, most don't. There is no difference from exception handlers in most other languages.

The syntax is a little different. Is that where you've become confused?

Post reply on HN