Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't defer Close() on writable files (2017)

joeshaw.org

Post reply on HN
Print view View original

121–130 of 311 posts

Re: Don't defer Close() on writable files (2017)

#121

Post by eatonphil »

praptak wrote: post #113
masklinn wrote: post #7

There are more wrinkles with this: - if you are creating a file, to ensure full synchronisation you also need to fsync the parent directory, otherwise the file can be fsynced but the update to the directory lost - if sync fails, you can not assume anything about the file, whether on-disk or in memory , critically one understanding which got dubbed "fsyncgate" and lead to many RDBMS having to be updated is that you ca…

> otherwise the file can be fsynced but the update to the directory lost It also goes the other way - the update to the directory can be fsynced but the file lost. This can break the "create temp file, write, close, rename to current" scenario (when the intention is to replace file contents atomically). POSIX doesn't guarantee the order in which data hits the disk, so the above scenario can become "create temp file,…

Can anyone provide links to learn more about these misconceptions and the bugs they caused?

Edit: talking about filesystem misconceptions, not fsyncgate.

Re: Don't defer Close() on writable files (2017)

#122

Post by foldr »

saurik wrote: post #19
devjab wrote: post #10

Earlier quoted context omitted.

Explicit error handling is a choice and implicit error handling through exceptions is not necessarily a feature. Both have advantages and disadvantages, I’d say the more “modern” approach actually the opposite to what you state here, and is in my opinion the way to Go (pun int intended), though it’s also how Haskell does it. You’ll find the same philosophy in Rust, Zig, Swift and others which all build on the previou…

I feel like you might be claiming that Haskell does something like Go does, but it actually doesn't: it supports monads, and so uses a monad to hide the error semantics entirely, providing exception-like syntax with automatic propagation. (I might misunderstand your use of "though", though? It could be that you were just noting in passing how Haskell disagrees with all of these supposedly-"modern" languages, and inst…

It's worth noting that using Either style monadic error handling in IO code in Haskell is arguably an antipattern, as the Haskell runtime has its own built in exception handling (which actually works in a pretty conventional way). Some more info on this under 'Exceptions best practices ' here: https://tech.fpcomplete.com/haskell/tutorial/exceptions/

Re: Don't defer Close() on writable files (2017)

#123

Post by lmz »

lionkor wrote: post #108
jiggawatts wrote: post #21

Earlier quoted context omitted.

Crazy is the right term. File system APIs in general have too many sharp edges and need a ground-up rethink. Consider S3-like protocols: these recognise that 99% of the time applications just want “create file with given contents” or “read back what they’ve previously written.” The edge cases should be off the beaten path, not in your way tripping you up to when you want the simple scenario.

It sounds more like you are asking for an abstraction

The file system is already an abstraction. I think they are asking if it's the right one.

Re: Don't defer Close() on writable files (2017)

#124

Post by blcknight »

joeshaw wrote: post #76
nickcw wrote: post #23

Here is my favorite solution to this problem // CheckClose is a utility function used to check the return from // Close in a defer statement. func CheckClose(c io.Closer, err *error) { cerr := c.Close() if *err == nil { *err = cerr } } Use like this - you must name the error return func whatever() (err error) { f, err := os.Open(blah) // ... defer CheckClose(f, &err) // ... } This closes the file and if there wasn't…

I caution against this approach, as you are not really dealing with the error when it occurs. If the work you do after the defer has other side effects, you may have just gotten your application into an inconsistent state and it's very hard to see in code why this might be. `defer` is really not well-suited for error handling, its benefit is mainly in resource cleanup where failure is impossible or doesn't matter. (T…

> If the work you do after the defer has other side effects

Defer is by definition the last work you do in a function, there won't be more work except by the caller who will get the error returned to them.

If you are structuring a function that writes a file, and then does something with it, defer isn't appropriate, since you should close it before you do any more work.

Re: Don't defer Close() on writable files (2017)

#125

Post by Agingcoder »

I dislike the multiple close pattern - I was bitten by this behavior years ago, when the second close() ended up closing another file which had been opened between the first and second close ( I think they were actually sockets ). It was a bona fide bug on my side , but it made for unpleasant memories, and a general distrust of such idioms on my side unless there's a language wide guarantee somewhere in the picture.

Re: Don't defer Close() on writable files (2017)

#127

Post by jsndnnd »

Agingcoder wrote: post #125

I dislike the multiple close pattern - I was bitten by this behavior years ago, when the second close() ended up closing another file which had been opened between the first and second close ( I think they were actually sockets ). It was a bona fide bug on my side , but it made for unpleasant memories, and a general distrust of such idioms on my side unless there's a language wide guarantee somewhere in the picture.

I don't understand how that could happen, since the original file handle would have been invalidated.

Which operating system did you experience this under and was it the operating system, your Libc or what else in the stack which caused this?

Re: Don't defer Close() on writable files (2017)

#128

Post by snatchpiesinger »

"Commit pending writes" and "discard file handle" should ideally be separate operations, with the former potentially returning errors and the latter being infallible. "Discard file handle" should be called from defer/destructiors/other language RAII constructs, while "commit pending writes" should be called explicitly in the program's control flow, with its error appropriately handled or passed up the stack.

Whether if it's allowed to write again after a "commit pending writes" operation or not is a separate design decision. If not, then in some languages it can be expressed as an ownership taking operation that still allows for handling the error.

Re: Don't defer Close() on writable files (2017)

#129

Post by SPBS »

I think anything other than deferring Close() and then calling Close() again explicitly is overengineering. Like anything that requires creating a cleanup function, capturing a named return value, requires contorting in unnatural ways to handle a very common scenario. Just... defer Close() the soonest you can (after checking for errors), then call Close() again at the end. Most sane providers of Close() should handle being called multiple times (I know os.File does, as well as sql.Tx ).

Re: Don't defer Close() on writable files (2017)

#130

Post by dwattttt »

jsndnnd wrote: post #127
Agingcoder wrote: post #125

I dislike the multiple close pattern - I was bitten by this behavior years ago, when the second close() ended up closing another file which had been opened between the first and second close ( I think they were actually sockets ). It was a bona fide bug on my side , but it made for unpleasant memories, and a general distrust of such idioms on my side unless there's a language wide guarantee somewhere in the picture.

I don't understand how that could happen, since the original file handle would have been invalidated. Which operating system did you experience this under and was it the operating system, your Libc or what else in the stack which caused this?

The scenario is that after the original file handle is closed, a new open occurs and the file is assigned the same handle value.

Then code acting on the stale handle of the first file closes it, and accidentally closes the new file instead.

Post reply on HN