Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Filesystem devs should aim to make “badly written” app code “just work” (2009)

lwn.net

Post reply on HN
Print view View original

21–30 of 117 posts

Re: Filesystem devs should aim to make “badly written” app code “just work” (2009)

#21

Post by teddyh »

mehrdadn wrote: post #4

Quite a bizarre read considering Linux's decision to e.g. overcommit memory makes 100% correctly written code break nondeterministically...

The difference is that people do tend to check return values of write() (in cases where it actually matters), but not check the return value of malloc(). Since Linux must work with the code which exists, and since most programs tend to malloc() memory but never use it, the overcommitting practice begins to make sense. A twisted and ugly sense, but that is the world we live in.

Re: Filesystem devs should aim to make “badly written” app code “just work” (2009)

#22

Post by mehrdadn »

teddyh wrote: post #21
mehrdadn wrote: post #4

Quite a bizarre read considering Linux's decision to e.g. overcommit memory makes 100% correctly written code break nondeterministically...

The difference is that people do tend to check return values of write() (in cases where it actually matters), but not check the return value of malloc(). Since Linux must work with the code which exists, and since most programs tend to malloc() memory but never use it, the overcommitting practice begins to make sense. A twisted and ugly sense, but that is the world we live in.

That's at best a reason to allow over-commitment to be enabled or disabled on a per-program basis, not a reason to force it on the whole system so that it becomes outright impossible to write correct code.

Re: Filesystem devs should aim to make “badly written” app code “just work” (2009)

#23

Post by blazespin »

pdonis wrote: post #8

The thread title omits a crucial word: "Filesystem" (as in "filesystem people", not just "people"). The point he is making is that filesystems are supposed to be utterly reliable; applications should not have to take extreme precautions to avoid having the filesystem lose their data. And the fact that practically nobody actually takes any precautions, let alone extreme ones, is strong evidence that programmers do in…

“File system people should make badly written application code just work.”

Re: Filesystem devs should aim to make “badly written” app code “just work” (2009)

#24

Post by blazespin »

1gor wrote: post #17

APIs you provide to consumers should aim to make their badly written code just work. That's what Linus said.

Yeah. So many api writers aim to force clients do all the heavy lifting. The whole point of a good api is that it reduces heavy lifting. Anyone can write pass through apis that don’t do anything.

Re: Filesystem devs should aim to make “badly written” app code “just work” (2009)

#25

Post by peterwwillis »

mehrdadn wrote: post #4

Quite a bizarre read considering Linux's decision to e.g. overcommit memory makes 100% correctly written code break nondeterministically...

I think the difference between "don't implement workarounds" and "use overcommit" is that the kernel can try to be clever, but userland should not have to be clever. The kernel is supposed to just make things work for userland.

I think that's why overcommit exists. It's generally not easy to re-design all userland applications to deal with difficult memory management problems in a complex system, but it is easy to just lie to them all so they can continue to do stupid things and not crash. If you have a system with only "100% correctly written" software, just disable overcommit (echo 2 > /proc/sys/vm/overcommit_memory).

Re: Filesystem devs should aim to make “badly written” app code “just work” (2009)

#26

Post by dpark »

blazespin wrote: post #24
1gor wrote: post #17

APIs you provide to consumers should aim to make their badly written code just work. That's what Linus said.

Yeah. So many api writers aim to force clients do all the heavy lifting. The whole point of a good api is that it reduces heavy lifting. Anyone can write pass through apis that don’t do anything.

> The whole point of a good api is that it reduces heavy lifting.

Isn’t that the opposite of what Torvalds is saying? He seems to be arguing for simplicity. APIs that do a bunch of magic for you are the opposite of simple and tend to be mountains of subtle bugs and unexpected behavior.

Re: Filesystem devs should aim to make “badly written” app code “just work” (2009)

#28

Post by sn41 »

blazespin wrote: post #23
pdonis wrote: post #8

The thread title omits a crucial word: "Filesystem" (as in "filesystem people", not just "people"). The point he is making is that filesystems are supposed to be utterly reliable; applications should not have to take extreme precautions to avoid having the filesystem lose their data. And the fact that practically nobody actually takes any precautions, let alone extreme ones, is strong evidence that programmers do in…

“File system people should make badly written application code just work.”

That makes much more sense. And it turns out to be the opposite of what I thought when I read the title. I had initially thought - just write bad code whose only merit is that it works.

Re: Filesystem devs should aim to make “badly written” app code “just work” (2009)

#29

Post by bvinc »

I think he's got a point.

As a developer I find myself in a different scenario. I'm usually trying to find out what exactly the 100% guaranteed way to do something is. Instead, I find incomplete documentation and different people with different opinions on what the guarantees are, and most people writing bad code that they assume will usually work.

Just modifying a file in an atomic way requires a complicated dance of multiple files and multiple syncs and a rarely tested cleanup routine the next time the file is opened. No one does this.

I don't know what the solution is.

Re: Filesystem devs should aim to make “badly written” app code “just work” (2009)

#30

Post by sillysaurusx »

zer00eyz wrote: post #5

"Anybody who wants more complex and subtle filesystem interfaces is just crazy. Not only will they never get used, they'll definitely not be stable." I think there is a more universal truism here - that "complex and subtle" are sources of pain, problems and headaches. I want to write "cool" and "magical" code as much as the next person, but that's the stuff that I look at later WTF because I am no longer in the same…

I am currently trying to wring maximum performance from Cloud TPUs. This comment really resonated with me, because Cloud TPUs are complex, subtle, overly complicated, and have twenty opaque ways that you can ruin your performance.

Compare that with a GPU. It took decades, but (at least for nvidia GPUs) they have finally reached the point where you can be at least reasonably certain that high-performance is the default. Sure, you can ruin your performance by doing some stupid things, but it's at least harder to do those stupid things. The defaults are high-performance.

It's exactly the opposite for Cloud TPUs. By default, it only uses one core. You have to use their TPUEstimator API, which is a byzantine mess. Since two weeks ago, I've spent roughly 3 solid working days of effort solely trying to read and understand (a) what is the TPUEstimator actually doing? (b) why is it supposedly so much faster?

I have some half-hearted justifications -- the answer seems to be "you have to colocate your gradients with the device; you have to scope your tensorflow graph to a specific device; and each TPU core is a separate device." But there are unanswered questions. For example, you're supposed to pass your tensorflow computation to tpu.rewrite(). Yet I've never done that. The defaults seem to just work. So does that mean it does the rewriting for me automatically? Is it being emulated in software, and I'm ruining my performance? Tensorflow, why don't you just crash instead of being so damn slow on TPUs? That would at least let me aim my optimizations!

If only I had low level access to the actual TPU operations, I could just write a compiler that specifically emits instructions to give me the performance I need. But this tensorflow graph abstraction makes everything "easy" yet exponentially more complicated.

Anyway. Yes. More of your mindset, please. Simplicity is such a lovely metric seldom optimized for.

(Anyone who's curious can see a dramatic 11x difference in performance on GPUs vs TPUs in Colab: https://twitter.com/theshawwn/status/1196593451174891520 ... this was very surprising, since Google's marketing would have you believe that TPUs are the bee's knees. Yet by default a TPUv2 is 11x slower than a K80 GPU for some basic matrix multiplications.)

Post reply on HN