Tmpfs considered harmful • Hacker News Board

Re: Tmpfs considered harmful

#71

Post by ta12121 » Wed, Sep 12, 2012, 7:56 PM UTC

saurik wrote: post #41

rwmj wrote: post #25

Earlier quoted context omitted.

This is exactly the point of the posting. Every program has to be examined for this problem, and it forces a complex decision about the storage hierarchy onto every programmer.

That's like saying "RAM considered harmful", because every time a developer has to think "do I just store this in memory, or do I sync it to disk" "it forces a complex decision about the storage hierarchy": /tmp is simply RAM that has the API of a file; it used to be implemented using a horrible kludge with files on disk that were periodically deleted, but with tmpfs the dream became a reality.

    /tmp is simply RAM that has the API of a file

...which used to behave, physically, as disk, and now behaves as RAM. If I'm doing large file manipulations that require temporary files larger than RAM where should I put them?

Re: Tmpfs considered harmful

#72

Post by regularfry » Wed, Sep 12, 2012, 8:07 PM UTC

ta12121 wrote: post #71

saurik wrote: post #41

Earlier quoted context omitted.

That's like saying "RAM considered harmful", because every time a developer has to think "do I just store this in memory, or do I sync it to disk" "it forces a complex decision about the storage hierarchy": /tmp is simply RAM that has the API of a file; it used to be implemented using a horrible kludge with files on disk that were periodically deleted, but with tmpfs the dream became a reality.

/tmp is simply RAM that has the API of a file ...which used to behave, physically, as disk, and now behaves as RAM. If I'm doing large file manipulations that require temporary files larger than RAM where should I put them?

/tmp. If it's tmpfs, it can expand into swap space.

Re: Tmpfs considered harmful

#73

Post by regularfry » Wed, Sep 12, 2012, 8:12 PM UTC

RobAley wrote: post #22

raverbashing wrote: post #7

"Everyone must now be careful never to store a file in /tmp that might grow large" If you're going to trash my /tmp with gigantic files please don't. Really "it’s better to fix the filesystem to make it faster" Yes, let's spend adding more complexity to the file system to work out every /tmp abuser out there. But his original point was avoiding adding complexity in the first place So basically, he contradicts himself…

>If you're going to trash my /tmp with gigantic files please don't. Really I write statistical programs that often generate temporary intermediate files of multiple GB size, which need to persist at the end of runs for trouble-shooting, partial re-runs etc. i.e. the software shouldn't delete them when it's finished, but they don't need to be kept long term. I kept forgetting to delete them when I finished a session a…

Sounds like a job for /var/cache.

Re: Tmpfs considered harmful

#74

Post by ta12121 » Wed, Sep 12, 2012, 9:10 PM UTC

regularfry wrote: post #72

ta12121 wrote: post #71

Earlier quoted context omitted.

/tmp is simply RAM that has the API of a file ...which used to behave, physically, as disk, and now behaves as RAM. If I'm doing large file manipulations that require temporary files larger than RAM where should I put them?

/tmp. If it's tmpfs, it can expand into swap space.

Swap is usually no bigger than two or three times your RAM. Disk in general is 10 to 100 times the size of your RAM.

Re: Tmpfs considered harmful

#75

Post by luser001 » Wed, Sep 12, 2012, 9:26 PM UTC

ta12121 wrote: post #71

saurik wrote: post #41

Earlier quoted context omitted.

That's like saying "RAM considered harmful", because every time a developer has to think "do I just store this in memory, or do I sync it to disk" "it forces a complex decision about the storage hierarchy": /tmp is simply RAM that has the API of a file; it used to be implemented using a horrible kludge with files on disk that were periodically deleted, but with tmpfs the dream became a reality.

/tmp is simply RAM that has the API of a file ...which used to behave, physically, as disk, and now behaves as RAM. If I'm doing large file manipulations that require temporary files larger than RAM where should I put them?

According to the comments on the original article, that location is /var/tmp. Apparently it's some sort of convention/standard.

First time I've heard about /var/tmp.

Re: Tmpfs considered harmful

#76

Post by TylerE » Wed, Sep 12, 2012, 9:52 PM UTC

prodigal_erik wrote: post #53

saurik wrote: post #41

Earlier quoted context omitted.

That's like saying "RAM considered harmful", because every time a developer has to think "do I just store this in memory, or do I sync it to disk" "it forces a complex decision about the storage hierarchy": /tmp is simply RAM that has the API of a file; it used to be implemented using a horrible kludge with files on disk that were periodically deleted, but with tmpfs the dream became a reality.

RAM really is a harmful kludge. Neither Turing machines nor lambda calculus call for some of an algorithm's state to suddenly be obliterated. The only reason we put up with it is that disk and SSD are so much slower.

I've never understood the obsession with Turing machines. It's like basing one's core belief of what a car is off one of Da Vinci's crazy wind-up things.

Re: Tmpfs considered harmful

#77

Post by rogerbinns » Wed, Sep 12, 2012, 10:07 PM UTC

notatoad wrote: post #57

Why on earth would anybody ever use tmpfs for /tmp? If I'm writing a program that deals with a big file, and I want to temporarily write something to the filesystem, it is because I dont want it in memory. Why would you ever write to /tmp for any reason other than to get something out of ram for a while?

I have been using tmpfs for /tmp for many years (since 2005 IIRC). Note that I do have large amounts of swap space (typically 64GB).

One simple benefit is that files there do not get caught in backups - they are temporary detritus so I don't want backup space wasted by them.

The major benefit is that fsync is really slow on Linux. Typically it turns into a sync, and a lot of programs like to ensure you don't lose data by being fsync happy. For temporary stuff that is even more painful - you don't care about data loss - that is why it is in tmp in the first place.

TLDR: I prefer filesystem operations on transient/temporary files and data to run at the speed of RAM, and not end up in backups or waiting for syncs.

Re: Tmpfs considered harmful

#78

Post by dredmorbius » Wed, Sep 12, 2012, 10:51 PM UTC

alexchamberlain wrote: post #5

praptak wrote: post #4

Minor disagreement about the following: "Everyone must worry about whether their files need to survive a reboot" I believe there was never such expectation of /tmp. On many systems it's being wiped periodically or even when the user's last session ends. The example app(s) that use /tmp for drafts need(s) to be fixed.

Totally agree, /tmp should be wiped at the end of each session.

The key is that it may be wiped. Files stored in /tmp are not guaranteed to survive boot. Whether or not they are there or not is a bit of a gamble (a regularly losing one should you choose tmpfs).

Re: Tmpfs considered harmful

#79

Post by digikata » Wed, Sep 12, 2012, 11:54 PM UTC

notatoad wrote: post #57

Why on earth would anybody ever use tmpfs for /tmp? If I'm writing a program that deals with a big file, and I want to temporarily write something to the filesystem, it is because I dont want it in memory. Why would you ever write to /tmp for any reason other than to get something out of ram for a while?

tmpfs is useful for embedded systems where the primary filesystem is mounted read-only. Generally /tmp and /var may be tmpfs backed in that situation. Not sure why a desktop system would want this though.

Re: Tmpfs considered harmful

#80

Post by cma » Thu, Sep 13, 2012, 1:09 AM UTC

prodigal_erik wrote: post #53

saurik wrote: post #41

Earlier quoted context omitted.

That's like saying "RAM considered harmful", because every time a developer has to think "do I just store this in memory, or do I sync it to disk" "it forces a complex decision about the storage hierarchy": /tmp is simply RAM that has the API of a file; it used to be implemented using a horrible kludge with files on disk that were periodically deleted, but with tmpfs the dream became a reality.

RAM really is a harmful kludge. Neither Turing machines nor lambda calculus call for some of an algorithm's state to suddenly be obliterated. The only reason we put up with it is that disk and SSD are so much slower.

Ram is just state. Turing machines require a non-tape state machine mechanism. You could consider ram a part of that, 4 gigs corresponding to many many possible states.