Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Does memory leak? (1995)

groups.google.com

Post reply on HN
Print view View original

71–80 of 289 posts

Re: Does memory leak? (1995)

#71

Post by jmpman »

GordonS wrote: post #42

A bit OT, but I wonder how I'd feel if I was offered a job working on software for missiles. I'm sure the technical challenge would be immensely interesting, and I could tell myself that I cared more about accuracy and correctness than other potential hires... but from a moral standpoint, I don't think I could bring myself to do it. I realise of course that the military uses all sorts of software, including line of b…

Straight out of college, I was offered a job writing software for missiles. Extremely interesting area, working for my adjunct professor’s team, who I highly admired and whose class was the best of my college career. The pay was on par with all my other offers. I didn’t accept for two reasons.

First, I logically agreed that the missiles were supporting our armed services and I believed that our government was generally on the right side of history and needed the best technology to continue defending our freedoms. However, a job, when executed with passion, becomes a very defining core of your identity. I didn’t want death and destruction as my core. I support and admire my college friends who did accept such jobs, but it just wasn’t for me.

Second, I had interned at a government contractor, (not the missile manufacturer), and what I saw deeply disturbed me. I came on to a project which was 5 years into a 3 year schedule, and not expected to ship for another 2 years. Shocked, I asked my team lead “Why didn’t the government just cancel the contract and assign the work to another company?”, her reply, “If they did that, the product likely wouldn’t be delivered in under two years, so they stick with us”. I understood that this mentality was pervasive, and would ultimately become part of me, if I continued to work for that company. That mentality was completely unacceptable in the competitive commercial world, and I feared the complacency which would infect me and not prepare me for the eventual time when I’d need to look for a job outside that company. As a graduating senior, I attended our college job fair, and when speaking with another (non missile) government contractor, I told the recruiter that I was hesitant working for a his company because I thought it wouldn’t keep me as competitive throughout my career. I repeated the story from my internship, and asked if I’d find the same mentality at his company. His face dropped the cheerful recruiter facade, when he pulled me aside and sternly instructed “You should never repeat that story”. I took that as an overwhelming “yes”. So, my concern was that working for this missile manufacturer, this government contractor mentality would work its way into their company (if it hadn’t already), and it would be bad for my long term career. I wanted to remain competitive on a global commercial scale, without relying upon government support.

Re: Does memory leak? (1995)

#72

Post by thaumasiotes »

derefr wrote: post #61
thaumasiotes wrote: post #43

Earlier quoted context omitted.

I mean, 'cat' does something so simple (apply the identity function to the input) that it has no need to be reusable because there's no point using it in the first place. If you have input, processing it with cat just means you wasted your time to produce something you already had.

The point of cat(1), short for concatenate , is to feed a pipeline multiple concatenated files as input, whereas shell stdin redirection only allows you to feed a shell a single file as input. This is actually highly flexible, since cat(1) recognizes the “-“ argument to mean stdin, and so you can `cat a - b` in the middle of a pipeline to “wrap” the output of the previous stage in the contents of files a and b (which…

But that is a case where you have several filenames and you want to concatenate the files. The work you're using cat to do is to locate and read the files based on the filename. If you already have the data stream(s), cat does nothing for you; you have to choose the order you want to read them in, but that's also true when you invoke cat.

This is the conceptual difference between

    pipeline | cat       # does nothing
and

    pipeline | xargs cat # leverages cat's ability to open files
Opening files isn't really something I think of cat as doing in its capacity as cat. It's something all the command line utilities do equally.

Re: Does memory leak? (1995)

#73

Post by ksherlock »

simias wrote: post #13

I think it's a bad mindset to leak resources even when it doesn't effectively matter. In non-garbage collected languages especially, because it's important to keep in mind who owns what and for how long. It also makes refactoring easier because leaked resources effectively become some sort of implicit global state you need to keep track of. If a function that was originally called only once at startup is not called r…

"I gave cp more than a day to [free memory before exiting], before giving up and killing the process."

https://news.ycombinator.com/item?id=8305283

https://lists.gnu.org/archive/html/coreutils/2014-08/msg0001...

Re: Does memory leak? (1995)

#74

Post by acqq »

crawshaw wrote: post #7

This is an example of garbage collection being more CPU efficient than manual memory management. It has limited application, but there is a more common variant: let process exit clean up the heap. You can use an efficient bump allocator for `malloc` and make `free` a no-op.

There was also a variant of it with the hard drives: building Windows produced a huge amount of object files, so the trick used was to use a whole hard disk (or a partition) for that. Before the next rebuild, deleting all the files would took far more time than a "quick" reformatting of the whole hard disk, so the later was used.

(I am unable to find a link that talks about that, however).

In general, throwing away at once the set of the things together with the structures that maintain it is always faster than throwing away every item one by one while maintaining the consistency of the structures, in spite of the knowledge that all that is not needed at the end.

An example of arenas in C: "Fast Allocation and Deallocation of Memory Based on Object Lifetimes", Hanson, 1988:

ftp://ftp.cs.princeton.edu/techreports/1988/191.pdf

Re: Does memory leak? (1995)

#75

Post by [deleted] »

GordonS wrote: post #42

A bit OT, but I wonder how I'd feel if I was offered a job working on software for missiles. I'm sure the technical challenge would be immensely interesting, and I could tell myself that I cared more about accuracy and correctness than other potential hires... but from a moral standpoint, I don't think I could bring myself to do it. I realise of course that the military uses all sorts of software, including line of b…

[deleted]

Re: Does memory leak? (1995)

#76

Post by derefr »

thaumasiotes wrote: post #72
derefr wrote: post #61

Earlier quoted context omitted.

The point of cat(1), short for concatenate , is to feed a pipeline multiple concatenated files as input, whereas shell stdin redirection only allows you to feed a shell a single file as input. This is actually highly flexible, since cat(1) recognizes the “-“ argument to mean stdin, and so you can `cat a - b` in the middle of a pipeline to “wrap” the output of the previous stage in the contents of files a and b (which…

But that is a case where you have several filenames and you want to concatenate the files . The work you're using cat to do is to locate and read the files based on the filename. If you already have the data stream(s), cat does nothing for you; you have to choose the order you want to read them in, but that's also true when you invoke cat. This is the conceptual difference between pipeline | cat # does nothing and pi…

    pipeline | cat    # does nothing
This is actually re-batching stdin into line-oriented write chunks, IIRC. If you write a program to manually select(2) + fread(2) from stdin, then you’ll observe slightly different behaviour between e.g.

    dd if=./file | myprogram
and

    dd if=./file | cat | myprogram
On the former, select(2) will wake your program up with dd(1)’s default obs (output block size) worth of bytes in the stdin kernel buffer; whereas, on the latter, select(2) will wake your program up with one line’s worth of input in the buffer.

Also, if you have multiple data streams, by using e.g. explicit file descriptor redirection in your shell, ala

    (baz | quux) >4
...then cat(1) won’t even help you there. No tooling from POSIX or GNU really supports consuming those streams, AFAIK.

But it’s pretty simple to instead target the streams into explicit fifo files, and then concatenate those with cat(1).

Re: Does memory leak? (1995)

#77

Post by wbhart »

willvarfar wrote: post #62
wbhart wrote: post #46

Missiles don't always hit their intended target. They can go off course, potentially be hacked, fall into the wrong hands, be sold to mass murderers, fail to explode, accidentally fall out of planes (even nuclear bombs have historically done this), miss their targets, encounter countermeasures, etc. Nobody is claiming that this was done for reasons of good software design. It's perfectly reasonable to suspect it was…

A simple bump allocator with no reclaim is fairly common in embedded code. Garbage collection makes the performance of the code much less deterministic. A lot of embedded loops running on embedded in-order cpus without an operating system use cycle count as a timing mechanism etc.

Right, but that isn't the argument that was being used here, which is my point. The way I read it, the contractor cared only enough to get the design over the line so the customer would sign off on it. Their argument was that you shouldn't care about leaks due to scheduled deconstruction, not because of a technical consideration.

There exist options between no reclaim and using a garbage collector which could be considered, depending on the exact technical specifications of the hardware it was running on and the era in which it happened.

But retrofitting technical reasoning about why this may have been done is superfluous. The contractor already said why they did it, and the subtext of the original post is that it was flippant and hilarious.

Re: Does memory leak? (1995)

#78

Post by bob1029 »

Out_of_Characte wrote: post #37

What an interesting concept. Good programmers always consider certain behaviours to be wrong. Memory 'leaks' being one of them. But this real application of purposefully not managing memory is also an interesting thought exercise. However counter intuitive, a memory leak in this case might be the most optimal solution in this problem space. I just never thought I would have to think of an object's lifetime in such a…

It is interesting what you can come up with if you rely on constraints in the physical realm to inform your virtual realm choices. I've been looking at various highly-available application architectures and came across a similar idea to the missile equation in the article. If you are on a single box your hands are tied. But, if you have an N+1 (or more) architecture, things can get fun.

In theory, you could have a cluster of identical nodes each handling client requests (i.e. behind a load balancer). Each node would monitor its own application memory utilization and automatically cycle itself after some threshold is hit (after draining its buffers). From the perspective of the programmer, you now get to operate in a magical domain where you can allocate whatever you want and never think about how it has to be cleaned up. Obviously, you wouldn't want to maliciously use malloc, but as long as the cycle time of each run is longer than a few minutes I feel the overhead is accounted for.

Also, the above concept could apply to a single node with multiple independent processes performing the same feat, but there may be some increased concerns with memory fragmentation at the OS-level. Worst case with the distributed cluster of nodes, you can simply power cycle the entire node to wipe memory at the physical level and then bring it back up as a clean slate.

Re: Does memory leak? (1995)

#79

Post by FpUser »

"Since the missile will explode when it hits it's target or at the end of it's flight, the ultimate in garbage collection is performed without programmer intervention."

I just can't stop laughing over this "ultimate in garbage collection". What a guy.

Btw we dealt a lot with Rational in the 90's. I might have even met him.

Re: Does memory leak? (1995)

#80

Post by saagarjha »

derefr wrote: post #68

Erlang has a parameter called initial_heap_size. Each new actor-process in Erlang gets its own isolated heap, for which it does its own garbage-collection on its own execution thread. This initial_heap_size parameter determines how large each newly-spawned actor’s heap will be. Why would you tune it? Because, if you set it high enough, then for all your short-lived actors, memory allocation will become a no-op (= bum…

Note that most general-purpose allocators also keep around internal arenas from which they hand out memory.
Post reply on HN