Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go: Severe memory problems on 32bit Linux

groups.google.com

Post reply on HN
Print view View original

111–120 of 159 posts

Re: Go: Severe memory problems on 32bit Linux

#111

Post by ArbitraryLimits »

ghusbands wrote: post #87
batista wrote: post #67

Earlier quoted context omitted.

>I know from experience that the gc 64-bit version is production ready. Is it really production ready though, or is it the same half-arsed implementation as the 32-bit one, just taking advantage of the larger availability of virtual memory on a 64 bit system?

On 32-bit systems, the problem is that there's a very high chance of non-pointer data looking like pointers to existing data, which, in turn might have its own pointer-like data. This means that you can end up keeping an excessive amount of unreferenced data. However, a 64-bit address space is so much larger that you can't really suffer the same issue. Unlike on 32-bit systems, neither high entropy data nor text will…

Can someone help me out here with a TL;DR? This is my first exposure ever to any technical aspect of Go, and this discussion makes it sound from this discussion like it doesn't use any tag bits in its pointers, but has its garbage collector run heuristics on the data it examines to see whether it "looks like text" to decide whether to collect it? I know that can't possibly be correct.

Re: Go: Severe memory problems on 32bit Linux

#112

Post by 4ad »

blinkingled wrote: post #109
4ad wrote: post #98

Earlier quoted context omitted.

Physical memory does not have anything to do with it, the virtual address space is all that matters. It might be useful to consider that 2^64 is 2^32 * 2^32. This means the problem becomes important on 64 bit only if an application will use 10 orders of magnitude more memory. By considering the historic growth in memory capacity and usage, this will only happen around 2060.

Check out http://code.google.com/p/go/issues/detail?id=909#c32 and the following comment that agrees with it. This isn't just a address space leak - it is a real memory leak. On 64-bit the GC may not be so easily fooled as on 32-bit but it can still be fooled and that is a fundamental problem that will result in memory leaks - if I have 2GB RAM VPS - it doesn't help to have 2^64 bits of address space (actually it is…

Again, this has nothing to do with physical memory. It only has to do with the virtual address space.

Of course if you bump into this, it's a real leak, who said otherwise? And yes, it's possible to artificially generate the collision on 64 bit as it's the same mechanism as with 32 bit. It's about whether it happens frequent enough under normal usage patterns to be a concern. Youtube, and everybody who tried Go in production say it isn't, and that's because of reasons outlined in my first reply to you.

Re: Go: Severe memory problems on 32bit Linux

#113

Post by blinkingled »

4ad wrote: post #112
blinkingled wrote: post #109

Earlier quoted context omitted.

Check out http://code.google.com/p/go/issues/detail?id=909#c32 and the following comment that agrees with it. This isn't just a address space leak - it is a real memory leak. On 64-bit the GC may not be so easily fooled as on 32-bit but it can still be fooled and that is a fundamental problem that will result in memory leaks - if I have 2GB RAM VPS - it doesn't help to have 2^64 bits of address space (actually it is…

Again, this has nothing to do with physical memory. It only has to do with the virtual address space. Of course if you bump into this, it's a real leak, who said otherwise? And yes, it's possible to artificially generate the collision on 64 bit as it's the same mechanism as with 32 bit. It's about whether it happens frequent enough under normal usage patterns to be a concern. Youtube, and everybody who tried Go in pr…

Care to explain why? You keep insisting without explaining. Have you checked #C32 and how it clearly says it is a memory leak on both 32 and 64 bit platforms?

What is your explanation as to why this is not a memory leak and only a address space leak?

[EDIT POST YOUR UPDATE] Ok - so we are on the same page. I wasn't arguing about the likelihood at all - just the fact that it is possible troubled me as a bad GC design. Sure people use lots of crappy software on servers - doesn't mean it's a sound idea :)

Re: Go: Severe memory problems on 32bit Linux

#114

Post by ezyang »

Funnily enough, I understood why they went the conservative GC route. It has to do with the overall Go philosophy, which is that they really do not want features to affect data representation. This has meant no boxing (and no easy polymorphism), and a decision like that has logical consequences for GC too.

Here's to hoping they find a cool solution! It's been a problem for GC's since forever, and if they find a general way of handling the problem I'm sure it will be picked up by many other runtimes.

Re: Go: Severe memory problems on 32bit Linux

#115

Post by 4ad »

blinkingled wrote: post #113
4ad wrote: post #112

Earlier quoted context omitted.

Again, this has nothing to do with physical memory. It only has to do with the virtual address space. Of course if you bump into this, it's a real leak, who said otherwise? And yes, it's possible to artificially generate the collision on 64 bit as it's the same mechanism as with 32 bit. It's about whether it happens frequent enough under normal usage patterns to be a concern. Youtube, and everybody who tried Go in pr…

Care to explain why? You keep insisting without explaining. Have you checked #C32 and how it clearly says it is a memory leak on both 32 and 64 bit platforms? What is your explanation as to why this is not a memory leak and only a address space leak? [EDIT POST YOUR UPDATE] Ok - so we are on the same page. I wasn't arguing about the likelihood at all - just the fact that it is possible troubled me as a bad GC design.…

But it is a real leak, it's just an artificially created leak. These might be interesting to investigate for DOS potential, but they don't happen under regular usage because you are searching a needle in a haystack.

[edit after your update]

Each GC strategy has its drawbacks, for example the one used by Go has the least overhead in extra memory usage, and it's also simple to understand and implement. Mono got its precise GC only last year, it survived 8 years with a conservative GC. Go is only two years old.

Re: Go: Severe memory problems on 32bit Linux

#116

Post by zvrba »

el_muchacho wrote: post #80
zvrba wrote: post #79

Interesting piece of information below in the thread: in reply to "Go being advertised as a systems PL", David Symonds replies with: "It's not. It used to be, but it's not any more."

What is the definition of a system PL ?

The ability to access the raw memory underlying any object.

Re: Go: Severe memory problems on 32bit Linux

#117

Post by pcwalton »

ezyang wrote: post #114

Funnily enough, I understood why they went the conservative GC route. It has to do with the overall Go philosophy, which is that they really do not want features to affect data representation. This has meant no boxing (and no easy polymorphism), and a decision like that has logical consequences for GC too. Here's to hoping they find a cool solution! It's been a problem for GC's since forever, and if they find a gener…

In Rust we're working on a solution for this problem. Essentially, the plan is to have RTTI on garbage-collected data (this is already completed and is used in the cycle collector) and precise stack information for every root on the stack. The latter is in a fork of LLVM: https://github.com/pcwalton/llvm

But aside from the shameless plug, C# and D (I believe), have had precise garbage collectors for quite a while now, and they have similar memory management to Go. It's well-known how to implement it (but that doesn't make it any less hard — I can totally understand why Google opted for conservative GC in the first version).

Re: Go: Severe memory problems on 32bit Linux

#118

Post by rand_r »

4ad wrote: post #84
0xABADC0DA wrote: post #27

It's amazing how nonchalant they are about this 'oh sure we'll fix it in a year or so' when the garbage collector is basically all there needs to be in a Google Go runtime Some comments were asking why early Java's GC wasn't this bad even though it was conservative. The reason is that Java can't take references to fields of an object, so the data mistaken as a pointer has to actually point to an object header. In Goo…

It's not Google Go, it's simply Go, there's not a single Google reference on the web page and that's intentional. Looking at your previous posts here and on Reddit I see you are a notorious anti-Google troll that adds "Google" to "Go" so that your negative posts will be associated with both Google and Go. It's absurd to call it Google Go when there's a first class GNU implementation that has been in development from…

Any recommendation on how we should google for Go related pages?

Re: Go: Severe memory problems on 32bit Linux

#119

Post by drey08 »

nakkiel wrote: post #48
moilolita wrote: post #22

Earlier quoted context omitted.

D has a nice allocation management(i loved the manual+gc approach) and in my benchmarks, optimized D was slightly faster than Go at almost anything and consumed up to 70% less memory (I assumed that was because the Go gc kicked in later and was a bit lazy) but D was weak at threading/synchronization and the documentation of the standard lib was quite messy and lacking. So I decided that in the long run Go will be bet…

I wrote a couple of D programs long ago and I still have a taste of an unfinished language. If I recall correctly, arrays manipulation is pretty weird and the language feels a bit the same PHP feels: a bunch of different things pieced together with no coherence. Compared to Python or Go, it's a whole different world.

What did you find weird about array manipulation in D?

Re: Go: Severe memory problems on 32bit Linux

#120

Post by 4ad »

rand_r wrote: post #118
4ad wrote: post #84

Earlier quoted context omitted.

It's not Google Go, it's simply Go, there's not a single Google reference on the web page and that's intentional. Looking at your previous posts here and on Reddit I see you are a notorious anti-Google troll that adds "Google" to "Go" so that your negative posts will be associated with both Google and Go. It's absurd to call it Google Go when there's a first class GNU implementation that has been in development from…

Any recommendation on how we should google for Go related pages?

Golang or Go programming works fine. If searching for Google Go produces meaningful results, by all means, use it. I was merely complaining about referring to the project as such.

There's also this: http://go-lang.cat-v.org/go-search

Post reply on HN