[deleted]
Go: Severe memory problems on 32bit Linux
11–20 of 159 posts
Re: Go: Severe memory problems on 32bit Linux
#12Wow. That third message, with suggestions for avoiding the bug, reads like a twisted joke. Highlights: "avoid struct types which contain both integer and pointer fields" "avoid data structures which form densely interconnected graphs at run-time" "avoid integer values which may alias at run-time to an address; make sure most integer values are fairly low (such as: below 10000)" I understand that this isn't a complete…
Unfortunately you are right. For me this is a very unpleasant surprise after I put aside D and rewrote my little framework in Go. Everybody said that the gc is not final, that there are some performance issues and they are working on it but I never imagined that such catastrophic bugs are not solved by now.
Re: Go: Severe memory problems on 32bit Linux
#13Wow. That third message, with suggestions for avoiding the bug, reads like a twisted joke. Highlights: "avoid struct types which contain both integer and pointer fields" "avoid data structures which form densely interconnected graphs at run-time" "avoid integer values which may alias at run-time to an address; make sure most integer values are fairly low (such as: below 10000)" I understand that this isn't a complete…
Go 1 defines and the backwards-compatibility guarantees one can expect as the language matures. It's more a statement about the language specification than it is about the implementations.
I know from experience that the gc 64-bit version is production ready. It's unfortunate that the limitations of the 32-bit version are not called out clearly on the website.
Re: Go: Severe memory problems on 32bit Linux
#14However, it does not address the base issue, which is that Go uses a conservative garbage collector, and more values look like pointers in a 32-bit world.
The only real fix would be to improve the garbage collector's understanding of which values are pointers and which are something else (e.g., floating point numbers that happen to look like pointers). And that is not an easy fix.
How does this GC work? Is it literally just marching through the heap looking for pointer-sized values in the range that has been mapped to the process?
Re: Go: Severe memory problems on 32bit Linux
#15Wow. That third message, with suggestions for avoiding the bug, reads like a twisted joke. Highlights: "avoid struct types which contain both integer and pointer fields" "avoid data structures which form densely interconnected graphs at run-time" "avoid integer values which may alias at run-time to an address; make sure most integer values are fairly low (such as: below 10000)" I understand that this isn't a complete…
Unfortunately you are right. For me this is a very unpleasant surprise after I put aside D and rewrote my little framework in Go. Everybody said that the gc is not final, that there are some performance issues and they are working on it but I never imagined that such catastrophic bugs are not solved by now.
I also looked at D/Go -- I predict the reign of JVM is over (quote me).
I wouldn't bother with the 32 bit hiccups. Go hits the sweet spot pretty well. You have chosen well. Hang in there.
Re: Go: Severe memory problems on 32bit Linux
#16However, it does not address the base issue, which is that Go uses a conservative garbage collector, and more values look like pointers in a 32-bit world. The only real fix would be to improve the garbage collector's understanding of which values are pointers and which are something else (e.g., floating point numbers that happen to look like pointers). And that is not an easy fix. How does this GC work? Is it literal…
The heap AND the data section. Although apparently, gcc-go doesn't do that for the data section.
Re: Go: Severe memory problems on 32bit Linux
#17This is the relevant bug: http://code.google.com/p/go/issues/detail?id=909 As the discussion there and in the thread says, the root problem is that Go uses a conservative garbage collector, and on 32-bit a lot more values look like pointers than on 64-bit, so many more things don't get freed in long-running processes. Seems not to be easy to fix.
Publicizing the bug may speed the fix along. Here's the real test: is hitting the front page of HN going to make a difference?
[deleted]
Re: Go: Severe memory problems on 32bit Linux
#18Earlier quoted context omitted.
Publicizing the bug may speed the fix along. Here's the real test: is hitting the front page of HN going to make a difference?
I don't think this one is lacking interest, but an obvious fix. It looks like Russ Cox did commit a fix for some portion of the cases. But a better fix would either need to significantly modify the way things are laid out to reduce the odds of false positives, or else move to a precise GC. (Precise GCs are possible in C-like languages, but trickier to implement. Here's a recent paper on one: http://www.cs.utah.edu/~r…
Von Neumann architecture falsely lead us to the path with the singular notion of "memory". We need to distinguish at the hardware architecture level between "working memory" and "persistent memory".
I doodle all the time when working. Pristine set of diagrams emerges from the chaos. I wish my computer would play along with this regime ..
Re: Go: Severe memory problems on 32bit Linux
#19However, it does not address the base issue, which is that Go uses a conservative garbage collector, and more values look like pointers in a 32-bit world. The only real fix would be to improve the garbage collector's understanding of which values are pointers and which are something else (e.g., floating point numbers that happen to look like pointers). And that is not an easy fix. How does this GC work? Is it literal…
Yep, that's basically it.
It sounds crazy (and it is!) - but it often works reasonably well in practice. SBCL (one of the most performant Common Lisp implementations) has an 'imprecise gc' that works the same way - and I've seen reasonably heavily stressed processes with uptimes in the weeks/months.
Remember, even a few years ago (before fastthread, etc) a reasonably loaded Ruby on Rails app couldn't stay up for more than ~10 minutes w/o memory leaks forcing a restart (DHH said 37Signals as doing ~400 restarts/day per process, IIRC) because the runtime was such a piece of crap. Yet, many people still used it to solve real problems and make real money.
At least Go's memory leaks are much slower than Ruby's ;-)
Re: Go: Severe memory problems on 32bit Linux
#20However, it does not address the base issue, which is that Go uses a conservative garbage collector, and more values look like pointers in a 32-bit world. The only real fix would be to improve the garbage collector's understanding of which values are pointers and which are something else (e.g., floating point numbers that happen to look like pointers). And that is not an easy fix. How does this GC work? Is it literal…
The heap AND the data section. Although apparently, gcc-go doesn't do that for the data section.
That would seem to also present a DOS vector (even on 64 bit) if a user can get the program to store data (of any type, e.g. char or floating point) that happen to be binary-equivalent to pointers to large allocations.