Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go: Severe memory problems on 32bit Linux

groups.google.com

Post reply on HN
Print view View original

41–50 of 159 posts

Re: Go: Severe memory problems on 32bit Linux

#41

Post by derleth »

pcwalton wrote: post #31
smanek wrote: post #30

Earlier quoted context omitted.

I don't think any GC scheme could work with that. Languages that allow pointer arithmetic like that basically can't be GC'd. The problem with an imprecise GC is that having an integer that looks like a pointer, could prevent an object from being freed.

Almost — C# has a cool feature whereby you can do pointer arithmetic if you pin the objects in question first, so the GC won't collect or move them. (The language statically enforces this by forbidding you from taking the address of a value until you pin it.)

That's cool; I think another way of doing it is to allow programs to request a sandbox where the GC doesn't go so they can do all of the pointer-arithmeticking they want as long as all their pointers fall within the sandbox by the time they're written through or dereferenced.

Re: Go: Severe memory problems on 32bit Linux

#42

Post by sliverstorm »

smanek wrote: post #19
jedbrown wrote: post #14

However, it does not address the base issue, which is that Go uses a conservative garbage collector, and more values look like pointers in a 32-bit world. The only real fix would be to improve the garbage collector's understanding of which values are pointers and which are something else (e.g., floating point numbers that happen to look like pointers). And that is not an easy fix. How does this GC work? Is it literal…

Yep, that's basically it. It sounds crazy (and it is!) - but it often works reasonably well in practice. SBCL (one of the most performant Common Lisp implementations) has an 'imprecise gc' that works the same way - and I've seen reasonably heavily stressed processes with uptimes in the weeks/months. Remember, even a few years ago (before fastthread, etc) a reasonably loaded Ruby on Rails app couldn't stay up for more…

Sometimes I wonder if the incredible fall in DRAM prices was really a good thing... ;)

Re: Go: Severe memory problems on 32bit Linux

#43

Post by sanderjd »

flomo wrote: post #35
smanek wrote: post #29

Earlier quoted context omitted.

There was some back-and-forth between Zed Shaw and DHH that I remember being interested by at the time. Zed deleted all his posts - but here are some places to get started: - Shaw's opening volley: http://web.archive.org/web/20080103072111/http://www.zedshaw... [lots of stupid personal flames - but his technical points are consistent with what I remember from the time] - DHH's response: http://david.heinemeierhansson…

I certainly wasn't anywhere in the loop, but at least on my site, FastCGI didn't work, and Mongrel did. Zed's rant/flame goes into so much personal detail that he doesn't really articulate the point. Rails was receiving this enormous amount of hype, but there wasn't even a working application server yet.

I started using rails after all of that, but regardless of how it was back then, I think it's worth pointing out for people who aren't aware - there are really good app servers for rails now.

Re: Go: Severe memory problems on 32bit Linux

#44

Post by joe_the_user »

smanek wrote: post #19
jedbrown wrote: post #14

However, it does not address the base issue, which is that Go uses a conservative garbage collector, and more values look like pointers in a 32-bit world. The only real fix would be to improve the garbage collector's understanding of which values are pointers and which are something else (e.g., floating point numbers that happen to look like pointers). And that is not an easy fix. How does this GC work? Is it literal…

Yep, that's basically it. It sounds crazy (and it is!) - but it often works reasonably well in practice. SBCL (one of the most performant Common Lisp implementations) has an 'imprecise gc' that works the same way - and I've seen reasonably heavily stressed processes with uptimes in the weeks/months. Remember, even a few years ago (before fastthread, etc) a reasonably loaded Ruby on Rails app couldn't stay up for more…

But Ruby is more or less a "rapid development" environment.

You pay for quick development turn-around with machines.

Go is not in that kind of space as far as I know since, as I understand it, Go is sold as low-ish level language competing with C++ and C for system development. For something that would replace those languages, unexplained, systematic leaks would be bad (I mean, my C++ programs might leak but it's reasonably easy to discover why. That matters).

Re: Go: Severe memory problems on 32bit Linux

#45

Post by akeefer »

r00tbeer wrote: post #21
_delirium wrote: post #3

This is the relevant bug: http://code.google.com/p/go/issues/detail?id=909 As the discussion there and in the thread says, the root problem is that Go uses a conservative garbage collector, and on 32-bit a lot more values look like pointers than on 64-bit, so many more things don't get freed in long-running processes. Seems not to be easy to fix.

I know early JVM garabage collectors were also "conservative", but I don't recall JVMs running into these sorts of problems. Maybe folks are just using more of the 4G address space these days than back in the day of conservative GC JVMs?

Within the JVM, object references are strictly separate from integers. Object references are stored within a structure they call an OOP, short for "ordinary object pointers", and the gc works by inspecting the contents of oops and potentially changing them if objects are moved around in memory (as happens if the gc is moving things between generations, or doing heap compaction). If you're curious about it, you can get some information about OOPs on the page discussing the CompressedOops flag implementation: https://wikis.oracle.com/display/HotSpotInternals/Compressed...

Re: Go: Severe memory problems on 32bit Linux

#46

Post by nakkiel »

BarkMore wrote: post #13
wtallis wrote: post #2

Wow. That third message, with suggestions for avoiding the bug, reads like a twisted joke. Highlights: "avoid struct types which contain both integer and pointer fields" "avoid data structures which form densely interconnected graphs at run-time" "avoid integer values which may alias at run-time to an address; make sure most integer values are fairly low (such as: below 10000)" I understand that this isn't a complete…

Go 1 defines and the backwards-compatibility guarantees one can expect as the language matures. It's more a statement about the language specification than it is about the implementations. I know from experience that the gc 64-bit version is production ready. It's unfortunate that the limitations of the 32-bit version are not called out clearly on the website.

Clearly, there was a need for "Hey, things won't change as much from now on". Now, I'm still afraid little work will appear on the GC, which is IMO Go's biggest weakness to date.

Re: Go: Severe memory problems on 32bit Linux

#47

Post by moilolita »

eternalban wrote: post #15
moilolita wrote: post #12

Earlier quoted context omitted.

Unfortunately you are right. For me this is a very unpleasant surprise after I put aside D and rewrote my little framework in Go. Everybody said that the gc is not final, that there are some performance issues and they are working on it but I never imagined that such catastrophic bugs are not solved by now.

I also looked at D/Go -- I predict the reign of JVM is over (quote me). I wouldn't bother with the 32 bit hiccups. Go hits the sweet spot pretty well. You have chosen well. Hang in there.

If Ceylon/Rust are not ready today it means they will be immature for at least 2 years from their release date. Without a large community behind them, they will slowly fade away like D did.In order to get community, they have to have a good/clean/pragmatic design and a library as comprehensive as possible. If they don't have a good library they better have excellent interop with other platforms or they will be doomed. IMO Go is a solid step in the right direction and if the authors will resist the temptation to complicate it with (too many) new features&paradigms I think it will fare well.

Re: Go: Severe memory problems on 32bit Linux

#48

Post by nakkiel »

moilolita wrote: post #22
eternalban wrote: post #15

Earlier quoted context omitted.

I also looked at D/Go -- I predict the reign of JVM is over (quote me). I wouldn't bother with the 32 bit hiccups. Go hits the sweet spot pretty well. You have chosen well. Hang in there.

D has a nice allocation management(i loved the manual+gc approach) and in my benchmarks, optimized D was slightly faster than Go at almost anything and consumed up to 70% less memory (I assumed that was because the Go gc kicked in later and was a bit lazy) but D was weak at threading/synchronization and the documentation of the standard lib was quite messy and lacking. So I decided that in the long run Go will be bet…

I wrote a couple of D programs long ago and I still have a taste of an unfinished language. If I recall correctly, arrays manipulation is pretty weird and the language feels a bit the same PHP feels: a bunch of different things pieced together with no coherence. Compared to Python or Go, it's a whole different world.

Re: Go: Severe memory problems on 32bit Linux

#49

Post by agentS »

joe_the_user wrote: post #44
smanek wrote: post #19

Earlier quoted context omitted.

Yep, that's basically it. It sounds crazy (and it is!) - but it often works reasonably well in practice. SBCL (one of the most performant Common Lisp implementations) has an 'imprecise gc' that works the same way - and I've seen reasonably heavily stressed processes with uptimes in the weeks/months. Remember, even a few years ago (before fastthread, etc) a reasonably loaded Ruby on Rails app couldn't stay up for more…

But Ruby is more or less a "rapid development" environment. You pay for quick development turn-around with machines. Go is not in that kind of space as far as I know since, as I understand it, Go is sold as low-ish level language competing with C++ and C for system development. For something that would replace those languages, unexplained, systematic leaks would be bad (I mean, my C++ programs might leak but it's rea…

From experience, development in Go can definitely be classified as rapid. The type system stays out of your way, and compile times are insignificant. Try compiling on play.golang.org, or tour.golang.org to see what I mean.

About this issue: yeah, its unfortunate, but its a property of the class of garbage collector that Go uses atm. My servers are all 64-bit, so doesn't really affect me. But I do feel bad for those who are trying to run Go on ARM, or on 32-bit servers.

Re: Go: Severe memory problems on 32bit Linux

#50

Post by agentS »

jedbrown wrote: post #20
krakensden wrote: post #16

Earlier quoted context omitted.

The heap AND the data section. Although apparently, gcc-go doesn't do that for the data section.

That would seem to also present a DOS vector (even on 64 bit) if a user can get the program to store data (of any type, e.g. char or floating point) that happen to be binary-equivalent to pointers to large allocations.

This is realistically not an issue. First, on a 64 bit machine, the range of actually mapped addresses is small relative to all the possible values that can fit into 64 bits. Second, from an attacker's perspective, the values corresponding to mapped memory are extremely difficult to predict, and the values binary-equivalent to large allocations are impossible to predict, even with access to the source.

If you think you can still do it, all of *.golang.org and golang.org are running Go on Appengine, with the source code being freely available. This is your opportunity to get a back door into Google's servers.

Post reply on HN