Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go: Severe memory problems on 32bit Linux

groups.google.com

Post reply on HN
Print view View original

31–40 of 159 posts

Re: Go: Severe memory problems on 32bit Linux

#31

Post by pcwalton »

smanek wrote: post #30
rand_r wrote: post #26

Earlier quoted context omitted.

Would that gc scheme potentially cause the following (pseudo) code to break? x = malloc(1); // allocates block 'a' in memory int i = (int) x; x = 0; i = i - 1; // gc runs here and frees 'a' *( (int*)(i + 1) ) = 123; // failure

I don't think any GC scheme could work with that. Languages that allow pointer arithmetic like that basically can't be GC'd. The problem with an imprecise GC is that having an integer that looks like a pointer, could prevent an object from being freed.

Almost — C# has a cool feature whereby you can do pointer arithmetic if you pin the objects in question first, so the GC won't collect or move them. (The language statically enforces this by forbidding you from taking the address of a value until you pin it.)

Re: Go: Severe memory problems on 32bit Linux

#32

Post by [deleted] »

rand_r wrote: post #26
smanek wrote: post #19

Earlier quoted context omitted.

Yep, that's basically it. It sounds crazy (and it is!) - but it often works reasonably well in practice. SBCL (one of the most performant Common Lisp implementations) has an 'imprecise gc' that works the same way - and I've seen reasonably heavily stressed processes with uptimes in the weeks/months. Remember, even a few years ago (before fastthread, etc) a reasonably loaded Ruby on Rails app couldn't stay up for more…

Would that gc scheme potentially cause the following (pseudo) code to break? x = malloc(1); // allocates block 'a' in memory int i = (int) x; x = 0; i = i - 1; // gc runs here and frees 'a' *( (int*)(i + 1) ) = 123; // failure

[deleted]

Re: Go: Severe memory problems on 32bit Linux

#33

Post by eternalban »

eternalban wrote: post #18
_delirium wrote: post #10

Earlier quoted context omitted.

I don't think this one is lacking interest, but an obvious fix. It looks like Russ Cox did commit a fix for some portion of the cases. But a better fix would either need to significantly modify the way things are laid out to reduce the odds of false positives, or else move to a precise GC. (Precise GCs are possible in C-like languages, but trickier to implement. Here's a recent paper on one: http://www.cs.utah.edu/~r…

Von Neumann architecture falsely lead us to the path with the singular notion of "memory". We need to distinguish at the hardware architecture level between "working memory" and "persistent memory". I doodle all the time when working. Pristine set of diagrams emerges from the chaos. I wish my computer would play along with this regime ..

I haven't mentioned downvotes to date, but the down votes on this is puzzling. Would the down voters care to comment as to why the comment is deemed down vote worthy? (I would think those who work/grok on memory managers do get the point.)

Re: Go: Severe memory problems on 32bit Linux

#34

Post by [deleted] »

0xABADC0DA wrote: post #27

It's amazing how nonchalant they are about this 'oh sure we'll fix it in a year or so' when the garbage collector is basically all there needs to be in a Google Go runtime Some comments were asking why early Java's GC wasn't this bad even though it was conservative. The reason is that Java can't take references to fields of an object, so the data mistaken as a pointer has to actually point to an object header. In Goo…

[deleted]

Re: Go: Severe memory problems on 32bit Linux

#35

Post by flomo »

smanek wrote: post #29
eblume wrote: post #23

Earlier quoted context omitted.

That's really interesting - is there a place where I can read more about these early Rails performance issues?

There was some back-and-forth between Zed Shaw and DHH that I remember being interested by at the time. Zed deleted all his posts - but here are some places to get started: - Shaw's opening volley: http://web.archive.org/web/20080103072111/http://www.zedshaw... [lots of stupid personal flames - but his technical points are consistent with what I remember from the time] - DHH's response: http://david.heinemeierhansson…

I certainly wasn't anywhere in the loop, but at least on my site, FastCGI didn't work, and Mongrel did.

Zed's rant/flame goes into so much personal detail that he doesn't really articulate the point. Rails was receiving this enormous amount of hype, but there wasn't even a working application server yet.

Re: Go: Severe memory problems on 32bit Linux

#36

Post by luriel »

gccgo is better on 32bit systems: https://groups.google.com/d/msg/golang-nuts/qxlxu5RZAl0/NS71...

Note that work on the garbage collector is ongoing post-Go1, a faster parallel GC is in process being merged: http://codereview.appspot.com/5279048/

But ultimately 32bit systems don't seem to be a big issue for Google or anyone else using Go in production (and there are quite a few big organizations using it: http://go-lang.cat-v.org/organizations-using-go ).

Most people moved to 64bits a while ago so the amount of attention the 32bit port gets will never be the same.

Re: Go: Severe memory problems on 32bit Linux

#37

Post by riobard »

sunyc wrote: post #4

to go back to 32 bit hardware, you will be running very old machines that you can buy off ebay for <200$ and the power cost offset the new hardware cost in several months.

This is what I thought. Go is primarily targeted at server-side applications, which are almost exclusively running on 64bit hardware/OS. Memory is so cheap now that it's silly not to have more than 16GB plugged in the chasis.

Then I realized there are many small VPS/EC2 instances with <1GB memory (I have two right now running on 64bit) and there are people who try to squeeze every last bit out of them by going 32bit...

Re: Go: Severe memory problems on 32bit Linux

#38

Post by silon3 »

eternalban wrote: post #15
moilolita wrote: post #12

Earlier quoted context omitted.

Unfortunately you are right. For me this is a very unpleasant surprise after I put aside D and rewrote my little framework in Go. Everybody said that the gc is not final, that there are some performance issues and they are working on it but I never imagined that such catastrophic bugs are not solved by now.

I also looked at D/Go -- I predict the reign of JVM is over (quote me). I wouldn't bother with the 32 bit hiccups. Go hits the sweet spot pretty well. You have chosen well. Hang in there.

I strongly prefer ceylon/rust... But they're not ready yet.

Re: Go: Severe memory problems on 32bit Linux

#39

Post by PPGualtieri »

_delirium wrote: post #3

This is the relevant bug: http://code.google.com/p/go/issues/detail?id=909 As the discussion there and in the thread says, the root problem is that Go uses a conservative garbage collector, and on 32-bit a lot more values look like pointers than on 64-bit, so many more things don't get freed in long-running processes. Seems not to be easy to fix.

Isn't this issue also going to be a problem with Linux upcoming X32 ABI?

Re: Go: Severe memory problems on 32bit Linux

#40

Post by derleth »

luser001 wrote: post #7
sunyc wrote: post #4

to go back to 32 bit hardware, you will be running very old machines that you can buy off ebay for <200$ and the power cost offset the new hardware cost in several months.

I would run a 32-bit OS (even if the hardware supported 64-bit) if I wanted to squeeze every last bit of usable memory out of a VPS.

This sounds like precisely the use-case for x32: Running userland programs that only use 32-bit pointers on x86-64 bit hardware in 64-bit mode; you only get to address 4GB of RAM, but your pointers fit two to a register and you have all the x86-64 registers, opcodes, and special hardware.

Sadly, it isn't here yet.

https://sites.google.com/site/x32abi/

http://en.wikipedia.org/wiki/X32_ABI

(To forestall the obvious objection: No, there's no way Torvalds will allow this to re-introduce the 2038 Problem into a new ABI. None.)

Post reply on HN