Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go: Severe memory problems on 32bit Linux

groups.google.com

Post reply on HN
Print view View original

131–140 of 159 posts

Re: Go: Severe memory problems on 32bit Linux

#131

Post by sixbrx »

4ad wrote: post #108
bad_user wrote: post #88

Earlier quoted context omitted.

You know, that kind of sucks ... there are many machines around that are still on 32 bits. I keep my own Ubuntu laptop (dual-boots to Windows) on 32-bit builds (both Linux and Windows), simply because I have less problems that way (mostly with hardware drivers, but also with software). I keep the Amazon EC2 instances I maintain on 32-bit images, simply because they are cheaper. My Android phone is also 32-bit and wil…

I've been exclusively using 64 bit computers and operating systems, both Windows and Linux, for about 7 years now, never had the reported driver problems, well, never had any issue, really. For me at least, 32 bit is only important for ARM. On the other hand the issue is very much blown out of proportion, most people haven't seen it, even if they run 32 bit servers. Most usual servers written in Go, like web servers,…

I do wish the desktop world would just move wholesale to 64 bit, but do note that Ubuntu still marks the 32 bit install the "recommended" one, so I would guess there must be some sort of problems that are lingering in the 64 bit versions. Maybe just flash support or something like that?

Re: Go: Severe memory problems on 32bit Linux

#132

Post by ianlancetaylor »

papaf wrote: post #56
_delirium wrote: post #3

This is the relevant bug: http://code.google.com/p/go/issues/detail?id=909 As the discussion there and in the thread says, the root problem is that Go uses a conservative garbage collector, and on 32-bit a lot more values look like pointers than on 64-bit, so many more things don't get freed in long-running processes. Seems not to be easy to fix.

Can anyone knowledgeable about compilers/GCs say why GO went with a conservative garbage collector? From any GO source code its trivial to pick out pointers from values. Is this information hard to preserve at runtime?

A conservative GC is easier to implement. No other reason.

Re: Go: Severe memory problems on 32bit Linux

#133

Post by ianlancetaylor »

rand_r wrote: post #26
smanek wrote: post #19

Earlier quoted context omitted.

Yep, that's basically it. It sounds crazy (and it is!) - but it often works reasonably well in practice. SBCL (one of the most performant Common Lisp implementations) has an 'imprecise gc' that works the same way - and I've seen reasonably heavily stressed processes with uptimes in the weeks/months. Remember, even a few years ago (before fastthread, etc) a reasonably loaded Ruby on Rails app couldn't stay up for more…

Would that gc scheme potentially cause the following (pseudo) code to break? x = malloc(1); // allocates block 'a' in memory int i = (int) x; x = 0; i = i - 1; // gc runs here and frees 'a' *( (int*)(i + 1) ) = 123; // failure

When Go gets a precise collector, simple implementations of this will work. Doing this kind of thing in Go requires importing the "unsafe" package, and any memory allocations done by code importing "unsafe" could be marked as possibly a pointer.

However, it would probably be possible to write code involving two packages, one of which does not import "unsafe", to lead to dangling pointers and eventual crashes. That is why you should be careful about code that imports "unsafe".

Re: Go: Severe memory problems on 32bit Linux

#134

Post by ianlancetaylor »

jedbrown wrote: post #20
krakensden wrote: post #16

Earlier quoted context omitted.

The heap AND the data section. Although apparently, gcc-go doesn't do that for the data section.

That would seem to also present a DOS vector (even on 64 bit) if a user can get the program to store data (of any type, e.g. char or floating point) that happen to be binary-equivalent to pointers to large allocations.

You need more than that. You need a server that regularly allocates large amounts of memory and then leaves them unreferenced so that the garbage collector can collect them. Then you also need the program to store data that you control, and to also keep references to that data--after all, if that data is collected, then the faux-pointers no longer pin the other allocations. Overall this does not sound like a common allocation pattern for servers.

Re: Go: Severe memory problems on 32bit Linux

#135

Post by ezyang »

pcwalton wrote: post #129
ezyang wrote: post #127

Earlier quoted context omitted.

I know how to do precise GC if you allow me to add a (pointer-size) header to all data living in the heap; i.e. to maintain the RTTI. I don't know how you do that if you're not allowed a header. Do C# and D have headers?

I'm sure they do. There are a few things to note here: (1) In order for malloc to work, you need a header anyway (at least, unless your allocation fits in one of the fixed-size bins). (2) You can get around the header to some extent by sorting the fields of your objects so that pointers come first, and then all you need to do is to store the number of pointers (or a sentinel value). This is what Haskell does. Of cour…

So, the thing that always gets Haskell folks when dealing with an implementation (2) is that you can't get uniform data representation when dealing with things like arrays. It means you have to unbox things. Arguably, the situation is not much better in malloc land; if you malloc a large multiple of your object size, you're explicitly saying, "I want this to be unboxed", but by this point you've wandered into generics land.

(3) is annoying. Who likes 31-bit integers? Not I!

Re: Go: Severe memory problems on 32bit Linux

#136

Post by he_the_great »

moilolita wrote: post #69
el_muchacho wrote: post #63

Earlier quoted context omitted.

D is pretty strong at concurrency today. See for instance http://www.informit.com/articles/article.aspx?p=1609144 and http://ddili.org/ders/d.en/parallelism.html

It's strange that with all those threading examples, they didn't notice the need for a WaitGroup primitive. I know that you could implement it yourself. You could simulate the goroutines, implement channels and SCGI/FCGI and so on, but why bother when there is Go ?

std.concurrency "is a low-level messaging API." The language provides low-level concurrency primitives. It is possible a higher level library providing "goroutines" might be made, though there is not effort or plan.

Re: Go: Severe memory problems on 32bit Linux

#137

Post by he_the_great »

moilolita wrote: post #47
eternalban wrote: post #15

Earlier quoted context omitted.

I also looked at D/Go -- I predict the reign of JVM is over (quote me). I wouldn't bother with the 32 bit hiccups. Go hits the sweet spot pretty well. You have chosen well. Hang in there.

If Ceylon/Rust are not ready today it means they will be immature for at least 2 years from their release date. Without a large community behind them, they will slowly fade away like D did.In order to get community, they have to have a good/clean/pragmatic design and a library as comprehensive as possible. If they don't have a good library they better have excellent interop with other platforms or they will be doomed…

> Without a large community behind them, they will slowly fade away like D did.

This is a rather strange assessment. The history of "successful" languages has been a mixture of "cool jump onboard" and "who can stay alive the longest to get a community." D is in the latter camp.

It seems to me, unreasonable to expect a language to be coming out the gate with guns blazing. People expect Nukes now!

Re: Go: Severe memory problems on 32bit Linux

#138

Post by pcwalton »

ianlancetaylor wrote: post #133
rand_r wrote: post #26

Earlier quoted context omitted.

Would that gc scheme potentially cause the following (pseudo) code to break? x = malloc(1); // allocates block 'a' in memory int i = (int) x; x = 0; i = i - 1; // gc runs here and frees 'a' *( (int*)(i + 1) ) = 123; // failure

When Go gets a precise collector, simple implementations of this will work. Doing this kind of thing in Go requires importing the "unsafe" package, and any memory allocations done by code importing "unsafe" could be marked as possibly a pointer. However, it would probably be possible to write code involving two packages, one of which does not import "unsafe", to lead to dangling pointers and eventual crashes. That is…

Actually, the code sample in the grandparent comment obfuscates the memory address by subtracting 1. Even a conservative GC will be confused in this case...

Re: Go: Severe memory problems on 32bit Linux

#139

Post by pcwalton »

ezyang wrote: post #135
pcwalton wrote: post #129

Earlier quoted context omitted.

I'm sure they do. There are a few things to note here: (1) In order for malloc to work, you need a header anyway (at least, unless your allocation fits in one of the fixed-size bins). (2) You can get around the header to some extent by sorting the fields of your objects so that pointers come first, and then all you need to do is to store the number of pointers (or a sentinel value). This is what Haskell does. Of cour…

So, the thing that always gets Haskell folks when dealing with an implementation (2) is that you can't get uniform data representation when dealing with things like arrays. It means you have to unbox things. Arguably, the situation is not much better in malloc land; if you malloc a large multiple of your object size, you're explicitly saying, "I want this to be unboxed", but by this point you've wandered into generic…

Yeah, I hate 31-bit integers too. It's not the only tagging scheme though; I prefer NaN boxing (used in SpiderMonkey among others). NaN boxing allows unboxed doubles and 32-bit ints, at the cost of increased register pressure and memory usage on 32-bit systems.

Re: Go: Severe memory problems on 32bit Linux

#140

Post by ianlancetaylor »

pcwalton wrote: post #138
ianlancetaylor wrote: post #133

Earlier quoted context omitted.

When Go gets a precise collector, simple implementations of this will work. Doing this kind of thing in Go requires importing the "unsafe" package, and any memory allocations done by code importing "unsafe" could be marked as possibly a pointer. However, it would probably be possible to write code involving two packages, one of which does not import "unsafe", to lead to dangling pointers and eventual crashes. That is…

Actually, the code sample in the grandparent comment obfuscates the memory address by subtracting 1. Even a conservative GC will be confused in this case...

Ah, yes, missed that. Nothing a GC can do about code like that. At least it remains true that this can only happen in Go if you explicitly import "unsafe".
Post reply on HN