Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go: Severe memory problems on 32bit Linux

groups.google.com

Post reply on HN
Print view View original

91–100 of 159 posts

Re: Go: Severe memory problems on 32bit Linux

#91

Post by microtherion »

4ad wrote: post #82
jsnell wrote: post #64

Earlier quoted context omitted.

The SBCL garbage collector doesn't work like described in the grandparent comment. It's precise for the heap, and only conservative for the registers and stacks. I can't say for sure how the Go GC works, but I would assume it likewise isn't fully conservative. E.g. the "avoid struct types with both integer and pointer fields" advice would be pointless for a fully conservative GC, but does make sense if structs contai…

I don't know anything about SBCL, but in Go the GC is used only for the heap, data allocated on the stack disappears as soon as the function returns, and registers have nothing to do with any of it.

Both data on the stack and registers may potentially contain the sole live pointer to a particular heap object, so unless you only run GC from some sort of main event loop, it's generally necessary to treat the stack and registers as potential roots as well.

Re: Go: Severe memory problems on 32bit Linux

#92

Post by [deleted] »

derleth wrote: post #41
pcwalton wrote: post #31

Earlier quoted context omitted.

Almost — C# has a cool feature whereby you can do pointer arithmetic if you pin the objects in question first, so the GC won't collect or move them. (The language statically enforces this by forbidding you from taking the address of a value until you pin it.)

That's cool; I think another way of doing it is to allow programs to request a sandbox where the GC doesn't go so they can do all of the pointer-arithmeticking they want as long as all their pointers fall within the sandbox by the time they're written through or dereferenced.

[deleted]

Re: Go: Severe memory problems on 32bit Linux

#93

Post by stiff »

dchest wrote: post #75
stiff wrote: post #72

Earlier quoted context omitted.

Well, since the people behind Go did not up front say anything about the language/compiler being specifically targeted at 64-bit platforms, I would expect someone more mature from the Go team to step up and say something like "We are sorry, we didn't foresee the consequences of some design decisions and hence screwed it up." and either "We will fix it ASAP" or "We cannot fix it because XYZ". They might not take money…

There is an "official" response: the post references a bug # where Russ Cox said "the rest of the issue will have to wait until after Go 1". Note that Go 1 is a "language freeze", not the implementation freeze. Now that it's known there's a bug that will be worked on later, people proposed possible workarounds, the easiest of which is to switch to 64-bit platform. I agree that downplaying the issue is wrong, but only…

The very next sentence in this response from Russ Cox is: "Or maybe all the 32-bit systems will be replaced by 64-bit ones.". So, they do not admit this is a serious problem that needs attention, it is not clear whether they will fix it, when they are going to fix and whether anyone cares about fixing it at all.

I couldn't find the adequate words to express this so far, but the reason I at all find this situation worth commenting on is that is reassembles to me a very common pattern of denial I observe among many professionals in various professions in cases where a problem appears that is very hard to tackle or even to analyse in the first place. Often a doctor who has troubles identifying a disease will tell you it's probably just something in your head, a programmer who has trouble reproducing a difficult bug will tell you it's you who probably did something wrong at some time, even a guy who I called to repair my washing machine that was stopping the washing at random told me to "keep it under observation" when he wasn't able to tell what's wrong. Many people simply do not want to put in the work needed to solve an unexpected and difficult problem, and thus, perhaps even subconsciously, try to handle it by pretending it doesn't exist. If you want to be a real professional and a leader in what you do, you can not behave like that, you can not repress a problem when someone reports one to you, you have to have the patience to examine the issue, the experience necessary to know when you can be certain that you have the complete picture of it, then sometimes the courage to admit there really is a problem and then finally you have to solve it, or people will not respect you.

Re: Go: Severe memory problems on 32bit Linux

#94

Post by Aissen »

derleth wrote: post #40
luser001 wrote: post #7

Earlier quoted context omitted.

I would run a 32-bit OS (even if the hardware supported 64-bit) if I wanted to squeeze every last bit of usable memory out of a VPS.

This sounds like precisely the use-case for x32: Running userland programs that only use 32-bit pointers on x86-64 bit hardware in 64-bit mode; you only get to address 4GB of RAM, but your pointers fit two to a register and you have all the x86-64 registers, opcodes, and special hardware. Sadly, it isn't here yet. https://sites.google.com/site/x32abi/ http://en.wikipedia.org/wiki/X32_ABI (To forestall the obvious obj…

x32 has been accepted(and merged) for Linux 3.4. Kernel support is here, now it's userspace's turn.

Re: Go: Severe memory problems on 32bit Linux

#95

Post by [deleted] »

4ad wrote: post #84
0xABADC0DA wrote: post #27

It's amazing how nonchalant they are about this 'oh sure we'll fix it in a year or so' when the garbage collector is basically all there needs to be in a Google Go runtime Some comments were asking why early Java's GC wasn't this bad even though it was conservative. The reason is that Java can't take references to fields of an object, so the data mistaken as a pointer has to actually point to an object header. In Goo…

It's not Google Go, it's simply Go, there's not a single Google reference on the web page and that's intentional. Looking at your previous posts here and on Reddit I see you are a notorious anti-Google troll that adds "Google" to "Go" so that your negative posts will be associated with both Google and Go. It's absurd to call it Google Go when there's a first class GNU implementation that has been in development from…

[deleted]

Re: Go: Severe memory problems on 32bit Linux

#96

Post by rubashov »

moilolita wrote: post #62
nakkiel wrote: post #48

Earlier quoted context omitted.

I wrote a couple of D programs long ago and I still have a taste of an unfinished language. If I recall correctly, arrays manipulation is pretty weird and the language feels a bit the same PHP feels: a bunch of different things pieced together with no coherence. Compared to Python or Go, it's a whole different world.

The Phobos standard library is quite unfinished indeed and poorly documented in some areas but the D2 language is rather complete. It has sh*tloads of features but this also makes it a bit harder to master. Go is lighter (very easy to take on) and has an impressive library. Long story short: with D2 I needed ~1 month to get a good grasp of the language and the std library (the library was the hardest part) while with…

> The Phobos standard library is quite unfinished indeed

How so? I think it's got pretty much what belongs in the standard library.

Re: Go: Severe memory problems on 32bit Linux

#98

Post by 4ad »

blinkingled wrote: post #83

I suspect the issue exists on 64-bit platforms as well, it's just that it doesn't impact as easily. In theory it is possible/common to have 64-bit machine with less than a ton of physical memory (VPS) and running 64-bit Go program which triggers this bug would result in similar impact as the 32-bit version. Why yes - Issue #909 essentially confirms this - running 64-bit doesn't fundamentally change anything - it just…

Physical memory does not have anything to do with it, the virtual address space is all that matters.

It might be useful to consider that 2^64 is 2^32 * 2^32. This means the problem becomes important on 64 bit only if an application will use 10 orders of magnitude more memory. By considering the historic growth in memory capacity and usage, this will only happen around 2060.

Re: Go: Severe memory problems on 32bit Linux

#99

Post by mkup »

smanek wrote: post #19
jedbrown wrote: post #14

However, it does not address the base issue, which is that Go uses a conservative garbage collector, and more values look like pointers in a 32-bit world. The only real fix would be to improve the garbage collector's understanding of which values are pointers and which are something else (e.g., floating point numbers that happen to look like pointers). And that is not an easy fix. How does this GC work? Is it literal…

Yep, that's basically it. It sounds crazy (and it is!) - but it often works reasonably well in practice. SBCL (one of the most performant Common Lisp implementations) has an 'imprecise gc' that works the same way - and I've seen reasonably heavily stressed processes with uptimes in the weeks/months. Remember, even a few years ago (before fastthread, etc) a reasonably loaded Ruby on Rails app couldn't stay up for more…

Probably Golang developers/implementers should change their compiler to insert "magic" constant before each 32-bit pointer in the GC memory, so all such pointers could be quickly discovered during full linear scan of the heap during GC, regardless of target address.

This "magic" constant should be non-valid value in 32-bit single precision float point format, far away from usual mmap()'ped address ranges, and far away from small integers.

Maybe this "magic" value should be randomized to prevent DoS attacks on Go runtime library.

Re: Go: Severe memory problems on 32bit Linux

#100

Post by thrownaway2424 »

oskarth wrote: post #73
batista wrote: post #67

Earlier quoted context omitted.

>I know from experience that the gc 64-bit version is production ready. Is it really production ready though, or is it the same half-arsed implementation as the 32-bit one, just taking advantage of the larger availability of virtual memory on a 64 bit system?

If Google is, and has been for quite some time, using it for large-scale systems in production, I would call it production ready.

Is there any reason to believe that Google uses Go for large-scale production systems?
Post reply on HN