Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go: Severe memory problems on 32bit Linux

groups.google.com

Post reply on HN
Print view View original

121–130 of 159 posts

Re: Go: Severe memory problems on 32bit Linux

#121

Post by flomo »

fleitz wrote: post #74
flomo wrote: post #35

Earlier quoted context omitted.

I certainly wasn't anywhere in the loop, but at least on my site, FastCGI didn't work, and Mongrel did. Zed's rant/flame goes into so much personal detail that he doesn't really articulate the point. Rails was receiving this enormous amount of hype, but there wasn't even a working application server yet.

No one cares how the sausage is made. They care that they have sausage. I write code to make money, not for uptime competitions. If something that makes money needs to be restarted every 4 minutes and customers are willing to pay for it why should I give a shit?

FCGI problems were very customer-visible. Prior to Zed coming along, the issue wasn't being discussed much anywhere, making it unclear if it was your code, ruby issue, framework issue, or appserver. Now it is your problem to fix!

Re: Go: Severe memory problems on 32bit Linux

#122

Post by markatto »

rand_r wrote: post #118
4ad wrote: post #84

Earlier quoted context omitted.

It's not Google Go, it's simply Go, there's not a single Google reference on the web page and that's intentional. Looking at your previous posts here and on Reddit I see you are a notorious anti-Google troll that adds "Google" to "Go" so that your negative posts will be associated with both Google and Go. It's absurd to call it Google Go when there's a first class GNU implementation that has been in development from…

Any recommendation on how we should google for Go related pages?

'golang' seems to be what people are using.

Re: Go: Severe memory problems on 32bit Linux

#123

Post by Arelius »

ArbitraryLimits wrote: post #111
ghusbands wrote: post #87

Earlier quoted context omitted.

On 32-bit systems, the problem is that there's a very high chance of non-pointer data looking like pointers to existing data, which, in turn might have its own pointer-like data. This means that you can end up keeping an excessive amount of unreferenced data. However, a 64-bit address space is so much larger that you can't really suffer the same issue. Unlike on 32-bit systems, neither high entropy data nor text will…

Can someone help me out here with a TL;DR? This is my first exposure ever to any technical aspect of Go, and this discussion makes it sound from this discussion like it doesn't use any tag bits in its pointers, but has its garbage collector run heuristics on the data it examines to see whether it "looks like text" to decide whether to collect it? I know that can't possibly be correct.

It's much simpler than that, it's called a conservative collector, It looks at every bit of data, at just pretends it is a pointer, if anything points to a valid allocated address, that address is retained. Otherwise, just like every collection when there are no references, the object is collected.

Re: Go: Severe memory problems on 32bit Linux

#124

Post by aflott »

rand_r wrote: post #118
4ad wrote: post #84

Earlier quoted context omitted.

It's not Google Go, it's simply Go, there's not a single Google reference on the web page and that's intentional. Looking at your previous posts here and on Reddit I see you are a notorious anti-Google troll that adds "Google" to "Go" so that your negative posts will be associated with both Google and Go. It's absurd to call it Google Go when there's a first class GNU implementation that has been in development from…

Any recommendation on how we should google for Go related pages?

go lang works quite well for me

Re: Go: Severe memory problems on 32bit Linux

#125

Post by [deleted] »

rand_r wrote: post #118
4ad wrote: post #84

Earlier quoted context omitted.

It's not Google Go, it's simply Go, there's not a single Google reference on the web page and that's intentional. Looking at your previous posts here and on Reddit I see you are a notorious anti-Google troll that adds "Google" to "Go" so that your negative posts will be associated with both Google and Go. It's absurd to call it Google Go when there's a first class GNU implementation that has been in development from…

Any recommendation on how we should google for Go related pages?

[deleted]

Re: Go: Severe memory problems on 32bit Linux

#126

Post by r00tbeer »

kevinpet wrote: post #25
r00tbeer wrote: post #21

Earlier quoted context omitted.

I know early JVM garabage collectors were also "conservative", but I don't recall JVMs running into these sorts of problems. Maybe folks are just using more of the 4G address space these days than back in the day of conservative GC JVMs?

I've never heard of a JVM that uses this kind of GC. As far as I know, every Sun JVM has used explicit "this is a pointer" to identify things for GC.

Ah, you're right. Its only the thread stacks that were conservatively scanned, not all of memory (see "Mostly Accurate Stack Scanning" for some indirect evidence of such). Its not clear to me if modern JVMs are still conservatively scanning stacks or not. But clearly its much less of a problem than conservatively scanning all of memory.

Re: Go: Severe memory problems on 32bit Linux

#127

Post by ezyang »

pcwalton wrote: post #117
ezyang wrote: post #114

Funnily enough, I understood why they went the conservative GC route. It has to do with the overall Go philosophy, which is that they really do not want features to affect data representation. This has meant no boxing (and no easy polymorphism), and a decision like that has logical consequences for GC too. Here's to hoping they find a cool solution! It's been a problem for GC's since forever, and if they find a gener…

In Rust we're working on a solution for this problem. Essentially, the plan is to have RTTI on garbage-collected data (this is already completed and is used in the cycle collector) and precise stack information for every root on the stack. The latter is in a fork of LLVM: https://github.com/pcwalton/llvm But aside from the shameless plug, C# and D (I believe), have had precise garbage collectors for quite a while now…

I know how to do precise GC if you allow me to add a (pointer-size) header to all data living in the heap; i.e. to maintain the RTTI. I don't know how you do that if you're not allowed a header. Do C# and D have headers?

Re: Go: Severe memory problems on 32bit Linux

#128

Post by pwf »

ghusbands wrote: post #87
batista wrote: post #67

Earlier quoted context omitted.

>I know from experience that the gc 64-bit version is production ready. Is it really production ready though, or is it the same half-arsed implementation as the 32-bit one, just taking advantage of the larger availability of virtual memory on a 64 bit system?

On 32-bit systems, the problem is that there's a very high chance of non-pointer data looking like pointers to existing data, which, in turn might have its own pointer-like data. This means that you can end up keeping an excessive amount of unreferenced data. However, a 64-bit address space is so much larger that you can't really suffer the same issue. Unlike on 32-bit systems, neither high entropy data nor text will…

Is it that you can't suffer the same issue, or that you can't 'really' suffer the same issue?

Even if there's a smaller chance of it happening, any language that has ANY chance of killing your system when running as expected is a language I'll never bother to learn.

Is there any sort of analysis tool they might be able to put into the compiler to tell you if a data structure you created has a high chance of looking like a pointer?

Re: Go: Severe memory problems on 32bit Linux

#129

Post by pcwalton »

ezyang wrote: post #127
pcwalton wrote: post #117

Earlier quoted context omitted.

In Rust we're working on a solution for this problem. Essentially, the plan is to have RTTI on garbage-collected data (this is already completed and is used in the cycle collector) and precise stack information for every root on the stack. The latter is in a fork of LLVM: https://github.com/pcwalton/llvm But aside from the shameless plug, C# and D (I believe), have had precise garbage collectors for quite a while now…

I know how to do precise GC if you allow me to add a (pointer-size) header to all data living in the heap; i.e. to maintain the RTTI. I don't know how you do that if you're not allowed a header. Do C# and D have headers?

I'm sure they do. There are a few things to note here:

(1) In order for malloc to work, you need a header anyway (at least, unless your allocation fits in one of the fixed-size bins).

(2) You can get around the header to some extent by sorting the fields of your objects so that pointers come first, and then all you need to do is to store the number of pointers (or a sentinel value). This is what Haskell does. Of course, this prevents low-level control over data representation.

(3) You can tag (or NaN box) all your values. This is what most MLs do, as well as JS, many Lisps, etc.

(4) You can use a map on the side from pointer to type info to avoid a header. This is what Rust in its early days did. It's worse than a header for memory consumption though, so it doesn't really buy much.

Re: Go: Severe memory problems on 32bit Linux

#130

Post by cpeterso »

PPGualtieri wrote: post #39
_delirium wrote: post #3

This is the relevant bug: http://code.google.com/p/go/issues/detail?id=909 As the discussion there and in the thread says, the root problem is that Go uses a conservative garbage collector, and on 32-bit a lot more values look like pointers than on 64-bit, so many more things don't get freed in long-running processes. Seems not to be easy to fix.

Isn't this issue also going to be a problem with Linux upcoming X32 ABI?

Yes, because x32 ABI's pointers are still 32 bits, so pointer values are in a small enough range that they can fool the GC into thinking they are legitimate int values.
Post reply on HN