Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go: Severe memory problems on 32bit Linux

groups.google.com

Post reply on HN
Print view View original

101–110 of 159 posts

Re: Go: Severe memory problems on 32bit Linux

#101

Post by bhurt »

papaf wrote: post #56
_delirium wrote: post #3

This is the relevant bug: http://code.google.com/p/go/issues/detail?id=909 As the discussion there and in the thread says, the root problem is that Go uses a conservative garbage collector, and on 32-bit a lot more values look like pointers than on 64-bit, so many more things don't get freed in long-running processes. Seems not to be easy to fix.

Can anyone knowledgeable about compilers/GCs say why GO went with a conservative garbage collector? From any GO source code its trivial to pick out pointers from values. Is this information hard to preserve at runtime?

Probably ease of implementation. You can just grab the Boehm conservative GC, slap it in, and bang- you've got GC. You can even add GC this way to C/C++ programs.

Unfortunately, Boehm has draw backs- because it's getting no help from the compiler, it can't tell integers from pointers. So it has to treat everything that looks like it might be a pointer as a pointer, even if it's an integer (or floating point number). Which means that it's possible for garbage to not be collected, because there is an integer that happens to have the same value as the address of the garbage object. And, of course, once you can't collect that object, you can't collect all the objects it refers to (including false pointers), and so on.

The odds of this happening are a function of what percentage of the virtual address space is in use- once some critical threshold is reached, the amount of garbage that can't be collected due to false pointers just explodes. On 32-bit platforms, I've seen this happen with heap sizes of only a few hundred megabytes. And the advice to work around this is exactly what the responder said- use less memory, don't use large ints (which are more likely to be mistaken for pointers), etc. Also, the problem goes away (for the time being) on 64 bits, because the percentage of memory used drops. A terabyte of memory on a 64-bit system is the same fraction of the total address space as a kilobyte of memory is on a 32-bit system.

Re: Go: Severe memory problems on 32bit Linux

#102

Post by _delirium »

riffraff wrote: post #81
papaf wrote: post #56

Earlier quoted context omitted.

Can anyone knowledgeable about compilers/GCs say why GO went with a conservative garbage collector? From any GO source code its trivial to pick out pointers from values. Is this information hard to preserve at runtime?

I recall reading something along the lines of it being hard to use a precise GC due to the the "unsafe" package, but I am not knowledgeable at all.

Here's an early LtU thread where someone predicted it'd probably have to use a conservative GC due to some of the addressing features: http://lambda-the-ultimate.org/node/3676#comment-52560

Re: Go: Severe memory problems on 32bit Linux

#103

Post by burgerbrain »

4ad wrote: post #84
0xABADC0DA wrote: post #27

It's amazing how nonchalant they are about this 'oh sure we'll fix it in a year or so' when the garbage collector is basically all there needs to be in a Google Go runtime Some comments were asking why early Java's GC wasn't this bad even though it was conservative. The reason is that Java can't take references to fields of an object, so the data mistaken as a pointer has to actually point to an object header. In Goo…

It's not Google Go, it's simply Go, there's not a single Google reference on the web page and that's intentional. Looking at your previous posts here and on Reddit I see you are a notorious anti-Google troll that adds "Google" to "Go" so that your negative posts will be associated with both Google and Go. It's absurd to call it Google Go when there's a first class GNU implementation that has been in development from…

The number of times people have needed to tell this guy this is staggering.

Re: Go: Severe memory problems on 32bit Linux

#104

Post by jerf »

ghusbands wrote: post #87
batista wrote: post #67

Earlier quoted context omitted.

>I know from experience that the gc 64-bit version is production ready. Is it really production ready though, or is it the same half-arsed implementation as the 32-bit one, just taking advantage of the larger availability of virtual memory on a 64 bit system?

On 32-bit systems, the problem is that there's a very high chance of non-pointer data looking like pointers to existing data, which, in turn might have its own pointer-like data. This means that you can end up keeping an excessive amount of unreferenced data. However, a 64-bit address space is so much larger that you can't really suffer the same issue. Unlike on 32-bit systems, neither high entropy data nor text will…

Hmmmmm... I'm smelling memory-based DoS if I, say, upload carefully crafted files to a webserver. What accidentally happens in 32-bit may be deliberately triggered in 64-bit.

Re: Go: Severe memory problems on 32bit Linux

#106

Post by talentdeficit »

el_muchacho wrote: post #80
zvrba wrote: post #79

Interesting piece of information below in the thread: in reply to "Go being advertised as a systems PL", David Symonds replies with: "It's not. It used to be, but it's not any more."

What is the definition of a system PL ?

deterministic memory usage is a good start

Re: Go: Severe memory problems on 32bit Linux

#107

Post by 4ad »

thrownaway2424 wrote: post #100
oskarth wrote: post #73

Earlier quoted context omitted.

If Google is, and has been for quite some time, using it for large-scale systems in production, I would call it production ready.

Is there any reason to believe that Google uses Go for large-scale production systems?

Yes, it's used by YouTube, in a very critical path, large scale enough :)?

https://groups.google.com/forum/#!msg/golang-nuts/U5ilNZpXzN...

http://code.google.com/p/vitess/

Re: Go: Severe memory problems on 32bit Linux

#108

Post by 4ad »

bad_user wrote: post #88
luriel wrote: post #36

gccgo is better on 32bit systems: https://groups.google.com/d/msg/golang-nuts/qxlxu5RZAl0/NS71... Note that work on the garbage collector is ongoing post-Go1, a faster parallel GC is in process being merged: http://codereview.appspot.com/5279048/ But ultimately 32bit systems don't seem to be a big issue for Google or anyone else using Go in production (and there are quite a few big organizations using it: http://go-l…

You know, that kind of sucks ... there are many machines around that are still on 32 bits. I keep my own Ubuntu laptop (dual-boots to Windows) on 32-bit builds (both Linux and Windows), simply because I have less problems that way (mostly with hardware drivers, but also with software). I keep the Amazon EC2 instances I maintain on 32-bit images, simply because they are cheaper. My Android phone is also 32-bit and wil…

I've been exclusively using 64 bit computers and operating systems, both Windows and Linux, for about 7 years now, never had the reported driver problems, well, never had any issue, really.

For me at least, 32 bit is only important for ARM. On the other hand the issue is very much blown out of proportion, most people haven't seen it, even if they run 32 bit servers. Most usual servers written in Go, like web servers, use very little memory. I process 4k requests per second using 7MB of resident memory. There are many memory intensive applications, but you usually don't run those on 32 bit.

Re: Go: Severe memory problems on 32bit Linux

#109

Post by blinkingled »

4ad wrote: post #98
blinkingled wrote: post #83

I suspect the issue exists on 64-bit platforms as well, it's just that it doesn't impact as easily. In theory it is possible/common to have 64-bit machine with less than a ton of physical memory (VPS) and running 64-bit Go program which triggers this bug would result in similar impact as the 32-bit version. Why yes - Issue #909 essentially confirms this - running 64-bit doesn't fundamentally change anything - it just…

Physical memory does not have anything to do with it, the virtual address space is all that matters. It might be useful to consider that 2^64 is 2^32 * 2^32. This means the problem becomes important on 64 bit only if an application will use 10 orders of magnitude more memory. By considering the historic growth in memory capacity and usage, this will only happen around 2060.

Check out http://code.google.com/p/go/issues/detail?id=909#c32 and the following comment that agrees with it.

This isn't just a address space leak - it is a real memory leak. On 64-bit the GC may not be so easily fooled as on 32-bit but it can still be fooled and that is a fundamental problem that will result in memory leaks - if I have 2GB RAM VPS - it doesn't help to have 2^64 bits of address space (actually it is more like 2^48 (http://en.wikipedia.org/wiki/X86-64#Virtual_address_space_de... ) - if the GC leaks memory sooner or later my process will be killed by the OS.

Re: Go: Severe memory problems on 32bit Linux

#110

Post by 4ad »

jedbrown wrote: post #85
agentS wrote: post #50

Earlier quoted context omitted.

This is realistically not an issue. First, on a 64 bit machine, the range of actually mapped addresses is small relative to all the possible values that can fit into 64 bits. Second, from an attacker's perspective, the values corresponding to mapped memory are extremely difficult to predict, and the values binary-equivalent to large allocations are impossible to predict, even with access to the source. If you think y…

If you make a huge allocation (many pages), isn't the Go runtime very likely to call malloc()? For large allocations, malloc() is going to get you a bunch of fresh pages and you will generally get the address of the start of a page. The offset of the pointer within the page is then likely to be deterministic, so you probably only need one unit of pointer-equivalent data per page. If you have enabled huge pages (e.g.…

The Go runtime does not call malloc(3) for heap, it reserves address space at known high locations (over 2^32) with mmap(2) using the MAP_FIXED flag, and it does so in 16GB increments (or is it just one 16GB allocation? can't remember).

I won't comment on the DOS concern until I've investigated further.

Post reply on HN