Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go: Severe memory problems on 32bit Linux

groups.google.com

Post reply on HN
Print view View original

61–70 of 159 posts

Re: Go: Severe memory problems on 32bit Linux

#61

Post by kevingadd »

eternalban wrote: post #33
eternalban wrote: post #18

Earlier quoted context omitted.

Von Neumann architecture falsely lead us to the path with the singular notion of "memory". We need to distinguish at the hardware architecture level between "working memory" and "persistent memory". I doodle all the time when working. Pristine set of diagrams emerges from the chaos. I wish my computer would play along with this regime ..

I haven't mentioned downvotes to date, but the down votes on this is puzzling. Would the down voters care to comment as to why the comment is deemed down vote worthy? (I would think those who work/grok on memory managers do get the point.)

I didn't downvote it, but it mostly read like rambling nonsense to me, so that might be why. Maybe write it so it's clear to people who don't work on memory managers all day?

Re: Go: Severe memory problems on 32bit Linux

#62

Post by moilolita »

nakkiel wrote: post #48
moilolita wrote: post #22

Earlier quoted context omitted.

D has a nice allocation management(i loved the manual+gc approach) and in my benchmarks, optimized D was slightly faster than Go at almost anything and consumed up to 70% less memory (I assumed that was because the Go gc kicked in later and was a bit lazy) but D was weak at threading/synchronization and the documentation of the standard lib was quite messy and lacking. So I decided that in the long run Go will be bet…

I wrote a couple of D programs long ago and I still have a taste of an unfinished language. If I recall correctly, arrays manipulation is pretty weird and the language feels a bit the same PHP feels: a bunch of different things pieced together with no coherence. Compared to Python or Go, it's a whole different world.

The Phobos standard library is quite unfinished indeed and poorly documented in some areas but the D2 language is rather complete. It has sh*tloads of features but this also makes it a bit harder to master. Go is lighter (very easy to take on) and has an impressive library. Long story short: with D2 I needed ~1 month to get a good grasp of the language and the std library (the library was the hardest part) while with Go I needed around a week.

Re: Go: Severe memory problems on 32bit Linux

#63

Post by el_muchacho »

moilolita wrote: post #22
eternalban wrote: post #15

Earlier quoted context omitted.

I also looked at D/Go -- I predict the reign of JVM is over (quote me). I wouldn't bother with the 32 bit hiccups. Go hits the sweet spot pretty well. You have chosen well. Hang in there.

D has a nice allocation management(i loved the manual+gc approach) and in my benchmarks, optimized D was slightly faster than Go at almost anything and consumed up to 70% less memory (I assumed that was because the Go gc kicked in later and was a bit lazy) but D was weak at threading/synchronization and the documentation of the standard lib was quite messy and lacking. So I decided that in the long run Go will be bet…

D is pretty strong at concurrency today. See for instance http://www.informit.com/articles/article.aspx?p=1609144 and http://ddili.org/ders/d.en/parallelism.html

Re: Go: Severe memory problems on 32bit Linux

#64

Post by jsnell »

smanek wrote: post #19
jedbrown wrote: post #14

However, it does not address the base issue, which is that Go uses a conservative garbage collector, and more values look like pointers in a 32-bit world. The only real fix would be to improve the garbage collector's understanding of which values are pointers and which are something else (e.g., floating point numbers that happen to look like pointers). And that is not an easy fix. How does this GC work? Is it literal…

Yep, that's basically it. It sounds crazy (and it is!) - but it often works reasonably well in practice. SBCL (one of the most performant Common Lisp implementations) has an 'imprecise gc' that works the same way - and I've seen reasonably heavily stressed processes with uptimes in the weeks/months. Remember, even a few years ago (before fastthread, etc) a reasonably loaded Ruby on Rails app couldn't stay up for more…

The SBCL garbage collector doesn't work like described in the grandparent comment. It's precise for the heap, and only conservative for the registers and stacks.

I can't say for sure how the Go GC works, but I would assume it likewise isn't fully conservative. E.g. the "avoid struct types with both integer and pointer fields" advice would be pointless for a fully conservative GC, but does make sense if structs containing no pointers are allocated in a separate memory region that's not scanned for potential pointers.

Re: Go: Severe memory problems on 32bit Linux

#65

Post by dchest »

stiff wrote: post #51

The problem itself seems quite serious and I would seriously reconsider using Go if I was interested in it in the first place, but after seeing the way this is treated by the Go "community", I am pretty sure I will never ever even think about using Go for anything.

the way this is treated by the Go "community"

What do you mean?

Re: Go: Severe memory problems on 32bit Linux

#66

Post by DisposaBoy »

riobard wrote: post #37
sunyc wrote: post #4

to go back to 32 bit hardware, you will be running very old machines that you can buy off ebay for <200$ and the power cost offset the new hardware cost in several months.

This is what I thought. Go is primarily targeted at server-side applications, which are almost exclusively running on 64bit hardware/OS. Memory is so cheap now that it's silly not to have more than 16GB plugged in the chasis. Then I realized there are many small VPS/EC2 instances with <1GB memory (I have two right now running on 64bit) and there are people who try to squeeze every last bit out of them by going 32bit.…

FWIW I currently have a bot(of sorts) that's been running months constantly crawling, processing(html and images) and logging other sites. It's doing all this in-memory and it peaks at about 34m even with over a hundred active goroutines (70+ are dedicated/never-ending that are usually doing something be it processing or interacting with remote services or external programs)... So I don't see the argument about servers with small amounts of memory. On the issue of improving memory use by going 32bit, I call bullshit on that one.

Re: Go: Severe memory problems on 32bit Linux

#67

Post by batista »

BarkMore wrote: post #13
wtallis wrote: post #2

Wow. That third message, with suggestions for avoiding the bug, reads like a twisted joke. Highlights: "avoid struct types which contain both integer and pointer fields" "avoid data structures which form densely interconnected graphs at run-time" "avoid integer values which may alias at run-time to an address; make sure most integer values are fairly low (such as: below 10000)" I understand that this isn't a complete…

Go 1 defines and the backwards-compatibility guarantees one can expect as the language matures. It's more a statement about the language specification than it is about the implementations. I know from experience that the gc 64-bit version is production ready. It's unfortunate that the limitations of the 32-bit version are not called out clearly on the website.

>I know from experience that the gc 64-bit version is production ready.

Is it really production ready though, or is it the same half-arsed implementation as the 32-bit one, just taking advantage of the larger availability of virtual memory on a 64 bit system?

Re: Go: Severe memory problems on 32bit Linux

#68

Post by el_muchacho »

moilolita wrote: post #62
nakkiel wrote: post #48

Earlier quoted context omitted.

I wrote a couple of D programs long ago and I still have a taste of an unfinished language. If I recall correctly, arrays manipulation is pretty weird and the language feels a bit the same PHP feels: a bunch of different things pieced together with no coherence. Compared to Python or Go, it's a whole different world.

The Phobos standard library is quite unfinished indeed and poorly documented in some areas but the D2 language is rather complete. It has sh*tloads of features but this also makes it a bit harder to master. Go is lighter (very easy to take on) and has an impressive library. Long story short: with D2 I needed ~1 month to get a good grasp of the language and the std library (the library was the hardest part) while with…

Agreed, D is a much bigger and complex language than Go.

Comparing the standard libraries, Phobos doesn't look far behind Go in scope. There are big holes though, like crypto, which is entirely missing, and a complete SQL driver (was in development, but we haven't heard from it for a while now), although there is a binding for SQLite3 and several drivers for major RDBMSs (not in the standard lib, tough). Logging will be included soon. Most of the rest is included (networking uses libCurl), and Phobos quality is continuously improving, some parts of it being excellent both in terms of functionality and performance, like the new regex library. On some other parts, like containers, Phobos seems much more advanced than Go. OTOH, there seems to be more 3-party libraries for Go than for D, but we can't comment on their quality. And of course, both languages allow to bind C libraries.

Re: Go: Severe memory problems on 32bit Linux

#69

Post by moilolita »

el_muchacho wrote: post #63
moilolita wrote: post #22

Earlier quoted context omitted.

D has a nice allocation management(i loved the manual+gc approach) and in my benchmarks, optimized D was slightly faster than Go at almost anything and consumed up to 70% less memory (I assumed that was because the Go gc kicked in later and was a bit lazy) but D was weak at threading/synchronization and the documentation of the standard lib was quite messy and lacking. So I decided that in the long run Go will be bet…

D is pretty strong at concurrency today. See for instance http://www.informit.com/articles/article.aspx?p=1609144 and http://ddili.org/ders/d.en/parallelism.html

It's strange that with all those threading examples, they didn't notice the need for a WaitGroup primitive. I know that you could implement it yourself. You could simulate the goroutines, implement channels and SCGI/FCGI and so on, but why bother when there is Go ?

Re: Go: Severe memory problems on 32bit Linux

#70

Post by dchest »

eternalban wrote: post #33
eternalban wrote: post #18

Earlier quoted context omitted.

Von Neumann architecture falsely lead us to the path with the singular notion of "memory". We need to distinguish at the hardware architecture level between "working memory" and "persistent memory". I doodle all the time when working. Pristine set of diagrams emerges from the chaos. I wish my computer would play along with this regime ..

I haven't mentioned downvotes to date, but the down votes on this is puzzling. Would the down voters care to comment as to why the comment is deemed down vote worthy? (I would think those who work/grok on memory managers do get the point.)

"Resist complaining about being downmodded. It never does any good, and it makes boring reading." http://ycombinator.com/newsguidelines.html
Post reply on HN