Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go hits the concurrency nail on the head

eli.thegreenplace.net

Post reply on HN
Print view View original

311–318 of 318 posts

Re: Go hits the concurrency nail on the head

#311

Post by wahern »

kasey_junk wrote: post #233
wahern wrote: post #171

Earlier quoted context omitted.

All of those interfaces are trivial to implement in Go precisely because Go implements a much stronger abstraction. By contrast, if you only have those other abstractions you're quite limited in how you can structure your code. (You might not realize how limited you are, however, if those are your only options.) As the article says, most languages settle for those interfaces because they can mostly be implemented as…

The biggest problem with go is that it’s not easy to implement those as libraries. This is a combination of the golang story around generics & their opinionated strategy on concurrency. Conversely most other modern, mainstream languages can mimic golang concurrency as a library.

You absolutely cannot mimic Go concurrency. You're not understanding (or not appreciating) the function color problem mentioned in the article. See, e.g., http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y...

Re: Go hits the concurrency nail on the head

#312

Post by wahern »

kasey_junk wrote: post #266
tandr wrote: post #264

Earlier quoted context omitted.

Would you be so kind and explain what term "high concurrency levels" implies? thank you in advance.

Consumer facing systems. System wide throughput between 6-12 million QPS (daily low/high) (query body average size is 1.5KB). Each server tops at ~130K QPS. On a system with 2 1 GB NICs we pop the NIC. On a 10G we pop the CPU. Current bottleneck is the golang http/net libs. Would likely need to rewrite it from the NIC up to do better.

That's an issue with the http/net libraries, not the concurrency model.

At really high throughout you can run into into issues with the kernel's networking and driver stack. I've encountered situations with my own homegrown event libraries (mostly C or Lua+C; I've never used Go) that were bottlenecked in the kernel. I've also seen issues that were fundamentally related to the use of poor buffering and processing pipeline strategies that resulted in horrible performance. For example, I can get an order of magnitude greater streaming throughput using my own protocol and framing implementations than when using ffmpeg's, though I use ffmpeg's codecs and a non-blocking I/O model in both cases, all in C. And that's because of how I structured the flow of data through my processing pipeline.

There's is no general model of concurrency that can solve that, and I've never seen any model that was easier in the abstract to tweak than any others. Those are implementation issues.

Re: Go hits the concurrency nail on the head

#313

Post by kasey_junk »

apta wrote: post #288
kasey_junk wrote: post #266

Earlier quoted context omitted.

Consumer facing systems. System wide throughput between 6-12 million QPS (daily low/high) (query body average size is 1.5KB). Each server tops at ~130K QPS. On a system with 2 1 GB NICs we pop the NIC. On a 10G we pop the CPU. Current bottleneck is the golang http/net libs. Would likely need to rewrite it from the NIC up to do better.

Do you think the JVM would have faired better at this? Otherwise, perhaps something like Rust would be a better fit.

I don’t know if it would have been holistically better, golang has lots of advantages.

But the concurrency would have been more straight forward on the JVM because the language allows for more choices & their are lots of options that get you there.

Re: Go hits the concurrency nail on the head

#314

Post by kasey_junk »

wahern wrote: post #311
kasey_junk wrote: post #233

Earlier quoted context omitted.

The biggest problem with go is that it’s not easy to implement those as libraries. This is a combination of the golang story around generics & their opinionated strategy on concurrency. Conversely most other modern, mainstream languages can mimic golang concurrency as a library.

You absolutely cannot mimic Go concurrency. You're not understanding (or not appreciating) the function color problem mentioned in the article. See, e.g., http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y...

Or it’s possible I don’t agree with its premise that go solves the problem better than other languages because it hides async behaviors from the type system.

In practice it doesn’t. Asynchronous functions leak into golang implementations in worse ways in golang. Everything from the near universal use of channels as poorly implemented promises to the horrendous Context being passed to everything for cancellations. The golang concurrency story is weak compared to any language that has a story at all.

Re: Go hits the concurrency nail on the head

#315

Post by zbentley »

Matthias247 wrote: post #230
pcwalton wrote: post #184

Earlier quoted context omitted.

I think more languages should have typed channels, but untyped channels are readily available via OS primitives. Unix has had pipe() since 1973.

True! Haven't thought about it, but it's actually an OS-backed untyped channel, which even supports select(). Guess the differences are: It operates on bytes and not on objects, so a custom protocol is needed on top of it. And it can't provide the same guarantees as an unbounded channel, where there are certain guarantees that sending an item actually reached the other site, and which makes guaranteed resource handof…

> It operates on bytes and not on objects, so a custom protocol is needed on top of it.

That's why typed channels are useful, at least at the copying level (when it's more than just an array of lockable memory addresses underneath--data has to actually get transferred)--the typing is the message delimiter, which itself is the protocol.

Re: Go hits the concurrency nail on the head

#317

Post by apta »

wahern wrote: post #310
apta wrote: post #287

Earlier quoted context omitted.

> If you think actors, futures, and async/await are so great, imagine having to write every single function invocation in that manner. I don't see how: val f = Future { foo() } // do work... f match { case Success(res) => useRes(res) case Failure(error) => handleError(error) } Is any more difficult than: done := make(chan bool) var res int go foo(done, &res) As a matter of fact, the first one is much easier. I don't…

You're not understanding (or at least not appreciating) the function color problem mentioned in the article. See, e.g., http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y...

I read that article before. Here's the thing, the only thing that the "go" keyword makes easier is having the ability to have any arbitrary function run in a green thread without having to modify it's signature. However, the moment you want to actually do something useful with it (e.g. communicate with it, cancel it, or read its returned value), you're going to have to pass a channel (or more) or a waitgroup to it anyway, modifying its signature and changing its "color". The issue remains pretty much the same.

Re: Go hits the concurrency nail on the head

#318

Post by sofaofthedamned »

Sharlin wrote: post #151
dnautics wrote: post #143

Earlier quoted context omitted.

Anecdotally, there are at least three people at work who say that "go is used at google so it will be around forever and I can be sure there will be support for it".

I'm sure somebody said the same thing about GWT too.

I actually really loved GWT and it was a shame it withered and died.
Post reply on HN