Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Beauty of Concurrency in Go

pragprog.com

Post reply on HN
Print view View original

11–20 of 95 posts

Re: The Beauty of Concurrency in Go

#11

Post by NyxWulf »

StavrosK wrote: post #3

Ugh, I just finished writing the XMPP frontend for an XMPP/IRC bot I'm working on ( http://www.getinstabot.com ). The frontends are in Go for concurrency, and ferry messages back and forth from the channels to the backend. Let me tell you, that problem is hard . Go coped pretty well, but the final thing is a mess of global states, and it's pretty elegant for what the problem is. I was hoping to avoid having many movi…

Observing other channels, or seeing when a channel does fire is what the select structure is for. It is one of the features that makes channels and goroutines very nice in go.

http://golang.org/ref/spec#Select_statements

http://blog.golang.org/2010/09/go-concurrency-patterns-timin...

http://talks.golang.org/2012/concurrency.slide#1

Video for the slides: http://www.youtube.com/watch?v=f6kdp27TYZs

Re: The Beauty of Concurrency in Go

#12

Post by StavrosK »

pandemicsyn wrote: post #10
StavrosK wrote: post #3

Ugh, I just finished writing the XMPP frontend for an XMPP/IRC bot I'm working on ( http://www.getinstabot.com ). The frontends are in Go for concurrency, and ferry messages back and forth from the channels to the backend. Let me tell you, that problem is hard . Go coped pretty well, but the final thing is a mess of global states, and it's pretty elegant for what the problem is. I was hoping to avoid having many movi…

heads up. Seems like https://getinstabot.com and http://www.getinstabot.com end up in a redirect loop.

Oh. I forgot about this comment and picked a great time to redirect everything over https. Deploying the fix now, thanks.

Re: The Beauty of Concurrency in Go

#13

Post by sirclueless »

Jabbles wrote: post #8

This article isn't bad... but it misses several important points of Go. I also note the article is 9 months old. In the hope that my criticism will be taken as constructive, with apologies for not writing detailed explanations: 1. Goroutines are not threads 2. type inference allows you to elide types in var declarations: var host = flag.String(... 3. Go's convention is to use camel case, not underscores. 4. Calling o…

> I think that simply moving line 58 to between 61,62 would fix this.

Doing that would erase the whole benefit of allocating a static array in the first place (you really don't want to be allocating memory for every packet that comes through). The funny thing is the author correctly handles this type of synchronization problem elsewhere, see lines 97-100. The right thing to do is to wait for acknowledgement from both loggers before reusing shared memory, but that sort of defeats his broader argument: if you are explicitly guarding every shared memory access with a barrier, then you might as well have been using any shared memory language besides Go. Buffered channels in both directions used to implement barriers are every bit as error-prone and complex as using a reader-writer lock explicitly.

Re: The Beauty of Concurrency in Go

#14

Post by StavrosK »

burntsushi wrote: post #9
StavrosK wrote: post #3

Ugh, I just finished writing the XMPP frontend for an XMPP/IRC bot I'm working on ( http://www.getinstabot.com ). The frontends are in Go for concurrency, and ferry messages back and forth from the channels to the backend. Let me tell you, that problem is hard . Go coped pretty well, but the final thing is a mess of global states, and it's pretty elegant for what the problem is. I was hoping to avoid having many movi…

> I know it's possible with channels, but when one goroutine is blocked on network Recv(), there's not much of a chance to listen to channels. Could you elaborate a bit more on this point? This seems like a natural pattern for a channel/goroutine. Spin up a goroutine that reads from a connection and send whatever is read on a channel. Then other goroutines can synchronize on the channel.

Sure, but what if you want to shut that goroutine down later on?

The best solution that I've found is to just close the connection, the .Recv() will error out immediately.

Re: The Beauty of Concurrency in Go

#15

Post by lobster_johnson »

StavrosK wrote: post #3

Ugh, I just finished writing the XMPP frontend for an XMPP/IRC bot I'm working on ( http://www.getinstabot.com ). The frontends are in Go for concurrency, and ferry messages back and forth from the channels to the backend. Let me tell you, that problem is hard . Go coped pretty well, but the final thing is a mess of global states, and it's pretty elegant for what the problem is. I was hoping to avoid having many movi…

> Something I miss from the language …

Does sound like you want to use channels, and break your logic into small independent parts.

You will have one goroutine using blocking Read() in a loop and feeding data to some channel. When it's done, you write to another channel that exists only for signaling:

    defer {
      doneChannel
and then in your other goroutine:

    for {
      select {
        case data :=
The only things shared here are the channels.

The way to avoid too much state and moving parts is to break the problem into isolated, manageable parts that communicate with channels. Often you will have hierarchical relationships like this, where one piece of dumb code exists to pass data from something lower down to somewhere higher up.

It's not hard, although some of the code gets a bit ugly and disjoint at times, especially in how anything synchronous has to use channels and goroutines. For example, today I wrote a simple worker pool implementation that runs a given function in parallel via goroutines and can adjust the number of workers dynamically at runtime. That function has to be declared not just as "func()" but as "func(abortChannel chan bool)", and the worker function has to honour the abort signal when it arrives from the pool. So channels do leak everywhere, even into APIs. (Yeah, I know I can use "chan struct{}" to avoid any storage, but I think "What is harder is to intelligently handle complex cascading failures. That's what Erlang, with its supervisor tree, is good at. Go's goroutines are "fire and forget" and cannot even be terminated programmatically from elsewhere in the program.

Re: The Beauty of Concurrency in Go

#16

Post by EllaMentry »

This article and many like suffer from one of my huge pet-peeves, absolutely terrible coding conventions.

I am a person who likes to scan articles, I'm busy and generally make a read now, read later, read never decision. The code from first scan was unreadable, short 1 character variable names, "why is there a hardcoded date marked 2006.01.02-15.04.05 there??", etc.

Readable code takes a little more time - but it's worth it!

Further, the entire example seems contrived. Am I right in thinking that simply firing up wireshark would solve this problem? Why is the author continuing to write something in nearly every language when a tool exists for exactly this purpose, is multi-featured and pluggable?

Even further, message passing! With the rise in recent years of message passing libraries in nearly every language, multi-threaded, distributed applications are becoming trivial to write. I do not see the Go code presented as anything other than messy, I have seen C++ code utilizing message passing libs that are smaller, prettier and infinitely more maintainable - again with no mutex or conditional variables!

If you want to sell me on Go, make the code pretty, and present a USP.

Re: The Beauty of Concurrency in Go

#17

Post by Jabbles »

sirclueless wrote: post #13
Jabbles wrote: post #8

This article isn't bad... but it misses several important points of Go. I also note the article is 9 months old. In the hope that my criticism will be taken as constructive, with apologies for not writing detailed explanations: 1. Goroutines are not threads 2. type inference allows you to elide types in var declarations: var host = flag.String(... 3. Go's convention is to use camel case, not underscores. 4. Calling o…

> I think that simply moving line 58 to between 61,62 would fix this. Doing that would erase the whole benefit of allocating a static array in the first place (you really don't want to be allocating memory for every packet that comes through). The funny thing is the author correctly handles this type of synchronization problem elsewhere, see lines 97-100. The right thing to do is to wait for acknowledgement from both…

> Doing that would erase the whole benefit of allocating a static array in the first place (you really don't want to be allocating memory for every packet that comes through).

The benefit is dubious. Performance is unlikely to be important in a single-connection logger. My advice is to lean on the garbage collector. Allocate away.

Buffered channels are certainly more complex, but if you're never sharing memory, shouldn't hurt you.

Re: The Beauty of Concurrency in Go

#18

Post by lobster_johnson »

StavrosK wrote: post #14
burntsushi wrote: post #9

Earlier quoted context omitted.

> I know it's possible with channels, but when one goroutine is blocked on network Recv(), there's not much of a chance to listen to channels. Could you elaborate a bit more on this point? This seems like a natural pattern for a channel/goroutine. Spin up a goroutine that reads from a connection and send whatever is read on a channel. Then other goroutines can synchronize on the channel.

Sure, but what if you want to shut that goroutine down later on? The best solution that I've found is to just close the connection, the .Recv() will error out immediately.

Yes, that is afaik the only way to abort a synchronous function call such as TCPConn.Read().

Re: The Beauty of Concurrency in Go

#19

Post by StavrosK »

lobster_johnson wrote: post #15
StavrosK wrote: post #3

Ugh, I just finished writing the XMPP frontend for an XMPP/IRC bot I'm working on ( http://www.getinstabot.com ). The frontends are in Go for concurrency, and ferry messages back and forth from the channels to the backend. Let me tell you, that problem is hard . Go coped pretty well, but the final thing is a mess of global states, and it's pretty elegant for what the problem is. I was hoping to avoid having many movi…

> Something I miss from the language … Does sound like you want to use channels, and break your logic into small independent parts. You will have one goroutine using blocking Read() in a loop and feeding data to some channel. When it's done, you write to another channel that exists only for signaling: defer { doneChannel and then in your other goroutine: for { select { case data := The only things shared here are the…

Yeah, the channel approach was rewrite #3 :P It required me to have shared state in a horrible way as well, ruining the whole thing. There really isn't an elegant/functional way to do it, and sharing channels is error-prone itself. I ended up closing the connection to terminate goroutine B from A, and messaging on an already-shared channel to terminate A from B.

Having cascading failures in Go would be fantastic. The problem, unfortunately, is not very amenable to elegant solutions. I might do a writeup at some point.

Re: The Beauty of Concurrency in Go

#20

Post by burntsushi »

StavrosK wrote: post #14
burntsushi wrote: post #9

Earlier quoted context omitted.

> I know it's possible with channels, but when one goroutine is blocked on network Recv(), there's not much of a chance to listen to channels. Could you elaborate a bit more on this point? This seems like a natural pattern for a channel/goroutine. Spin up a goroutine that reads from a connection and send whatever is read on a channel. Then other goroutines can synchronize on the channel.

Sure, but what if you want to shut that goroutine down later on? The best solution that I've found is to just close the connection, the .Recv() will error out immediately.

What other solution would you want? You could use `SetDeadline` if you like. But closing the connection seems reasonable too.

I think you're framing the problem wrong. You're not asking to shutdown a goroutine, you're asking "How do I abort from a synchronous read from a network connection that is blocked?"

Post reply on HN