Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

From Python to Go and Back Again

docs.google.com

Post reply on HN
Print view View original

131–140 of 167 posts

Re: From Python to Go and Back Again

#131

Post by agentgt »

I have my own Go Heka story. I attempted to switch from Apache Flume to Heka mainly because of Flume taking vast amount of memory. I was hoping Heka would work but I think there must be some problems with the Golang AMQP drivers as memory usage would just continue to grow. This might have been my fault as I had to alter the Heka AMQP drivers to do things with the AMQP message headers.

The problem was pretty simple: pull event messages from AMQP and then shove them into elastic search and file system. Heka and Flume were both sort of overkill so I decided to write it in Rust. I got extremely far but alas there were some issues with the Elastic Search Rust library that I'm still resolving. Surprisingly the AMQP library worked pretty well.

I will vouch for the OP's point on error handling as Rust has a similar issue to Golang but not as bad because of the awesome type system (still I hate to admit but I really miss exceptions at times).

Anyway to relate again to the OP I went back to what I know best.. boring ass Java and wrote the app in a hour or so. It took about the same memory as Heka (surprising since its Java) and appeared to be slightly faster than Heka (elastic search indexing became the bottleneck for both so take that with a grain of salt).

Long story short.. I think the drivers and libraries really are the deal breakers and not so much the languages themselves (with some minor exceptions like the GIL).

Re: From Python to Go and Back Again

#132

Post by windlep »

Jabbles wrote: post #98
windlep wrote: post #42

Earlier quoted context omitted.

I should note, we don't care about throughput for the most part. Our constraint is purely the memory use of holding open the connections. The aim is to hold as many connections as possible within 10-20% of the machines RAM, and not exceed it. As such, we need to be careful about resource usage and spikes. Goroutines feel cheap, but if you're holding 140k connections, and just 20k of them do something that spins up a…

Typically if you wish to limit the number of goroutines you would spawn N workers and have them read from a single channel. If 20k of your incoming connections want to do something they send on the channel, without spawning a goroutine themselves. Did you try something like that?

Yep, this is what I meant by 'goroutine pools'. The select statements were on the sending side to ensure if the feed channel was full we wouldn't retain too much additional state. It works, but at that point its starting to look like an async event-loop with a thread-pool....

Re: From Python to Go and Back Again

#133

Post by perlgeek »

falcolas wrote: post #12

I'll be interested in coming back to Python when it isn't a headache to deploy into production. I'm tired of an install requiring a GCC compiler on the target node. I'm also tired of having to work around the language and ecosystem to avoid dependency hell.

FWIW we deploy python code as debian packages that we build with dh-virtualenv.

This bakes a whole virtualenv with all python dependencies (including compiled C libraries) into a .deb package. The packages tend to be big-ish (3MB to 15MB), but the target system only needs the right python version, nothing else.

Re: From Python to Go and Back Again

#134

Post by yokohummer7 »

gnuvince wrote: post #115
BerislavLopac wrote: post #107

Earlier quoted context omitted.

"huge unit testing burden necessary" Why do you consider unit testing a burden? I find unit tests the best way to formalize specifications before even starting to write code.

Unit testing is a burden because (a) a programmer needs to think of the cases that must be tested, (b) a programmer needs to actually write them, (c) they can only act as a safety net. When possible, it's very advantageous to encode your invariants into the type system. Yaron Minsky gives a good example in his Effective ML talk[1] about how you can take a common data structure and refactor it in such a way that the t…

Type systems are a burden because (a) a programmer needs to think of the types he or she should design, (b) a programmer needs to actually utilize those types to write his or her program, (c) they can only act as a safety net.

Re: From Python to Go and Back Again

#136

Post by windlep »

ssfak wrote: post #97
windlep wrote: post #52

I have to note this, because I think it deserves quite a bit more attention. SSL is extremely expensive, on RAM (perhaps the implementations have optimized for throughput over RAM). I have yet to benchmark any SSL implementation in any language, with any binding, that can use less than 20kb per SSL connection. I mentioned in my talk here that SSL is very expensive, here's my benchmark suite that others may add to: ht…

Doesn't a TLS terminator proxy solve this? E.g. I usually put my application services behind HTTPS-enabled nginx and it works wonderfully.

Nope, so, the goal here is to reduce how many machines (each with their own RAM limits) are used. The task is holding open bidirectional SSL wrapped long-lived websocket connections. They're held open for hours at a time, since we need to send notifications when we get them.

Every connection has a base cost of the TCP kernel send/recv buffer, which in our case we dropped a bit to 4kb each. So that's still 8kb per connection right there. If we terminate the SSL on a separate machine from where we handle the connection, then it means we'll be using 8kb more memory per connection. Probably even greater because nginx has its own send/recv buffers for data.

I'm sure our use-case is a unique one, most people care about raw through-put so the majority of SSL optimization has focused on lowering CPU use under high load rather than memory use under massive amounts of connections.

Re: From Python to Go and Back Again

#137

Post by falcolas »

nulltype wrote: post #62
falcolas wrote: post #57

Earlier quoted context omitted.

The way I deploy Go apps at $EMPLOYER2: - go get - go test - go build - copy to target It's possible with Python, it's easier with Go. It's a place where we could use a lot of progress.

It seems weird how you can't easily package python into an executable without Docker.

You can with various levels of success with a few "freeze" programs. They basically bundle up the entire environment into an executable, so the executables are stupidly large (more-or-less the size of your /usr/lib/python directory plus the python binaries), but they mostly work.

Re: From Python to Go and Back Again

#138

Post by pm90 »

yokohummer7 wrote: post #134
gnuvince wrote: post #115

Earlier quoted context omitted.

Unit testing is a burden because (a) a programmer needs to think of the cases that must be tested, (b) a programmer needs to actually write them, (c) they can only act as a safety net. When possible, it's very advantageous to encode your invariants into the type system. Yaron Minsky gives a good example in his Effective ML talk[1] about how you can take a common data structure and refactor it in such a way that the t…

Type systems are a burden because (a) a programmer needs to think of the types he or she should design, (b) a programmer needs to actually utilize those types to write his or her program, (c) they can only act as a safety net.

As always, there is a tradeoff. Like the gp mentioned, type systems allow you elliminate much unit test code that would otherwise be required.

Personally, I do love me a type system. Even if you have to think harder about how to architect your code, I think this kind of thinking is required for software to be good.

Re: From Python to Go and Back Again

#139

Post by pcwalton »

thwd wrote: post #109
peteretep wrote: post #88

Earlier quoted context omitted.

> the middlebrow dismissers The totally misplaced condescension is one of many reasons the Go community appears to be a net negative.

Come visit, we're quite friendly! You know what really grinds our gears, though? People who don't read documentation. If you don't read documentation, you'll get a negative vibe. Because your question is literally sitting at the top of the FAQ. It's been asked 2^1024 times before. WHY WON'T YOU READ THE DOCS?

I don't even know what "your question" you're referring to is, much less where to find it in the Go FAQ.

Re: From Python to Go and Back Again

#140

Post by Matthias247 »

windlep wrote: post #42
sethammons wrote: post #41

Earlier quoted context omitted.

I should reach out to our team that took python/twisted dealing with sockets and lots of concurrency and ported to Go and see if they would put together a similar presentation. Our case is a bit different, but we saw over 130x improvement in throughput going to Go. While they were in there, they increased monitoring, stability, and maintainability. More case studies to help others make informed choices. Sending that…

I should note, we don't care about throughput for the most part. Our constraint is purely the memory use of holding open the connections. The aim is to hold as many connections as possible within 10-20% of the machines RAM, and not exceed it. As such, we need to be careful about resource usage and spikes. Goroutines feel cheap, but if you're holding 140k connections, and just 20k of them do something that spins up a…

Not exactly related to Go/PyPy, but I'm curious whether you can say something about how you handle memory and bandwidth constraints?

E.g. what do you do if you want to send notifications to lots of clients but for some the connection is very slow (you would probably need to buffer the data)? Do you have hard limits of maximum buffered data until you close the connection? End to end backpressure (for which channels are quite good) doesn't seem like the best option for 1:N broadcasts, because then the slowest receiver slows down all others.

And what do you do with connections which are sending you lots of (probably unexcepted) data? Stop reading from that socket?

Post reply on HN