Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Taming Go’s memory usage, or how we avoided rewriting our client in Rust

akitasoftware.com

Post reply on HN
Print view View original

151–160 of 231 posts

Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust

#151

Post by klyrs »

Zababa wrote: post #101
pjmlp wrote: post #87

Earlier quoted context omitted.

To be fair, that is now the common "I rewrote X in Y" theme, which followed upon the Y ∈ { Ruby, Clojure, Scala, Kotlin,.... } from previous years.

And Go too! It's always fun to see posts from around 2014/2015 complaining about how every submission to Hacker News is now "I wrote X in Go", while now Go is the boring stuff and Rust is the hot new thing. I wonder what will be the next Rust though.

Write it in Malbolge? Slightly less silly, modern C++ and modern Python are evergreen.

Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust

#152

Post by SAI_Peregrinus »

tptacek wrote: post #104
Zababa wrote: post #101

Earlier quoted context omitted.

And Go too! It's always fun to see posts from around 2014/2015 complaining about how every submission to Hacker News is now "I wrote X in Go", while now Go is the boring stuff and Rust is the hot new thing. I wonder what will be the next Rust though.

BPF-verified C.

Naw, BPF-verified BF. Or someone will make "BrainFuck Plus" so we can get BPF-verified BFP.

Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust

#153

Post by klyrs »

silisili wrote: post #141
tptacek wrote: post #39

The big wins in this article, in what I believe was the order of impact: * They do raw packet reassembly using gopacket, and gopacket keeps TCP reassembly buffers that can grow without bound when you miss a TCP segment. They capped the buffers, and the huge 5G spikes went away. * They were reading whole buffers into memory before handing them off to YAML and JSON parsers. They passed readers instead. * They were usin…

> They stopped compiling regexps on the fly and moved the regexps to package variables. (I actually don't know if this was a significant win; there might just be the three big wins.) Anecdotally, this could be a huge win, depending on how often it's called. A guy I was working with, new to Go, was writing a router config parser and asked why it was so slow. The first thing I did was moved regexp.Compile from a hot pa…

I've seen the same in Python, probably a dozen times. Sometimes folks think it's ugly (un-pythonic) but there's plenty of cases in the standard library to point to.

Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust

#154

Post by Karrot_Kream »

jhgb wrote: post #130
Karrot_Kream wrote: post #113

Earlier quoted context omitted.

> it's an intrinsic Law Of The Universe that if data comes in a X bytes per second, and leaves at X-k bytes per second, then eventually you will use all storage space in the Universe for your buffer, This is known as Little's Law. Using Little's Law, you know that if the average time spent in queue is more than the average time it takes for a new entry to be added to the queue, then your queue fills up.

Did Little formulate multiple eponymous laws? Since that does not seem to be the Little's law that I'm familiar with.

Here's a good introduction to Little's Law and associated operational rules derived from it on queues: http://web.eng.ucsd.edu/~massimo/ECE158A/Handouts_files/Litt...

Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust

#155

Post by marcos100 »

pm90 wrote: post #138
marcos100 wrote: post #128

Earlier quoted context omitted.

People who is interested in rust maybe want to see how it was used. The author could have just kept "Taming Go's Memory Usage". Maybe they never considered rewriting in rust. The pros and cons looks like just some random arguments to add rust to the title.

> The author could have just kept "Taming Go's Memory Usage". It’s their article. They can choose to write it however they want. You may find this type of humor distasteful, fine, write your articles that way. As a user of both rust and golang, I chuckled at the headline and then forgot about it.

Good for you that you like clickbait headlines.

Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust

#156

Post by jhgb »

Karrot_Kream wrote: post #154
jhgb wrote: post #130

Earlier quoted context omitted.

Did Little formulate multiple eponymous laws? Since that does not seem to be the Little's law that I'm familiar with.

Here's a good introduction to Little's Law and associated operational rules derived from it on queues: http://web.eng.ucsd.edu/~massimo/ECE158A/Handouts_files/Litt...

Thanks, but I had already had courses on that. We never associated the condition for stability (λ<μ) with Little's law (L=λW).

Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust

#157

Post by makeworld »

klyrs wrote: post #153
silisili wrote: post #141

Earlier quoted context omitted.

> They stopped compiling regexps on the fly and moved the regexps to package variables. (I actually don't know if this was a significant win; there might just be the three big wins.) Anecdotally, this could be a huge win, depending on how often it's called. A guy I was working with, new to Go, was writing a router config parser and asked why it was so slow. The first thing I did was moved regexp.Compile from a hot pa…

I've seen the same in Python, probably a dozen times. Sometimes folks think it's ugly (un-pythonic) but there's plenty of cases in the standard library to point to.

That's because the Python regex module caches the regexes it compiles, so it only happens once. It's proper and good usage to specify the regex string inline, even in a hot path.

I'd only use a variable when I'm using the same regex multiple times in code, and even then I could still just have the variable be the string.

Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust

#159

Post by zarzavat »

roca wrote: post #135
jgrant27 wrote: post #111

After using Rust for a few years professionally it's my take that people that really want to use it haven't had much experience with it on real world projects. It just doesn't live up to the hype that surrounds it. The memory and CPU savings are negligible between Go and Rust in practice no matter what people might claim in theory. However, the side effects of making your team less productive by using Rust is a much…

> the side effects of making your team less productive by using Rust is a much higher price to pay than just running you Go service on more powerful hardware. This entirely depends on the ratio of development effort to deployed instances. At one end of the spectrum, lots of developers work for years on a system which is only deployed on one machine; obviously you optimize for developer effort and buy a single massive…

My problem with Rust: I'm sure that if I used it as my primary language for a couple of years, I would be able to claw the productivity loss back. But I can't find any reason to justify using it at my current productivity level.

There is a vicious cycle: few projects use Rust because the productivity hit is large, and programmers do not get enough experience using Rust because few projects use it.

Re: Taming Go’s memory usage, or how we avoided rewriting our client in Rust

#160

Post by klyrs »

makeworld wrote: post #157
klyrs wrote: post #153

Earlier quoted context omitted.

I've seen the same in Python, probably a dozen times. Sometimes folks think it's ugly (un-pythonic) but there's plenty of cases in the standard library to point to.

That's because the Python regex module caches the regexes it compiles, so it only happens once. It's proper and good usage to specify the regex string inline, even in a hot path. I'd only use a variable when I'm using the same regex multiple times in code, and even then I could still just have the variable be the string.

Yeah, it's been a while since I've benchmarked that, I'll try it out
Post reply on HN