Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Rust Lets Us Monitor 30k API calls/min

blog.bearer.sh

Post reply on HN
Print view View original

71–80 of 81 posts

Re: How Rust Lets Us Monitor 30k API calls/min

#71

Post by jeffbee »

staticassertion wrote: post #66
jeffbee wrote: post #65

Earlier quoted context omitted.

Then you just weren't paying attention. There's a list of approved languages for new development and Rust isn't on it. There's a list of languages that are forbidden, for which you need high-level approval to use for new work, and Rust is on that one, next to C++.

Rust support is categorized as Tier 2 at Dropbox. Do you work at Dropbox? You can go look for the language approval list, which documents this. Tier 2 means it requires approval. That is massively different from "forbidden". There is, for example, a Tier 3 list - Java is on there, some other languages, they're discouraged a lot more strongly than Rust though you'll still find them in some parts of the codebase (prima…

You've (probably unintentionally, judging by your tone) shined a light on the corrupt culture of that engineering org. It's written down that Tier 2 languages require special permission to begin new projects. It doesn't sound like we disagree on that. The corrupt cultural aspect is there exists an in-group clique of engineers who can and will start projects in any language they want, and an out-group against whom the written policy will be used to stop new efforts.

It is definitely not the case that any randomly selected backend engineer at that company can just pick up Rust and solve any problem with it, because the Tier 2 status is used as a cudgel to stop most such efforts.

Re: How Rust Lets Us Monitor 30k API calls/min

#72

Post by staticassertion »

jeffbee wrote: post #71
staticassertion wrote: post #66

Earlier quoted context omitted.

Rust support is categorized as Tier 2 at Dropbox. Do you work at Dropbox? You can go look for the language approval list, which documents this. Tier 2 means it requires approval. That is massively different from "forbidden". There is, for example, a Tier 3 list - Java is on there, some other languages, they're discouraged a lot more strongly than Rust though you'll still find them in some parts of the codebase (prima…

You've (probably unintentionally, judging by your tone) shined a light on the corrupt culture of that engineering org. It's written down that Tier 2 languages require special permission to begin new projects. It doesn't sound like we disagree on that. The corrupt cultural aspect is there exists an in-group clique of engineers who can and will start projects in any language they want, and an out-group against whom the…

I disagree entirely with the "forbidden" wording. That's it. Tier 2 is not "Forbidden" it is a statement on the level of support.

I agree with your point about political issues - the entire tier list, specifically even Rust being tier 2, was political. I saw stupid shit like that too many times at Dropbox.

But it is simply a fact that new projects are being built in Rust, regardless of the political aspects of why that is the case despite it being tier 2. "Forbidden" does not convey the state of things - everything else you've said is agreeable.

Re: How Rust Lets Us Monitor 30k API calls/min

#73

Post by bluestreak »

masklinn wrote: post #68
TheTank wrote: post #13

Earlier quoted context omitted.

One step beyond good Java GCs is to write fully zero-GC Java code. The advantage of it is complete control over your performance which means your software is going to be consistently fast. The disadvantage is that it is relatively difficult to obtain. If you want to see an example of fully zero-GC Java, you can check out QuestDB on Github [1] - Disclaimer I work for QuestDB. [1] https://github.com/questdb/questdb

> One step beyond good Java GCs is to write fully zero-GC Java code. The advantage of it is complete control over your performance which means your software is going to be consistently fast. The disadvantage is that it is relatively difficult to obtain. I don't know that it's actually possible in the general case, as Java's support for value types remains wholly insufficient. IIRC the ixy folks never managed to remov…

It is possible, we've done it.

Re: How Rust Lets Us Monitor 30k API calls/min

#74

Post by eggsnbacon1 »

lostcolony wrote: post #29
eggsnbacon1 wrote: post #25

Earlier quoted context omitted.

Shenandoah and ZGC collectors have worse case pauses of ~10ms and average pauses of 0.5ms . The average pause is faster than malloc() sometimes in C, so you won't really be introducing more latency than C does. The other option is to avoid allocating memory at all, which you could do in C/Rust but also in Java. The vast majority of shops given the choice for low/no allocation performance use Java anyways (HFT)

But those GCs also don't guarantee all garbage has been collected, nor how much processing time you'll get before they run again. So op could still end up stuck with their code barely executing, due to memory and CPU pressure, and throughput/latency drops to zilch. "The vast majority of shops" is an interesting metric given the vast majority had to pick a language before Rust existed. Java and trying to minimize allo…

> But those GCs also don't guarantee all garbage has been collected, nor how much processing time you'll get before they run again. So op could still end up stuck with their code barely executing, due to memory and CPU pressure, and throughput/latency drops to zilch.

You could malloc() and free() so much that the code doesn't have time to do anything too. And these operations aren't bounded in time either. Just using C doesn't save you from memory allocation, its just done manually instead of automatic. In every system I've worked on you won't have this kind of GC pressure unless you do something profoundly bad

> The vast majority of shops" is an interesting metric given the vast majority had to pick a language before Rust existed.

true

Re: How Rust Lets Us Monitor 30k API calls/min

#75

Post by [deleted] »

meritt wrote: post #6

Sorry, I must be missing something in this blog post because the requirements here sound incredibly minimal. You just needed an HTTP service (sitting behind an Envoy proxy) to process a mere 500 requests/second (up to 1MB payload) and pipe them to Kinesis? How much data preparation is happening in Rust? It sounds like all the permission/rate-limiting/etc happens between Envoy/Redis before it ever reaches Rust? I know…

[deleted]

Re: How Rust Lets Us Monitor 30k API calls/min

#77

Post by lostcolony »

eggsnbacon1 wrote: post #74
lostcolony wrote: post #29

Earlier quoted context omitted.

But those GCs also don't guarantee all garbage has been collected, nor how much processing time you'll get before they run again. So op could still end up stuck with their code barely executing, due to memory and CPU pressure, and throughput/latency drops to zilch. "The vast majority of shops" is an interesting metric given the vast majority had to pick a language before Rust existed. Java and trying to minimize allo…

> But those GCs also don't guarantee all garbage has been collected, nor how much processing time you'll get before they run again. So op could still end up stuck with their code barely executing, due to memory and CPU pressure, and throughput/latency drops to zilch. You could malloc() and free() so much that the code doesn't have time to do anything too. And these operations aren't bounded in time either. Just using…

You could, sure. You could do (anything) and still block (other parts of the code). Point is, GC isn't necessary for executing code, malloc/free is (well, for non-trivial programs that have data whose size is unknown at compile time).

With GC, I've definitely had plenty of times where GC created very noticeable pressure on the running system. Nothing quite as bad as they describe here, true, but that would probably predispose me to ditching GC if I saw that kind of behavior, too (since it's so unlikely that either my case is truly deviant, or my devs have done something very wrong, and neither of those is something I want to bet on being able to fix).

Re: How Rust Lets Us Monitor 30k API calls/min

#78

Post by chrischen »

penagwin wrote: post #47
chrischen wrote: post #18

Earlier quoted context omitted.

Does Rust offer much of the same language features as F#?

Disclaimer: not a rust of .net dev, but my impression is they’re very different in what they’re trying to. It’d would be like comparing C to Python (as an example, not as an analogy). Rust is closer to a super fancy C, compiles natively and was made with a heavily focus on certain types of memory safety. F# has syntax more like Haskell/ML, and is compiled to a bytecode instead of an executable. It runs on .NET and ev…

That's actually why I'm asking about that since the OP is transitioning backend logic from F# to Rust. So either Rust can cover a lot of the language features or he has a use case that really warrants that performance (and F# on .NET is not exactly non-performant).

Re: How Rust Lets Us Monitor 30k API calls/min

#79

Post by [deleted] »

meritt wrote: post #6

Sorry, I must be missing something in this blog post because the requirements here sound incredibly minimal. You just needed an HTTP service (sitting behind an Envoy proxy) to process a mere 500 requests/second (up to 1MB payload) and pipe them to Kinesis? How much data preparation is happening in Rust? It sounds like all the permission/rate-limiting/etc happens between Envoy/Redis before it ever reaches Rust? I know…

[deleted]

Re: How Rust Lets Us Monitor 30k API calls/min

#80

Post by thaneross »

zozbot234 wrote: post #62
stuartd wrote: post #45

Earlier quoted context omitted.

> Is your code less functional and more imperative style due to Rust? I would imagine so. Rust doesn't support tail call optimization, and variables are immutable only by default.

LLVM should optimize tailcalls and sibcalls. But tail call optimization has unexpected interactions with the extended RAII that Rust uses because stuff has to be dropped at the end of its lifetime, so the code that's running in "tail" position is sometimes not what you expect.

As a beginner to Rust I'm surprised by this. Given the Rust compiler is able to figure out the lifetimes in the recursive case, you'd think the lifetimes within the tail-optimized loop would be same. Doesn't the lexical scoping of the loop's body have the equivalent lifecycle of a recursive call (drop at the end of the loop vs the end of the function)?
Post reply on HN