Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Brave overhauled its Rust adblock engine with FlatBuffers, cutting memory 75%

brave.com

Post reply on HN
Print view View original

141–150 of 289 posts

Re: Brave overhauled its Rust adblock engine with FlatBuffers, cutting memory 75%

#141

Post by josephg »

johncolanduoni wrote: post #134
shatsky wrote: post #101

Earlier quoted context omitted.

It does support dynamic libs, but virtually all important Rust software seems to be written without any consideration for it.

Rust ABI (as opposed to C ABI) dynamic libraries are incredibly fragile with regard to compiler/build environment changes. Trying to actually swap them out between separate builds is pretty much unsupported. So most of the benefits of dynamic libraries (sharing code between different builds, updating an individual dependency) are not achieved. They’re only really useful if you’re distributing multiple binary executab…

Yep that’s right. I’ve been working on a game with bevy. The Bevy game engine supports being dynamic linked during development in order to keep compile times down. It works great.

Re: Brave overhauled its Rust adblock engine with FlatBuffers, cutting memory 75%

#142

Post by undeveloper »

shatsky wrote: post #101
no_wizard wrote: post #99

Earlier quoted context omitted.

if I recall correctly Rust does not support any form of dynamic linking or library loading. Most of the community I’ve interacted with are big on either embedding a scripting engine or WASM. Lots of momentum on WASM based plugins for stuff. It’s a weakness for both Rust and Go if I recall correctly

It does support dynamic libs, but virtually all important Rust software seems to be written without any consideration for it.

the rust abi is explicitly unstable. there are community projects to bring dynamic linking, but it's mostly not worth it.

Re: Brave overhauled its Rust adblock engine with FlatBuffers, cutting memory 75%

#143

Post by ThierryBuilds »

antonok wrote: post #107
ThierryBuilds wrote: post #44

I am curious to know what was the contribution of switching to FlatBuffers in that improvement.

FlatBuffers was definitely the majority of the improvements here! On 64-bit systems, pointers themselves can really start to take up a lot of memory (especially if you multiply them across 100k+ adblock filters). Switching to array indices instead of pointers saves a lot of memory that's otherwise wasted when you don't need to address the entire possible memory space.

Insightful. Many thanks.

Flat buffers is know to bloat client code. Was any trick used to mitigate that?

Re: Brave overhauled its Rust adblock engine with FlatBuffers, cutting memory 75%

#144

Post by PunchyHamster »

no_wizard wrote: post #99
shatsky wrote: post #96

Earlier quoted context omitted.

And yet Rust ecosystem practically killed runtime library sharing, didn't it? With this mentality that every program is not a building block of larger system to be used by maintainers but a final product, and is statically linked with concrete dependency versions specified at development time. And then even multiple worker processes of same app can't share common code in memory like this lib, or ui toolkit, multimedi…

if I recall correctly Rust does not support any form of dynamic linking or library loading. Most of the community I’ve interacted with are big on either embedding a scripting engine or WASM. Lots of momentum on WASM based plugins for stuff. It’s a weakness for both Rust and Go if I recall correctly

Go supports plugins (essentially libraries) but its has a bunch of caveats. You can also

You can also link to C libs from both. I guess you could technically make a rust lib with C interface and load it from rust but that's obviously suboptimal

Re: Brave overhauled its Rust adblock engine with FlatBuffers, cutting memory 75%

#149

Post by tliltocatl »

terafo wrote: post #100
shatsky wrote: post #96

Earlier quoted context omitted.

And yet Rust ecosystem practically killed runtime library sharing, didn't it? With this mentality that every program is not a building block of larger system to be used by maintainers but a final product, and is statically linked with concrete dependency versions specified at development time. And then even multiple worker processes of same app can't share common code in memory like this lib, or ui toolkit, multimedi…

Dynamic libraries are a dumpster fire with how they are implemented right now, and I'd really prefer everything to be statically linked. But ideally, I'd like to see exploration of a hybrid solution, where library code is tagged inside a binary, so if the OS detects that multiple applications are using the same version of a library, it's not duplicated in RAM. Such a design would also allow for libraries to be update…

IMHO dynamic libraries are a dumpster fire because they are often used as a method to provide external interfaces, rather then just share common code.

Re: Brave overhauled its Rust adblock engine with FlatBuffers, cutting memory 75%

#150

Post by ensocode »

bqmjjx0kac wrote: post #2

I'm not sure how impressed I should feel about saving 45 MiB these days.

Be as impressed as you want but I think it is a very good sign that developers are taking care of it, and as this is a free to use product we can always be happy if someone boosts performance no matter how much
Post reply on HN