Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bpftune uses BPF to auto-tune Linux systems

github.com

Post reply on HN
Print view View original

41–50 of 78 posts

Re: Bpftune uses BPF to auto-tune Linux systems

#41

Post by pstuart »

spenczar5 wrote: post #34
nehal3m wrote: post #31

Earlier quoted context omitted.

If they can be reversed individually you can simply deduce by rolling back changes one by one, no?

Suppose you run a fleet of a thousand machines. They all autotune. They are, lets say, serving cached video, or something. You notice that your aggregate error rate been drifting upwards since using bpftune. It turns out, in reality, there is some complex interaction between the tuning and your routers, or your TOR switches, or whatever - there is feedback that causes oscillations in a tuned value, swinging between t…

In that scenario you could run it on a couple servers, compare and contrast, and then apply globally via whatever management tool you use.

Re: Bpftune uses BPF to auto-tune Linux systems

#42

Post by 6SixTy »

jiehong wrote: post #11
RandomThoughts3 wrote: post #8

Earlier quoted context omitted.

I would reverse the question: if it can be done by a BPF module, why should it be in the kernel? Distributions turning it on by default is another story. Maybe it deserves to be shiped on all the time but that's not the same thing as being part of the kernel.

Indeed! The kernel might already be too monolithic. This kernel parameters optimisation reminds me of PGO compilation in programs. Yet, perhaps the kernel could come with multiple defaults config files, each being a good base for different workloads: server, embedded, laptop, mobile, database, router, etc.

I second the different profiles for server, laptop, and so on. Though I know the kernel already comes with default configs, so I think there could be room for specialized kernel config options in addition to what's already there.

Though in my opinion, there's already kind of too much segmentation between the different use cases. A server is just a role for a computer, and embedded could literally mean anything. Quite a few WiFi access points come with a USB port on them so you can plug in a USB drive and start a SMB server.

Re: Bpftune uses BPF to auto-tune Linux systems

#43

Post by crest »

robinhoodexe wrote: post #6

Is tuning the TCP buffer size for instance worth it?

It depends mostly on the bandwidth-delay-product and packet loss you expect on each connection. A there is a vast difference between a local interactive SSH session and downloading a large VM image from across an ocean.

Re: Bpftune uses BPF to auto-tune Linux systems

#44

Post by KennyBlanken »

spenczar5 wrote: post #34
nehal3m wrote: post #31

Earlier quoted context omitted.

If they can be reversed individually you can simply deduce by rolling back changes one by one, no?

Suppose you run a fleet of a thousand machines. They all autotune. They are, lets say, serving cached video, or something. You notice that your aggregate error rate been drifting upwards since using bpftune. It turns out, in reality, there is some complex interaction between the tuning and your routers, or your TOR switches, or whatever - there is feedback that causes oscillations in a tuned value, swinging between t…

Presumably one would use autotune to find optimized parameters, and then roll those out via change control, either one parameter at a time, or a mix of parameters across the systems.

Alternatively: if you have a fleet of thousands of machines you can very easily do a binary search with them to a)establish the problem with the auto-tuner and then b)which of the changes it settled on are causing your problems.

I get the impression you've never actually managed a "fleet" of systems, because these techniques would have immediately occurred to you.

Re: Bpftune uses BPF to auto-tune Linux systems

#45

Post by spenczar5 »

KennyBlanken wrote: post #44
spenczar5 wrote: post #34

Earlier quoted context omitted.

Suppose you run a fleet of a thousand machines. They all autotune. They are, lets say, serving cached video, or something. You notice that your aggregate error rate been drifting upwards since using bpftune. It turns out, in reality, there is some complex interaction between the tuning and your routers, or your TOR switches, or whatever - there is feedback that causes oscillations in a tuned value, swinging between t…

Presumably one would use autotune to find optimized parameters, and then roll those out via change control, either one parameter at a time, or a mix of parameters across the systems. Alternatively: if you have a fleet of thousands of machines you can very easily do a binary search with them to a)establish the problem with the auto-tuner and then b)which of the changes it settled on are causing your problems. I get th…

Certainly when we managed Twitch’s ~10,000 boxes of video servers, neither of the tasks you describe would have been simple. We underinvested in tools, for sure. Even so, I don’t think you can really argue that dynamically changing configs like this are going to make life easier!

Re: Bpftune uses BPF to auto-tune Linux systems

#46

Post by westurner »

nine_k wrote: post #28
sgarland wrote: post #27

Earlier quoted context omitted.

Fair point, though I didn’t see any such option with this tool.

It's developed in the open; we can create Github issue. Actually https://github.com/oracle/bpftune/issues/99

In the existing issue, we can link to the code and docs that would need to be understood and changed:

usage, main() https://github.com/oracle/bpftune/blob/6a50f5ff619caeea6f04d...

- [ ] CLI opts: --pretend-allow or --log-only-allow or [...]

Probably relevant function headers in libbpftune.c:

bpftune_sysctl_write(

bpftuner_tunable_sysctl_write(

bpftune_module_load(

static void bpftuner_scenario_log(struct bpftuner *tuner, unsigned int tunable, ; https://github.com/oracle/bpftune/blob/6a50f5ff619caeea6f04d... https://github.com/oracle/bpftune/blob/6a50f5ff619caeea6f04d...

Re: Bpftune uses BPF to auto-tune Linux systems

#47

Post by LinuxBender »

usr1106 wrote: post #3

Interesting. But if tuning parameters to their best values were easy, shouldn't the kernel just do that in the first place?

HP-UX and to a lesser degree AIX had some of this. Instead of just static tunable values there were some tunable formulas exposed to the admin. If one knew what they were doing they could change the formula as needed without having to recompile the kernel. I would not mind something like this being exposed from the Linux kernel as tunable options as opposed to just static values. Such options could reduce the number of companies that need to recompile the kernel.

Re: Bpftune uses BPF to auto-tune Linux systems

#48

Post by zymhan »

sgarland wrote: post #25
mrbluecoat wrote: post #17

> bpftune is designed to be zero configuration; there are no options On behalf of every junior administrator, overworked IT admin, and security-concerned "cattle" wrangler, thank you. Having to learn a thousand+ knobs & dials means most will never be touched. I for one welcome automated assistance in this area, even if the results are imperfect.

I think it’s still important to know what those dials and knobs do, otherwise (as the currently top-voted comment says) when things break, you’ll be lost.

That's exactly why it's such a burden.

Re: Bpftune uses BPF to auto-tune Linux systems

#50

Post by withinboredom »

londons_explore wrote: post #16
viraptor wrote: post #9

Earlier quoted context omitted.

It depends. At home - probably not. On a fleet of 2000 machines where you want to keep network utilisation close to 100% with maximal throughput, and the non-optional settings translate to a not-trivial value in $ - yes.

TCP parameters are a classic example of where an autotuner might bite you in the ass... Imagine your tuner keeps making the congestion control more aggressive, filling network links up to 99.99% to get more data through... But then any other users of the network see super high latency and packet loss and fail because the tuner isn't aware of anything it isn't specifically measuring - and it's just been told to make t…

It literally covers this exact scenario in the readme and explains how it prevents that.
Post reply on HN