Live data from Hacker News

Bpftune uses BPF to auto-tune Linux systems

github.com

31–40 of 78 posts

Re: Bpftune uses BPF to auto-tune Linux systems

#31

Earlier quoted context omitted.

>"bpftune logs to syslog so /var/log/messages will contain details of any tuning carried out." (from OP GitHub readme) The rmem example seems to allay fears that it will make changes one can't reverse.

It’s not a questions of being able to reverse. It’s a question of being able to diagnose that one of these changes even was the problem and if so which one.

If they can be reversed individually you can simply deduce by rolling back changes one by one, no?

Re: Bpftune uses BPF to auto-tune Linux systems

#32
post #31

Earlier quoted context omitted.

It’s not a questions of being able to reverse. It’s a question of being able to diagnose that one of these changes even was the problem and if so which one.

If they can be reversed individually you can simply deduce by rolling back changes one by one, no?

Only if you already suspect that this tool caused the problem.

Re: Bpftune uses BPF to auto-tune Linux systems

#34
post #31

Earlier quoted context omitted.

It’s not a questions of being able to reverse. It’s a question of being able to diagnose that one of these changes even was the problem and if so which one.

If they can be reversed individually you can simply deduce by rolling back changes one by one, no?

Suppose you run a fleet of a thousand machines. They all autotune. They are, lets say, serving cached video, or something.

You notice that your aggregate error rate been drifting upwards since using bpftune. It turns out, in reality, there is some complex interaction between the tuning and your routers, or your TOR switches, or whatever - there is feedback that causes oscillations in a tuned value, swinging between too high and too low.

Can you see how this is not a matter of simple deduction and rollbacks?

This scenario is plausible. Autotuning generally has issues with feedback, since the overall system lacks control theoretic structure. And the premise here is that you use this to tune a large number of machines where individual admin is infeasible.

Re: Bpftune uses BPF to auto-tune Linux systems

#36
post #26

Earlier quoted context omitted.

Yes, it is. IMO, except for learning (which should not be done in prod), you shouldn’t make changes that you don’t understand. The tools seems to mostly tweak various networking settings. You could set up a test instance with monitoring, throw load at it, and change the parameters the tool modifies (one at a time!) to see how it reacts.

I'd run such a tool on prod in "advice mode". It should suggest the tweaks, explaining the reasoning behind them, and listing the actions necessary to implement them. Then humans would decide if they want to implement that as is, partly, modified, or not at all.

I agree. There's very little need to have a tool automatically changing complex kernel settings on the fly unless your infrastructure is undergoing dramatic changes in load and behavior on a daily basis, which seems unlikely for a modern server.

Re: Bpftune uses BPF to auto-tune Linux systems

#37

Earlier quoted context omitted.

>"bpftune logs to syslog so /var/log/messages will contain details of any tuning carried out." (from OP GitHub readme) The rmem example seems to allay fears that it will make changes one can't reverse.

It’s not a questions of being able to reverse. It’s a question of being able to diagnose that one of these changes even was the problem and if so which one.

I focused primarily on guesswho's "in ways I am unaware of".

Your issue appears to be true for any system change. Although, risk will of course vary.

Re: Bpftune uses BPF to auto-tune Linux systems

#38

It's great how it grew out of simple packet filtering into tracing and monitoring. It's one of those great tools most should know. Been using it for years.

The entire modern ML and most of the HPC grew out of very simple programmable shaders in GeForce 3.

Re: Bpftune uses BPF to auto-tune Linux systems

#39
post #31

Earlier quoted context omitted.

If they can be reversed individually you can simply deduce by rolling back changes one by one, no?

Suppose you run a fleet of a thousand machines. They all autotune. They are, lets say, serving cached video, or something. You notice that your aggregate error rate been drifting upwards since using bpftune. It turns out, in reality, there is some complex interaction between the tuning and your routers, or your TOR switches, or whatever - there is feedback that causes oscillations in a tuned value, swinging between t…

>not only can we observe the system and tune appropriately, we can also observe the effect of that tuning and re-tune if necessary. //

Does sound like a potential way to implement literal chaos.

Surely it's like anything else, you do pre-release testing and balance the benefits for you against the risks?

Re: Bpftune uses BPF to auto-tune Linux systems

#40
post #31

Earlier quoted context omitted.

If they can be reversed individually you can simply deduce by rolling back changes one by one, no?

Suppose you run a fleet of a thousand machines. They all autotune. They are, lets say, serving cached video, or something. You notice that your aggregate error rate been drifting upwards since using bpftune. It turns out, in reality, there is some complex interaction between the tuning and your routers, or your TOR switches, or whatever - there is feedback that causes oscillations in a tuned value, swinging between t…

Sounds like you have your answer of “don’t use it” then.
Post reply on HN