Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bpftune uses BPF to auto-tune Linux systems

github.com

Post reply on HN
Print view View original

51–60 of 78 posts

Re: Bpftune uses BPF to auto-tune Linux systems

#51

Post by toast0 »

spenczar5 wrote: post #34
nehal3m wrote: post #31

Earlier quoted context omitted.

If they can be reversed individually you can simply deduce by rolling back changes one by one, no?

Suppose you run a fleet of a thousand machines. They all autotune. They are, lets say, serving cached video, or something. You notice that your aggregate error rate been drifting upwards since using bpftune. It turns out, in reality, there is some complex interaction between the tuning and your routers, or your TOR switches, or whatever - there is feedback that causes oscillations in a tuned value, swinging between t…

When you have a thousand machines, you can usually get feedback pretty quick, in my experience.

Run the tune on one machine. Looks good? Put it on ten. Looks good? Put it on one hundred. Looks good? Put it on everyone.

Find an issue a week later, and want to dig into it? Run 100 machines back on the old tune, and 100 machines with half the difference. See what happens.

Re: Bpftune uses BPF to auto-tune Linux systems

#52

Post by toast0 »

robinhoodexe wrote: post #6

Is tuning the TCP buffer size for instance worth it?

In my experience running big servers, tuning TCP buffers is definitely worth it, because different kinds of servers have different needs. It doesn't often work miracles, but tuning buffers is low cost, so the potential for a small positive impact is often worth the time to try.

If your servers communicate at high datarates with a handful of other servers, some of which are far away, but all of which have large potential throughput, you want big buffers. Big buffers allow you to have a large amount of data in flight to remote systems, which lets you maintain throughput regardless of where your servers are. You'd know to look at making buffers bigger if your throughput to far away servers is poor.

If you're providing large numbers of large downloads to public clients that are worldwide from servers in the US only, you probably want smaller buffers. Larger buffers would help with throughput to far away clients, but slow, far away clients will use a lot of buffer space and limit your concurrency. Clients that disappear mid download will tie up buffers until the connection is torn down and it's nice if that's less memory for each instance. You'd know to look at making buffers smaller if you're using more memory than you think is appropriate for network buffers... a prereq is monitoring memory use by type.

If you're serving dynamic web pages, you want your tcp buffers to be at least as big as your largest page, so that your dynamic generation never has to block for a slow client. You'd know to look at this if you see a lot of servers blocked on sending to clients, and/or if you see divergent server measured response times for things that should be consistent. This is one case where getting buffer sizes right can enable miracles; Apache pre-fork+mod_PHP can scale amazingly well or amazingly poorly; it scales well when you can use an accept filter so apache doesn't get a socket until the request is ready to be read, and PHP/apache can send the whole response to the tcp buffer without waiting, then closes the socket; letting the kernel deal with it from there. Keep-alive and TLS make this a bit harder, but the basic idea of having enough room to buffer the whole page still fits.

Re: Bpftune uses BPF to auto-tune Linux systems

#53

Post by sgarland »

zymhan wrote: post #48
sgarland wrote: post #25

Earlier quoted context omitted.

I think it’s still important to know what those dials and knobs do, otherwise (as the currently top-voted comment says) when things break, you’ll be lost.

That's exactly why it's such a burden.

I continue to be amazed and frustrated that people will simultaneously believe that deeply understanding a programming language is a noble pursuit, but that deeply understanding the software that allows their code to run is somehow burdensome.

Ops remains extremely important and extremely real. You can abstract it away in exchange for higher costs and less control, but ultimately someone at some level has studied these parameters, and decided what is best for your code.

Re: Bpftune uses BPF to auto-tune Linux systems

#54

Post by gerdesj »

gausswho wrote: post #18

With this tool I am wary that I'll encounter system issues that are dramatically more difficult to diagnose and troubleshoot because I'll have drifted from a standard distro configuration. And in ways I'm unaware of. Is this a reasonable hesitation?

"because I'll have drifted from a standard distro configuration"

You will obviously have a change management system which describes all the changes you have made to your putative standard distro configs. You will also be monitoring those changes.

This tool logs all the changes it makes via the standard logging system, which can be easily captured, shipped and aggregated and then queried and reported on.

This is not a tool from Clown Cars R US, it's from a reasonably reputable source - Oracle (lol etc). Even better, you can read the code and learn or critique.

Not being funny but I'd rather this sort of thing by far than any amount of wooo handwavy wankery. Would you prefer openly described and documented or "take our word for it"?

Re: Bpftune uses BPF to auto-tune Linux systems

#55

Post by gerdesj »

6SixTy wrote: post #35

As an insane Gentoo user, sign me the F*** up

You are not insane. You are running a distro that enables you to patch both the distro package definitions and the source code that is compiled by those definitions. Then there is the day to day stuff such as USE ...

I have a VM that was rather unloved that I ended up using git to go back in time to gradually move it forwards in time and update. It took quite a while. It started off life in the noughties and now sports a 6.10 kernel. I won't bore you further but it did take a while but the data is still there and accessible on a modern platform.

What is mad about that?

Re: Bpftune uses BPF to auto-tune Linux systems

#56

Post by cortesoft »

gerdesj wrote: post #54
gausswho wrote: post #18

With this tool I am wary that I'll encounter system issues that are dramatically more difficult to diagnose and troubleshoot because I'll have drifted from a standard distro configuration. And in ways I'm unaware of. Is this a reasonable hesitation?

"because I'll have drifted from a standard distro configuration" You will obviously have a change management system which describes all the changes you have made to your putative standard distro configs. You will also be monitoring those changes. This tool logs all the changes it makes via the standard logging system, which can be easily captured, shipped and aggregated and then queried and reported on. This is not a…

> You will obviously have a change management system which describes all the changes you have made to your putative standard distro configs. You will also be monitoring those changes.

Which is now a list you will have to check for every issue. I don't think they are complaining they don't trust the writers of the code, just that it adds confounding variables to your system

Re: Bpftune uses BPF to auto-tune Linux systems

#57

Post by gerdesj »

cortesoft wrote: post #56
gerdesj wrote: post #54

Earlier quoted context omitted.

"because I'll have drifted from a standard distro configuration" You will obviously have a change management system which describes all the changes you have made to your putative standard distro configs. You will also be monitoring those changes. This tool logs all the changes it makes via the standard logging system, which can be easily captured, shipped and aggregated and then queried and reported on. This is not a…

> You will obviously have a change management system which describes all the changes you have made to your putative standard distro configs. You will also be monitoring those changes. Which is now a list you will have to check for every issue. I don't think they are complaining they don't trust the writers of the code, just that it adds confounding variables to your system

We (in IT security) are expected to abrogate responsibility to funky AI or whatevs anti virus and other stuff. Buy and install a security package from ... whoever ... and all will be well.

This is an expert system/advice run by real people (at a reasonably well respected firm) not an AI wankery thingie. It is literally expert advice and it is being given away and in code form which you can read.

What on earth is wrong with that?

Re: Bpftune uses BPF to auto-tune Linux systems

#58

Post by cortesoft »

gerdesj wrote: post #57
cortesoft wrote: post #56

Earlier quoted context omitted.

> You will obviously have a change management system which describes all the changes you have made to your putative standard distro configs. You will also be monitoring those changes. Which is now a list you will have to check for every issue. I don't think they are complaining they don't trust the writers of the code, just that it adds confounding variables to your system

We (in IT security) are expected to abrogate responsibility to funky AI or whatevs anti virus and other stuff. Buy and install a security package from ... whoever ... and all will be well. This is an expert system/advice run by real people (at a reasonably well respected firm) not an AI wankery thingie. It is literally expert advice and it is being given away and in code form which you can read. What on earth is wron…

If the alternative is those proprietary anti virus products, sure this is better.

The original comment was comparing to doing nothing and just using the standard distro, I believe.

Re: Bpftune uses BPF to auto-tune Linux systems

#59

Post by fragmede »

cortesoft wrote: post #56
gerdesj wrote: post #54

Earlier quoted context omitted.

"because I'll have drifted from a standard distro configuration" You will obviously have a change management system which describes all the changes you have made to your putative standard distro configs. You will also be monitoring those changes. This tool logs all the changes it makes via the standard logging system, which can be easily captured, shipped and aggregated and then queried and reported on. This is not a…

> You will obviously have a change management system which describes all the changes you have made to your putative standard distro configs. You will also be monitoring those changes. Which is now a list you will have to check for every issue. I don't think they are complaining they don't trust the writers of the code, just that it adds confounding variables to your system

> for every issue.

Only if you don't know what you're doing, which, with no judgement whatsoever, might be true for OP. Reading the source, it affects some networking related flags. If the local audio craps out, it's not related. If the Bluetooth keyboard craps out, it's not related. If the hard drive crashes, it's not related.

I get that is just adding more variables to the system, but this isn't Windows, where the changes under the hood are this mystery hotfix that got applied and we have no idea what it did and the vendor notes raise more questions than it asks and your computer working feels like this house of cards that's gonna fall over if you look at it funny. If the system is acting funny, just disable this, reset them all back to default, possibly by rebooting, and see if the problem persists. If you're technical enough to install this, I don't think disabling it and rebooting is beyond your abilities.

Post reply on HN