Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

No More Blue Fridays

brendangregg.com

Post reply on HN
Print view View original

171–180 of 282 posts

Re: No More Blue Fridays

#171

Post by School-Cotton »

javierhonduco wrote: post #142
School-Cotton wrote: post #120

Earlier quoted context omitted.

eBPF programs cannot crash the kernel, assuming there are no bugs in the eBPF verifier. There have been such bugs in the past but they seem to be getting more and more rare.

Or in other parts of the kernel. It's been the case in multiple occasions that buggy locking (or more generalised, missing 'resource' release) has caused problems for perfectly safe BPF programs. For example, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1033398 and the fix https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

This is actually exactly the bug I was thinking of, so fair point! (I work at PS now and am aware you worked on debugging it a while back).

Re: No More Blue Fridays

#172

Post by ec109685 »

kaliszad wrote: post #157

"These security agents will then be safe and unable to cause a Windows kernel crash." Unless of course there is a bug in eBPF ( https://access.redhat.com/solutions/7068083 ) @brendangregg and the kernel panics/ BSoDs anyway which you mention later in the article of course.

Benefit of fixing that bug is that all ebpf programs benefit versus every security vendor needing to ensure they write perfect c code.

Re: No More Blue Fridays

#173

Post by odyssey7 »

"The verifier is rigorous"

But the appeal-to-authority evidence that the article presents is not.

"-- the Linux implementation has over 20,000 lines of code -- with contributions from industry (e.g., Meta, Isovalent, Google) and academia (e.g., Rutgers University, University of Washington). The safety this provides is a key benefit of eBPF, along with heightened security and lower resource usage."

Re: No More Blue Fridays

#174

Post by riskable »

uticus wrote: post #121

> eBPF programs cannot crash the entire system because they are safety-checked by a software verifier and are effectively run in a sandbox. Isn’t one of the purposes of an OS to police software? I get that this has to do with the OS itself, but what does watching the watchers accomplish other than adding a layer which must then be watched? Why not reduce complexity instead of naively trusting that the new complexity…

eBPF isn't "watching the watchers" it's just a tool that lets other tools access low-level things in the kernel via a very picky sandbox. Think of it like this:

Old way: Load kernel driver, hook into bazillions of system calls (doing whatever it is you want to do), pray you don't screw anything up (otherwise you can get a panic though not necessarily--Linux is quite robust).

eBPF way: Just ask eBPF to tell you what you want by giving it some eBPF-specific instructions.

There's a rundown on how it works here: https://ebpf.io/what-is-ebpf/

Re: No More Blue Fridays

#175

Post by lazycog512 »

"The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at and repair."

- Douglas Adams

Re: No More Blue Fridays

#176

Post by riskable »

throw0101d wrote: post #159

Meta: > eBPF (no longer an acronym) […] Any reason why the official acronym was done away with?

Because it used to stand for extended Berkeley Packet Filter and it has since moved far, far beyond just packets. It now hooks into the entire network stack, security, and does observability/tracing for nearly anything and everything in the kernel ("nearly" because some stuff runs when the kernel boots up--before eBPF is loaded--and never again after that).

Re: No More Blue Fridays

#177

Post by ohmyiv »

kevin_nisbet wrote: post #52

I hate to dispute with someone like Brendan Gregg, but I'm hoping vendors in this space take a more holistic approach to investigating the complete failure chain. I personally tend to get cautious when there is a proposal that x will solve the problem that occurred on y date, especially 3 days after the failure. It may be true, but if we don't do the analysis we could leave ourselves open to blindspots. There may als…

> I personally tend to get cautious when there is a proposal that x will solve the problem that occurred on y date, especially 3 days after the failure.

Microsoft has been working on eBPF for a few years at least.

https://opensource.microsoft.com/blog/2021/05/10/making-ebpf...

https://lwn.net/Articles/857215/

If you're really concerned, they have discussions and communication channels where you're invited to air your concerns. They're listed on their github:

https://github.com/microsoft/ebpf-for-windows

Who knows, maybe they already have answers to your concerns. If not, they can address them there.

Re: No More Blue Fridays

#178

Post by numbsafari »

surajrmal wrote: post #73
brendangregg wrote: post #46

Earlier quoted context omitted.

Yes, we know eBPF must attach to equivalent events to Linux, but given there are already many event sources and consumers in Windows, the work is to make eBPF another consumer -- not to invent instrumentation frameworks from scratch. Just to use an analogy: Imagine people do their banking on JavaScript websites with Google Chrome, but if they use Microsoft Edge it says "JavaScript isn't supported, please download and…

This assumes eBPF becomes the standard. It's not clear Microsoft wants that. They could create something else which integrates with dot net and push for that instead. Also this problem of too much software running in the kernel in an unbounded manner has long existed. Why should Microsoft suddenly invest in solving it on Windows?

> Why should Microsoft suddenly invest in solving it on Windows?

If they can continue to avoid commercial repercussions for failing to provide a stable and secure system, then society should begin to hold them to account and force them to.

I’m not necessarily advocating for eBPF here, either. If they want to get there through some “proprietary” means, so be it. Apple is doing much the same on their end by locking down kexts and providing APIs for user mode system extensions instead. If MS wants to do this with some kind of .net-based solution (or some other fever dream out of MSR) then cool. The only caveat would seem to be that they are under a number of “consent decree” type agreements that would require that their own extensions be implemented on a level playing field.

So what. Windows Defender shouldn’t be in the kernel any more than CrowdStrike. Add an API. If that means being able to send eBPF type “programs” into kernel space, cool. If that means some user mode APIs, cool.

But lock it down already.

Re: No More Blue Fridays

#180

Post by whynotminot »

hello_moto wrote: post #149
whynotminot wrote: post #145

Earlier quoted context omitted.

Canary deployment doesn’t necessarily mean massive gaps between deployment waves. You can fast-follow. Sure, there may be scenarios with especially severe vulnerabilities where time is of the essence. I’m out of the loop if this crowdstrike update was such a scenario where best practices for software deployment were worth bypassing. If this is just how they roll with regular definition updates, then their deployment…

Let's walk this through: Canary deployment to Windows machines. If those Windows machines got hit with BSOD, they will go offline. How do you determine if they go offline because of Canary or because of regular maintenance by the customer's IT cycle? You can guess, but you cannot be 100% sure. What if the targeted canary deployments are Employees desktops that are OFFLINE during the time of rollout? >I’m out of the l…

> You can guess, but you cannot be 100% sure.

I worked in the cyber security space for a decent chunk of my career, and the most frustrating part was cyber security engineers thinking their problems were unique and being completely unaware of the lessons software engineering teams have already learned.

Yes, you need to tune your canary deployment groups to be large and diverse enough to give a reliable indicator of deployment failure, while still keeping them small enough that they achieve their purpose of limiting blast radius.

Again, if you follow industry best practices for software deployment, this is already something that should be considered. This is a relatively solved problem -- this is not new.

> I did post a question: what about other Cybersecurity vendors? Do you think they do canary deployment on their AV definitions?

I think that question is being asked right now by every company using Crowdstrike — what vendors are actually doing proper release engineering and how fast can we switch to them so that this never happens to us again?

Post reply on HN