Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Netflix Accurately Attributes eBPF Flow Logs

netflixtechblog.com

Post reply on HN
Print view View original

11–20 of 61 posts

Re: How Netflix Accurately Attributes eBPF Flow Logs

#11

Post by temp0826 »

steve_adams_86 wrote: post #9
temp0826 wrote: post #6

Earlier quoted context omitted.

I'm not sure I know anyone that _doesn't_ have it

Like seneca, I also don’t know anyone who uses it. That’s interesting. I haven’t used it for around 2 years. I wonder who uses it when I see their reports, because I don’t know them. I probably have a weird group of friends. I’m sure if I asked, some of my coworkers use it. My kids would love if we used it. Perhaps it’s big among younger people.

Maybe that's part of it?...my sibling has a toddler and they use it for a lot of children's shows. Conversely, my mother watches it often too (not for kids shows!)

Re: How Netflix Accurately Attributes eBPF Flow Logs

#12

Post by yuters »

ASalazarMX wrote: post #4
butlike wrote: post #2

All that logging and they cant figure out why people are going to other streaming services

Despite their awful UX, I'm always impressed with how reliable their service is, technically speaking. Video is always good and responsive even on less-than-stellar connections, you can leave a show paused for hours, and resume it almost instantly. Their fast.com speed test is always much faster than your regular internet access, I guess thanks to their Open Connect Appliances. It must be great to work for them in in…

I have an old fire tv and never tried to stop automatic updates on it, it has become so slow and unresponsive that I'm barely able to switch inputs to use something else. Netflix is the only app that still works on that tv.

Re: How Netflix Accurately Attributes eBPF Flow Logs

#13

Post by nikolay_sivko »

At Coroot, we solve the same problem, but in a slightly different way. The traffic source is always a container (Kubernetes pod, systemd slice, etc.). The destination is initially identified as an IP:PORT pair, which, in the case of Kubernetes services, is often not the final destination. To address this, our agent also determines the actual destination by accessing the conntrack table at the eBPF level. Then, at the UI level, we match the actual destination with metadata about TCP listening sockets, effectively converting raw connections into container-to-container communications.

The agent repo: https://github.com/coroot/coroot-node-agent

Re: How Netflix Accurately Attributes eBPF Flow Logs

#18

Post by faitswulff »

ASalazarMX wrote: post #4
butlike wrote: post #2

All that logging and they cant figure out why people are going to other streaming services

Despite their awful UX, I'm always impressed with how reliable their service is, technically speaking. Video is always good and responsive even on less-than-stellar connections, you can leave a show paused for hours, and resume it almost instantly. Their fast.com speed test is always much faster than your regular internet access, I guess thanks to their Open Connect Appliances. It must be great to work for them in in…

I can only remember one major outage from them in the past ~10 years (in the 2020s, not the 2012 outage), and if I recall correctly, it was fixed in short order...and they never released a postmortem

Re: How Netflix Accurately Attributes eBPF Flow Logs

#19

Post by Hikikomori »

thewisenerd wrote: post #10

so they didn't want to pay for AWS CloudWatch [1]; decided to roll their in-house network flow log collection; and had to re-implement attribution? i wonder how many hundreds of thousands of dollars network flow logs cost them; obviously at some point it is going to be cheaper to re-implement monitoring in-house. [1]: https://youtu.be/8C9xNVYbCVk?feature=shared&t=1685

Because vanilla flowlogs that you get from VPC/TGW are nearly useless outside the most basic use cases. All you get is how many bytes and which tcp flags were seen per connection per 10 minutes. Then you need to attribute ip addresses to actual resources yourself separately, which isn't simple when you have containers or k8s service networking.

Doing it with eBPF on end hosts you can get the same data, but you can attribute it directly as you know which container it originates from, snoop dns, then you can get extremely useful metrics like per tcp connection ack delay and retransmissions, etc.

AWS recently released Cloudwatch Network Monitoring that also uses an agent with eBPF, but its almost like a children's toy compared to something like Datadog NPM. I was working on a solution similar to Netflix's when NPM was released, was no point after that.

Re: How Netflix Accurately Attributes eBPF Flow Logs

#20

Post by itishappy »

steve_adams_86 wrote: post #9
temp0826 wrote: post #6

Earlier quoted context omitted.

I'm not sure I know anyone that _doesn't_ have it

Like seneca, I also don’t know anyone who uses it. That’s interesting. I haven’t used it for around 2 years. I wonder who uses it when I see their reports, because I don’t know them. I probably have a weird group of friends. I’m sure if I asked, some of my coworkers use it. My kids would love if we used it. Perhaps it’s big among younger people.

Do you use something different? Anecdotally, I find Disney+ to be a major divider. Friends of mine have kids and Disney+ and/or they're still using a decade-old Netflix subscription.
Post reply on HN