Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Netflix Accurately Attributes eBPF Flow Logs

netflixtechblog.com

Post reply on HN
Print view View original

31–40 of 61 posts

Re: How Netflix Accurately Attributes eBPF Flow Logs

#31

Post by acdha »

ciupicri wrote: post #28
yuters wrote: post #12

Earlier quoted context omitted.

I have an old fire tv and never tried to stop automatic updates on it, it has become so slow and unresponsive that I'm barely able to switch inputs to use something else. Netflix is the only app that still works on that tv.

I also have an old TV and guess what? Netflix stopped working last year. The application is not supported anymore. Beats me why.

Often it’s CA certificates expiring. My old Toshiba had app rot set in like that, where after about 5 years none of the built in apps worked any more and the errors appeared to be TLS related. I suspect that was due to pinned certs to prevent MITM pirating.

Re: How Netflix Accurately Attributes eBPF Flow Logs

#32

Post by toomuchtodo »

ASalazarMX wrote: post #4
butlike wrote: post #2

All that logging and they cant figure out why people are going to other streaming services

Despite their awful UX, I'm always impressed with how reliable their service is, technically speaking. Video is always good and responsive even on less-than-stellar connections, you can leave a show paused for hours, and resume it almost instantly. Their fast.com speed test is always much faster than your regular internet access, I guess thanks to their Open Connect Appliances. It must be great to work for them in in…

They also performed work to ensure it performed well for Starlink customers.

A Global Perspective on the Past, Present, and Future of Video Streaming over Starlink - https://dl.acm.org/doi/10.1145/3700412 | https://doi.org/10.1145/3700412

Re: How Netflix Accurately Attributes eBPF Flow Logs

#33

Post by slt2021 »

retiredpapaya wrote: post #30
slt2021 wrote: post #23

Question to the Netflix folks: I saw a lot of in-house developed tools being quoted, do you guys have service mesh like linkerd ? Have you guys evaluated vendors like Kentik? I would love to get more insight into what do you guys actually do with flow logs? for example if I store 1 TB of flow logs, what value can I actually derive from them that justify the cost of collection, processing, and storage.

I think Netflix does use an Envoy-based Service Mesh [1], and they roll their own control plane. https://netflixtechblog.com/zero-configuration-service-mesh-...

If the goal of gathering and attributing VPC flows is to have a workload granularity flow logs, then imho gathering mesh level logs is more direct and atraight forward approach, because mesh(and workload orchestrator) are uniquely qualified to know when workload A is running on a host X and is trying to connect to workload B.

Looking at Envoy access logs for example is more straightforward and simple aplroach, than running distributed ebpf and memory intensive large spark streaming job

Re: How Netflix Accurately Attributes eBPF Flow Logs

#34

Post by __turbobrew__ »

Maybe Im missing something but can’t you run workloads in separate network namespaces and then attach a bpf probe to the veth interface in the namespace? At that point you know all flows on that veth are from a specific workload as long as you keep track of what is running in which network namespaces?

I wonder if it is possible with ipv6 to never (or you roll through the addresses so reuse is temporally distant) re use addresses which removes the problems with staleness and false attribution.

Re: How Netflix Accurately Attributes eBPF Flow Logs

#35

Post by ndriscoll »

ASalazarMX wrote: post #4
butlike wrote: post #2

All that logging and they cant figure out why people are going to other streaming services

Despite their awful UX, I'm always impressed with how reliable their service is, technically speaking. Video is always good and responsive even on less-than-stellar connections, you can leave a show paused for hours, and resume it almost instantly. Their fast.com speed test is always much faster than your regular internet access, I guess thanks to their Open Connect Appliances. It must be great to work for them in in…

Not that this detracts from the wider point, but I'd expect unpause to just work unless you go out of your way to make it not work. Even if you drop the connection at some point, afaik they use ~15 Mb/s as their "premium" bitrate, so e.g. a 30 s buffer takes less than 64 MB. That gives plenty of time to re-establish streaming after an unpause. It's not like the computer forgets what it was doing if you leave it alone.

Re: How Netflix Accurately Attributes eBPF Flow Logs

#36

Post by nptr »

Hikikomori wrote: post #19
thewisenerd wrote: post #10

so they didn't want to pay for AWS CloudWatch [1]; decided to roll their in-house network flow log collection; and had to re-implement attribution? i wonder how many hundreds of thousands of dollars network flow logs cost them; obviously at some point it is going to be cheaper to re-implement monitoring in-house. [1]: https://youtu.be/8C9xNVYbCVk?feature=shared&t=1685

Because vanilla flowlogs that you get from VPC/TGW are nearly useless outside the most basic use cases. All you get is how many bytes and which tcp flags were seen per connection per 10 minutes. Then you need to attribute ip addresses to actual resources yourself separately, which isn't simple when you have containers or k8s service networking. Doing it with eBPF on end hosts you can get the same data, but you can at…

This is spot on. The AWS logs can also be orders of magnitude more expensive.

Re: How Netflix Accurately Attributes eBPF Flow Logs

#37

Post by nptr »

slt2021 wrote: post #33
retiredpapaya wrote: post #30

Earlier quoted context omitted.

I think Netflix does use an Envoy-based Service Mesh [1], and they roll their own control plane. https://netflixtechblog.com/zero-configuration-service-mesh-...

If the goal of gathering and attributing VPC flows is to have a workload granularity flow logs, then imho gathering mesh level logs is more direct and atraight forward approach, because mesh(and workload orchestrator) are uniquely qualified to know when workload A is running on a host X and is trying to connect to workload B. Looking at Envoy access logs for example is more straightforward and simple aplroach, than r…

The blog post mentioned that "The eBPF flow logs provide a comprehensive view of service topology and network health across Netflix’s extensive microservices fleet, regardless of the programming language, RPC mechanism, or application-layer protocol used by individual workloads."

Service mesh may have restrictions on the network protocols and may not cover all network traffic (like connections to Kafka and databases).

Re: How Netflix Accurately Attributes eBPF Flow Logs

#38

Post by VaiTheJice »

__turbobrew__ wrote: post #34

Maybe Im missing something but can’t you run workloads in separate network namespaces and then attach a bpf probe to the veth interface in the namespace? At that point you know all flows on that veth are from a specific workload as long as you keep track of what is running in which network namespaces? I wonder if it is possible with ipv6 to never (or you roll through the addresses so reuse is temporally distant) re u…

I think thats pretty reasonable tbf and probably at a more 'simpler' scale and i use simple loosely because Netflix’s container runtime is Titus, which is more bare metal oriented than, say, Kubernetes. It doesn’t always isolate workloads as cleanly in separate netns per container, especially for network optimisation purposes like IPv6-to-IPv4 sharing.

"I wonder if it is possible with ipv6 to never... re use addresses which removes the problems with staleness and false attribution."

Most VPCs (also AWS) don’t currently support "true" IPv6 scaleout behavior. Buttt!! if IPs were truly immutable and unique per workload, attribution becomes trivial. It’s just not yet realistic... maybe something to explore with the lads?

Post reply on HN