Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Netflix Accurately Attributes eBPF Flow Logs

netflixtechblog.com

Post reply on HN
Print view View original

41–50 of 61 posts

Re: How Netflix Accurately Attributes eBPF Flow Logs

#41

Post by r3trohack3r »

meltyness wrote: post #40

I wonder how much of Netflix infra is on AWS. Feels like building a castle on someone else's kingdom at that scale; in light of the Prime Video investment, and I guess twitch too.

Netflix serves nearly all of its video from a server down the street from you via its OpenConnect infrastructure. AWS only hosts its microservice graph that does stuff like determining which videos and qualities you should be offered.

That being said, its core product has been nearly comoditized. When Netflix entered the market, delivering long form high quality video over the public internet was nascent. Now everyone and their grandma can spin up a video streaming service from a vendor.

Re: How Netflix Accurately Attributes eBPF Flow Logs

#42

Post by meltyness »

r3trohack3r wrote: post #41
meltyness wrote: post #40

I wonder how much of Netflix infra is on AWS. Feels like building a castle on someone else's kingdom at that scale; in light of the Prime Video investment, and I guess twitch too.

Netflix serves nearly all of its video from a server down the street from you via its OpenConnect infrastructure. AWS only hosts its microservice graph that does stuff like determining which videos and qualities you should be offered. That being said, its core product has been nearly comoditized. When Netflix entered the market, delivering long form high quality video over the public internet was nascent. Now everyon…

I was aware of this, I think a talk about a performance regression on the BSD variant these appliances run was up here recently.

I mean I guess a similar argument holds for colocating with telecoms that would have recently been cable or IPTV providers.

It'd be pretty tough to design around though if any of their caching infrastructure reveals viewership, or engagement data to intermediaries.

Re: How Netflix Accurately Attributes eBPF Flow Logs

#43

Post by jandrese »

nimbius wrote: post #17

i refuse to believe a company that wasted $320 million dollars on "the electric state" could ever manage to do anything correctly. stripe the parking lot? stock the breakroom? clean the toilets? simply not possible.

The scriptwriters aren't managing the network flows across the backend of their infrastructure. Netflix's trouble with scripts doesn't affect their ability to move bits around.

Re: How Netflix Accurately Attributes eBPF Flow Logs

#44

Post by ilrwbwrkhv »

meltyness wrote: post #40

I wonder how much of Netflix infra is on AWS. Feels like building a castle on someone else's kingdom at that scale; in light of the Prime Video investment, and I guess twitch too.

And more importantly none of this is required. Pornhub pushes more video data on far more unreliable connections without any of this madness. This is purely a play to get the tech company multiple. Nothing else. This is Wework style coverup.

Re: How Netflix Accurately Attributes eBPF Flow Logs

#45

Post by __turbobrew__ »

VaiTheJice wrote: post #38
__turbobrew__ wrote: post #34

Maybe Im missing something but can’t you run workloads in separate network namespaces and then attach a bpf probe to the veth interface in the namespace? At that point you know all flows on that veth are from a specific workload as long as you keep track of what is running in which network namespaces? I wonder if it is possible with ipv6 to never (or you roll through the addresses so reuse is temporally distant) re u…

I think thats pretty reasonable tbf and probably at a more 'simpler' scale and i use simple loosely because Netflix’s container runtime is Titus, which is more bare metal oriented than, say, Kubernetes. It doesn’t always isolate workloads as cleanly in separate netns per container, especially for network optimisation purposes like IPv6-to-IPv4 sharing. "I wonder if it is possible with ipv6 to never... re use addresse…

Makes sense, I have worked in and around CNI stuff for k8s and generally netns+veth is how most of them work. That being said we run k8s on bare metal, there isn’t any reason why running things on bare metal excludes netns usage.

> Most VPCs (also AWS) don’t currently support "true" IPv6 scaleout behavior.

Thats a shame.

> if IPs were truly immutable and unique per workload, attribution becomes trivial

I would like to see that. IPAM for multi-tenant workloads always felt like a kludge. You need the network to understand how to route to a workloads, but the network when running on ipv4 has many more workloads than addresses. If you assign immutable addresses per workload (or say it takes you a month to chew through your ipv6 address space) it makes it so the network natively knows how to route to workloads without the need to kludge with IP reassignments.

I have had to deal with IP address pools being exhausted due to high pod churn in EC2 a number of times and it is always a pain.

Re: How Netflix Accurately Attributes eBPF Flow Logs

#46

Post by VaiTheJice »

__turbobrew__ wrote: post #45
VaiTheJice wrote: post #38

Earlier quoted context omitted.

I think thats pretty reasonable tbf and probably at a more 'simpler' scale and i use simple loosely because Netflix’s container runtime is Titus, which is more bare metal oriented than, say, Kubernetes. It doesn’t always isolate workloads as cleanly in separate netns per container, especially for network optimisation purposes like IPv6-to-IPv4 sharing. "I wonder if it is possible with ipv6 to never... re use addresse…

Makes sense, I have worked in and around CNI stuff for k8s and generally netns+veth is how most of them work. That being said we run k8s on bare metal, there isn’t any reason why running things on bare metal excludes netns usage. > Most VPCs (also AWS) don’t currently support "true" IPv6 scaleout behavior. Thats a shame. > if IPs were truly immutable and unique per workload, attribution becomes trivial I would like t…

Ahh! Nothing like watching pods fail to schedule because you ran out of assignable IPs in a subnet you thought was generous.

Immutable addressing per workload with IPv6 feels like such a clean mental model, especially for attribution, routing, and observability.

Curious if you have seen anyone pull that off cleanly in production, like truly immutable addressing at scale? Curious if it’s been battle tested somewhere or still mostly an ideal.

Re: How Netflix Accurately Attributes eBPF Flow Logs

#47

Post by rpmisms »

ilrwbwrkhv wrote: post #44
meltyness wrote: post #40

I wonder how much of Netflix infra is on AWS. Feels like building a castle on someone else's kingdom at that scale; in light of the Prime Video investment, and I guess twitch too.

And more importantly none of this is required. Pornhub pushes more video data on far more unreliable connections without any of this madness. This is purely a play to get the tech company multiple. Nothing else. This is Wework style coverup.

Back to basics.

Re: How Netflix Accurately Attributes eBPF Flow Logs

#48

Post by __turbobrew__ »

VaiTheJice wrote: post #46
__turbobrew__ wrote: post #45

Earlier quoted context omitted.

Makes sense, I have worked in and around CNI stuff for k8s and generally netns+veth is how most of them work. That being said we run k8s on bare metal, there isn’t any reason why running things on bare metal excludes netns usage. > Most VPCs (also AWS) don’t currently support "true" IPv6 scaleout behavior. Thats a shame. > if IPs were truly immutable and unique per workload, attribution becomes trivial I would like t…

Ahh! Nothing like watching pods fail to schedule because you ran out of assignable IPs in a subnet you thought was generous. Immutable addressing per workload with IPv6 feels like such a clean mental model, especially for attribution, routing, and observability. Curious if you have seen anyone pull that off cleanly in production, like truly immutable addressing at scale? Curious if it’s been battle tested somewhere o…

Unfortunately my place is still stuck on ipv4.

Hypothetically it is not hard, you split your ipv6 prefix per datacenter. Then you use etcd to coordinate access to the ipv6 pool to hand out immutable addresses. You just start from the lowest address and go to the highest address. If you get to the highest address you go back to the lowest address, as long as your churn is not too high and your pool is big enough you should only wrap addresses far enough apart in time that address reuse doesn't cause any problems with false attribution.

In the etcd store you can just store KV pairs of ipv6 -> workload ID. If you really want to be fancy you can watch those KV pairs using etcd clients and get live updates of new addresses being assigned to workloads. You can plug these updates into your system of choice which needs to map ipv6 to workload such as network flow tools.

Unless you are doing something insane, you should easily be able to keep up with immutable address requests with a DC local etcd quorum.

Re: How Netflix Accurately Attributes eBPF Flow Logs

#49

Post by adrianN »

ilrwbwrkhv wrote: post #44
meltyness wrote: post #40

I wonder how much of Netflix infra is on AWS. Feels like building a castle on someone else's kingdom at that scale; in light of the Prime Video investment, and I guess twitch too.

And more importantly none of this is required. Pornhub pushes more video data on far more unreliable connections without any of this madness. This is purely a play to get the tech company multiple. Nothing else. This is Wework style coverup.

People might be more tolerant to quality issues for pornhub‘s content than Netflix‘s.

Re: How Netflix Accurately Attributes eBPF Flow Logs

#50

Post by madduci »

slt2021 wrote: post #23

Question to the Netflix folks: I saw a lot of in-house developed tools being quoted, do you guys have service mesh like linkerd ? Have you guys evaluated vendors like Kentik? I would love to get more insight into what do you guys actually do with flow logs? for example if I store 1 TB of flow logs, what value can I actually derive from them that justify the cost of collection, processing, and storage.

Exactly my thought. Maybe it's the "not invented here" syndrome?

We use Istio as Service Mesh and get the same result, using the same architecture as shown in the blog post (especially the part where each workload has a sidecar container running Flow).

Post reply on HN