Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson

sportingnews.com

Post reply on HN
Print view View original

371–380 of 986 posts

Re: Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson

#371

Post by krysp »

spike021 wrote: post #366
strken wrote: post #360

Earlier quoted context omitted.

I'm not a manager and I don't stack rank people, but I am 100% capable of knowing when one of my co-workers or predecessors is a fucking moron. The trick is to use my massive brain to root cause several significant outages, discover that most of them originate in code written by the same employee, and notice that said employee liked to write things like // @ts-nocheck // obligatory disabling of typescript: static typ…

Isn't that a problem with your code review process? Why is that person's code making it to production? So again, maybe they're a bad employee but it seems like nothing's done to even try and minimize the risks they present.

There's a disincentive to actively block PRs if you don't want your coworkers to think you are a bad colleague / not on their side. So you often see suboptimal code making its way to production. This has a worse effect the more terrible engineers there are.

Re: Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson

#372

Post by new_user_final »

I am curious about their live streaming infrastructure.

I have done live streaming for around 100k concurrent users. I didn't setup infrastructure because it was CloudFront CDN.

Why it is hard for Netflix. They have already figured out CDN part. So it should not be a problem even if it is 1M or 100M. because their CDN infrastructure is already handling the load.

I have only work with HLS live streaming where playlist is constantly changing compared to VOD. Live video chunks work same as VOD. CloudFront also has a feature request collapsing that greatly help live streaming.

So, my question is if Netflix has already figured out CDN, why their live infrastructure failing?

Note: I am not saying my 100k is same scaling as their 100M. I am curious about which part is the bottleneck.

Re: Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson

#373

Post by junior44660 »

This is probably a naive question but very relevant to what we have here.

In a protocol where a oft-repeated request goes through multiple intermediaries, usually every intermediate will be able to cache the response for common queries (Eg: DNS).

In theory, ISPs would be able to do the same with the HTTP. Although I am not aware of anyone doing such (since it will rightfully raise concerns of privacy and tampering).

Now TLS (or other encryption) will break this abstraction. Every user, even if they request a live stream, receives a differently encrypted response.

But live stream of a popular boxing match has nothing to do with the "confidentiality" of encryption protocol, only integrity.

Do we have a protocol which allows downstream intermediates eg ISPs to cache content of the stream based on demand, while a digital signature / other attestation being still cryptographically verified by the client?

Re: Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson

#374

Post by spike021 »

krysp wrote: post #371
spike021 wrote: post #366

Earlier quoted context omitted.

Isn't that a problem with your code review process? Why is that person's code making it to production? So again, maybe they're a bad employee but it seems like nothing's done to even try and minimize the risks they present.

There's a disincentive to actively block PRs if you don't want your coworkers to think you are a bad colleague / not on their side. So you often see suboptimal code making its way to production. This has a worse effect the more terrible engineers there are.

Except in this case it's clearly affecting at minimum the rest of OP's team.

At that point it's not one person being obnoxious and never approving their team members diffs and more of a team effort to do so.

But at minimum if you have a culture of trying to improve your codebase you'll inevitably set up tests, ci/cd with checks, etc. before any code can be deployed. Which should really take any weight of responsibility out of any one member of the team. Whether trying to put out code or reject said code.

Re: Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson

#375

Post by ilaksh »

I guarantee this is a management issue. Somebody needed to bear down at some point and put the resources into load testing. The engineers told them it probably won't be sufficient.

I assume this came down to some technical manager saying they didn't have the human and server resources for the project to work smoothly and a VP or something saying "well, just do the best you can.. surely it will be at least a little better than last time we tried something live, right?"

I think there should be a $20 million class action lawsuit, which should be settled as automatic refunds for everyone who streamed the fight. And two executives should get fired.

At least.. that's how it would be if there was any justice in the world. But we now know there isn't -- as evidenced by the fact that Jake Paul's head is still firmly attached to his body.

Re: Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson

#376

Post by TylerE »

scott_w wrote: post #328
TylerE wrote: post #251

Earlier quoted context omitted.

At least half of that is on you. NEVER work unpaid/unlogged OT.

Can’t speak for every place but that’s not always an option. As a teenager, I worked at Sports Direct where the management would regularly work us after our allotted hours and bar us putting the extra time onto our timesheet. If I recall correctly, the company eventually got pulled for it but the money they’d have saved over years would have outweighed the fine. The timesheets were on paper so good luck putting your…

Report that shit to your local Department of Labor equivalent. They would have gotten you, and everyone else in that store, their owed money.

Re: Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson

#377

Post by throwawaythekey »

normie3000 wrote: post #304
throwawaythekey wrote: post #286

Earlier quoted context omitted.

> If your "dumbest engineer" got a job and a hefty raise going to Netflix, it means he was very capable engineer who was playing the part of moron It's also possible that there's very little correlation between capability, reputation and salary. Don't we all know someone who is overpaid? There are more than a few well known cases of particular employers who select for overpaid employees...

> well known cases of particular employers who select for overpaid employees Not well-known enough, apparently. Where should I be applying?

There are different forms of overpayment but to give some examples:

- The recent story of AWS using serverless for video processing comes to mind [1].

- Google is renowned for rest and vest.

- Many government jobs pay more than their private counterparts.

- Military contractors

- Most of the healthcare industry

- Lobbyists

[1] https://news.ycombinator.com/item?id=35811741

Re: Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson

#378

Post by scott_w »

TylerE wrote: post #376
scott_w wrote: post #328

Earlier quoted context omitted.

Can’t speak for every place but that’s not always an option. As a teenager, I worked at Sports Direct where the management would regularly work us after our allotted hours and bar us putting the extra time onto our timesheet. If I recall correctly, the company eventually got pulled for it but the money they’d have saved over years would have outweighed the fine. The timesheets were on paper so good luck putting your…

Report that shit to your local Department of Labor equivalent. They would have gotten you, and everyone else in that store, their owed money.

You’re asking children to have full understanding of their rights and how to enforce them. Also, investigations into this started in 2020: over a decade after I left. Do you think nobody had reported this in all that time? Looks like the system wasn’t working as well as you think it does.

Re: Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson

#379

Post by Yeul »

freditup wrote: post #8

I wonder if there will be any long term reputational repercussions for Netflix because of this. Amongst SWEs, Netflix is known for hiring the best people and their streaming service normally seems very solid. Other streaming services have definitely caught up a bit and are much more reliable then in the early days, but my impression still has always been that Netflix is a step above the rest technically. This sure do…

There's a difference between live broadcasts and serving up content that's sitting on a server I guess?

In my country every time there's a big football match the people who try to watch it on the internet face issues.

Re: Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson

#380

Post by strken »

spike021 wrote: post #366
strken wrote: post #360

Earlier quoted context omitted.

I'm not a manager and I don't stack rank people, but I am 100% capable of knowing when one of my co-workers or predecessors is a fucking moron. The trick is to use my massive brain to root cause several significant outages, discover that most of them originate in code written by the same employee, and notice that said employee liked to write things like // @ts-nocheck // obligatory disabling of typescript: static typ…

Isn't that a problem with your code review process? Why is that person's code making it to production? So again, maybe they're a bad employee but it seems like nothing's done to even try and minimize the risks they present.

In this specific case, the fucking moron in question was the one who designed the code review process and hired the other engineers, and it took place a significant length of time before my involvement.

Which, yes, does raise interesting questions about how someone who can't be trusted to handle errors in an API call ended up in a senior enough position to do that.

Post reply on HN