They didn’t miss anything.
Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson
721–730 of 986 posts
Re: Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson
#722Earlier quoted context omitted.
Doing dry runs regularly makes sense, but whether actually shipping it makes sense seems context-dependent. It depends on how much you can minimize the side effects of shipping a release. Consider publishing a new version of a library: you'd be bumping the version number all the time and invalidating caches, causing downstream rebuilds, for little reason. Or if clients are lazy about updating, any two clients would b…
It's very hard to do a representative dry run when the most likely potential points of failure are highly load-dependent. You can try and predict everything that'll happen in production, but if you have nothing to extrapolate from, e.g. because this is your very first large live event, the chances of getting that right are almost zero. And you can't easily import that knowledge either, because your system might have…
They could have done a dry run. They could have spun up a million virtual machines somewhere, and tested their video delivery for 30 minutes. Even my small team spins up 10,000 EC2 instances on the regular. Netflix has the money to do much more. I'm sure there are a dozen ways they could have stress-tested this beforehand. It's not like someone sprang this on them last week and they had to scramble to put together a system to do it.
Re: Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson
#723Earlier quoted context omitted.
Do people even have cable TV anymore? I have internet from my "cable" company but I don't have the "cable" connected to anything but the modem. Everything I watch is streamed. The only thing connected to my TV is a Roku.
I assume you know the answer to your questions is: of course they do. However, I’m in the same boat as you. The joke’s on us, I guess.
I guess my question is: why? "Cable boxes" are uniformly awful to use in my experience. The UI is clunky, they take up space and it's another remote and another tangle of wires to try to hide. What advantage do they offer in 2024?
Re: Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson
#724This topic is really just fun for me to read based on where I work and my role. Live is a lot harder than on demand especially when you can't estimate demand (which I'm sure this was hard to do). People are definitely not understanding that. Then there is that Netflix is well regarded for their engineering not quite to the point of snobbery. What is actually interesting to me is that they went for an event like this…
> The NFL seems to be quite particular (in a good way) about the quality of the delivery of their content
Alas, my experience with the NFL in the UK does not reflect that. DAZN have the rights to stream NFL games here, and there are aspects of their service that are very poor. My major, long-standing issue has been the editing of their full game “ad-free” replays - it is common for chunks of play to be cut out, including touchdowns and field goals. Repeated complaints to DAZN haven’t resulted in any improvements. I can’t help but think that if the NFL was serious about the quality of their offering, they’d be knocking heads together at DAZN to fix this.
Re: Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson
#725People just do not appreciate how many gotchas can pop up doing anything live. Sure, Netflix might have a great CDN that works great for their canned content and I could see how they might have assumed that's the hardest part. Live has changed over the years from large satellite dishes beaming to a geosat and back down to the broadcast center($$$$$), to microwave to a more local broadcast center($$$$), to running ded…
You’re talking about the contribution from the venue to the boardcast centre, increasingly not a full program but being mixed remotely. That’s a very different area to transmission of live to end users.
What are you talking about? The signal coming from a live event is the full package. The output of “the truck” has multiple outs including the full mix of all grafix, some only have the mix minus any branding, etc. While the isos get recorded in the truck, they are not pushed out to the broadcast center.
All of the “mixing” as you call it is done in the truck. If you’ve never seen it, it is quite impressive. In one part of the truck is the director and the technical director. The director is the one calling things like “ready camera 1”, “take 1”, etc. the TD is the one on the switcher pushing the actual buttons on the console to make it happen. Next to them is the graphics team prepping all of the stats made available to the TD to key in. In another area is the team of slomo/replay that are taking the feeds from all of the cameras to recorders that allow the operators to pull out the selects and make available for the director/TD to cut to. Typically in the back of the truck is the audio mixer that mixes all of the mics around the event in real time. All of that creates the signal you see on your screen. It leaves the back of the truck and heads out to wherever the broadcaster has better control
Re: Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson
#726It's incomprehensible to me that Netflix, one of the most highly skilled engineering teams in the world - completely sh*t the bed last night and provided a nearly unwatchable experience that was not even in the same league as pre-internet live broadcast from 30 years ago.
What’s amazing is that they have had several streaming flips before and are unable to fix it
Re: Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson
#727This topic is really just fun for me to read based on where I work and my role. Live is a lot harder than on demand especially when you can't estimate demand (which I'm sure this was hard to do). People are definitely not understanding that. Then there is that Netflix is well regarded for their engineering not quite to the point of snobbery. What is actually interesting to me is that they went for an event like this…
Netflix’s snobbery of engineering is so exhausting. Then seeing them be unable to fix this problem after several previous streaming failures is a bit rich.
To me it speaks to how most of the top tech companies of the 2010s have degraded as of late. I see it all the time with Google hiring some of the lower performing engineers on my teams because they crushed Leetcode.
Re: Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson
#728Every time a big company screws up, there are two highly informed sets of people who are guaranteed to be lurking, but rarely post, in a thread like this: 1) those directly involved with the incident, or employees of the same company. They have too much to lose by circumventing the PR machine. 2) people at similar companies who operate similar systems with similar scale and risks. Those people know how hard this is a…
The only time I worked on a project that had a live television launch, it absolutely tipped over within like 2 minutes, and people on HN and Reddit were making fun of it. And I know how hard everyone worked, and how competent they were, so I sympathize with the people in these cases. While the internet was teeing off with easy jokes, engineers were swarming on a problem that was just not resolving, PMs were pacing up…
> people were getting yelled at by leadership
this is where you get up and leave
Re: Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson
#729Earlier quoted context omitted.
Right? A common complaint by outsiders is that Netflix uses microservices. I'd love to hear exactly how a monolith application is guaranteed to perform better, with details . What is the magic difference that would have ensured the live stream would have been successful?
It's not guaranteed, but much fewer points of failure.
> It's not guaranteed, but much fewer points of failure.
Can you explain where this is relevant to buffering issues?
Also, you are very wrong regarding failure modes. The larger the service, the more failure modes it has. Moreover, in monoliths if a failure mode can take down/degrade the whole service, all other features are taken down/degraded. Is having a single failure mode that brings down the whole service what you call fewer points of failure?
Re: Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson
#730This topic is really just fun for me to read based on where I work and my role. Live is a lot harder than on demand especially when you can't estimate demand (which I'm sure this was hard to do). People are definitely not understanding that. Then there is that Netflix is well regarded for their engineering not quite to the point of snobbery. What is actually interesting to me is that they went for an event like this…
Why is live a lot harder?
Aside from latency (which isn't much of a problem unless you are competing with TV or some other distribution system), it seems easier than on-demand, since you send the same data to everyone and don't need to handle having a potentially huge library in all datacenters (you have to distribute the data, but that's just like having an extra few users per server).
My guess is that the problem was simply that the number of people viewing Netflix at once in the US was much larger than usual and higher than what they could scale too, or alternatively a software bug was triggered.