731–740 of 986 posts
Every time a big company screws up, there are two highly informed sets of people who are guaranteed to be lurking, but rarely post, in a thread like this: 1) those directly involved with the incident, or employees of the same company. They have too much to lose by circumventing the PR machine. 2) people at similar companies who operate similar systems with similar scale and risks. Those people know how hard this is a…
Earlier quoted context omitted.
It's very hard to do a representative dry run when the most likely potential points of failure are highly load-dependent. You can try and predict everything that'll happen in production, but if you have nothing to extrapolate from, e.g. because this is your very first large live event, the chances of getting that right are almost zero. And you can't easily import that knowledge either, because your system might have…
They could have done a dry run. They could have spun up a million virtual machines somewhere, and tested their video delivery for 30 minutes. Even my small team spins up 10,000 EC2 instances on the regular. Netflix has the money to do much more. I'm sure there are a dozen ways they could have stress-tested this beforehand. It's not like someone sprang this on them last week and they had to scramble to put together a…
Every time a big company screws up, there are two highly informed sets of people who are guaranteed to be lurking, but rarely post, in a thread like this: 1) those directly involved with the incident, or employees of the same company. They have too much to lose by circumventing the PR machine. 2) people at similar companies who operate similar systems with similar scale and risks. Those people know how hard this is a…
If you code it to utilize high-bandwidth users upload, the service becomes more available as more users are watching -- not less available.
It becomes less expensive with scale, more available, more stable.
The be more specific, if you encode the video in blocks with each new block hash being broadcast across the network, just managing the overhead of the block order, it should be pretty easy to stream video with boundless scale using a DHT.
Could even give high-bandwidth users a credit based upon how much bandwidth they share.
With a network like what Netflix already has, the seed-boxes would guarantee stability. There would be very little delay for realtime streams, I'd imagine 5 seconds top. This sort of architecture would handle planet-scale streams for breakfast on top of the already existing mechanism.
But then again, I don't get paid $500k+ at a large corp to serve planet scale content, so what do I know.
Earlier quoted context omitted.
In-group signaling for people who like playing or thinking about the stock market. Similar to how people who make travel a big part of their identity refer to cities by their airport code.
Or maybe they are pilots?
"I flew into EGLL on Monday" - pilot
Earlier quoted context omitted.
Sorry for the off topic but what’s this thing that I only come across in Hacker News about referring to a company by their stock exchange name (APPL, MSFT, etc) outside of a stock context? It seems really weird to me.
I just assume it's people who spend too much time thinking about the stock context.
The way to deal with this is to constantly do live events, and actually build organizational muscle. Not these massive one off events in an area the tech team has no experience in.
A company I used to work for ran a few Super Bowl ads. The level of traffic you get during a Super Bowl ad is immense, and it all comes at you in 30 seconds, before going back to a steady-state value just as quickly. The scale pattern is like nothing else I've ever seen.
Super Bowl ads famously seven million dollars. These are things we simply can't repeat year over year, even if we believed it'd generate the same bump in recognition each time.
Earlier quoted context omitted.
Yeah, the funny part is that Hulu, Amazon Prime, and Peacock have all demonstrated the ability to handle an event of this caliber with no issue. Netflix now may never get another opportunity like this again.
AFAIK those three farm out to CDNs with tons of edges who know what they're doing. I have a feeling Netflix said 'how hard could this be?' and is finding out right now.
Every time a big company screws up, there are two highly informed sets of people who are guaranteed to be lurking, but rarely post, in a thread like this: 1) those directly involved with the incident, or employees of the same company. They have too much to lose by circumventing the PR machine. 2) people at similar companies who operate similar systems with similar scale and risks. Those people know how hard this is a…
For an event like this, there already exists an architecture that can handle boundless scale: torrents. If you code it to utilize high-bandwidth users upload, the service becomes more available as more users are watching -- not less available. It becomes less expensive with scale, more available, more stable. The be more specific, if you encode the video in blocks with each new block hash being broadcast across the n…
1. Everyone only cares about the most recent "block". By the time a "user" has fully downloaded a block from Netflix's seedbox, the block is stale, so why would any other user choose to download from a peer rather from netflix directly?
2. If all the users would prefer to download from netflix directly rather than a p2p user, then you already have a somewhat centralized solution, and you gain nothing from torrents.
This topic is really just fun for me to read based on where I work and my role. Live is a lot harder than on demand especially when you can't estimate demand (which I'm sure this was hard to do). People are definitely not understanding that. Then there is that Netflix is well regarded for their engineering not quite to the point of snobbery. What is actually interesting to me is that they went for an event like this…
Why is live a lot harder? Aside from latency (which isn't much of a problem unless you are competing with TV or some other distribution system), it seems easier than on-demand, since you send the same data to everyone and don't need to handle having a potentially huge library in all datacenters (you have to distribute the data, but that's just like having an extra few users per server). My guess is that the problem w…
With on-demand you can push the episodes out through your entire CDN at your leisure. It doesn't matter if some bottleneck means it takes 2 hours to distribute a 1 hour show worldwide, if you're distributing it the day before. And if you want to test, or find something that needs fixing? You've got plenty of time.
And on-demand viewers can trickle in gradually - so if clients have to contact your DRM servers for a new key every 15 minutes, they won't all be doing it at the same moment.
And if you did have a brief hiccup with your DRM servers - could you rely on the code quality of abandonware Smart TV clients to save you?