Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FIFA 2014 World Cup live stream architecture

leandromoreira.com.br

Post reply on HN
Print view View original

11–20 of 30 posts

Re: FIFA 2014 World Cup live stream architecture

#11

Post by flavioribeiro »

jbochi wrote: post #10
jsaxton86 wrote: post #9

Interesting post! The author mentions that the first step was to "receive this input and to generate HLS output to a known folder". I have a couple questions about this step: 1: Did they re-encode the input? Or just repackage it? 2: I'm unfamiliar with EvoStream, but if it is ingesting RTMP and outputting HLS, why did Globo need to bother generating an HLS manifest? Couldn't they just use the one EvoStream created?

I am glad you liked. I am not the author of the blog post, but I was a member of the team. 1: We just repackage it. 2: The problem is that EvoStream stores the manifests and chunks locally and we needed high availability. That's why we use an external data storage. We have had up to 30 simultaneous streams, with 7 bitrates each and 2 hours of DVR.

Just adding one extra information, building the playlist on our side enabled us to define the DVR window (how many time one can seek back on the video). Since we built both the server and client side, it was also possible to add tags to the stream and we did it. tl;dr; It was possible to control the adaptive streaming on the server side.

When our CDN was very crowded, one tag on the HLS playlist was able to direct users to a lower quality, preventing all the users to fight for the same bandwidth and avoiding rebuffering events.

Re: FIFA 2014 World Cup live stream architecture

#15

Post by emilsedgh »

Just like everyone else, I really liked the article. And of course I have some questions.

What is the benefit of EvoStream over nginx-rtmp? I use nginx-rtmp as well and it creates HLS files just fine.

Also, how did you handle the network load? How many edge servers were actually delivering the content to users?

Did you use 10G ethernets, link bonding, or did you use cloud services (eg cloudfront) to deliver the content?

Thanks a lot.

Re: FIFA 2014 World Cup live stream architecture

#16

Post by Finster »

It feels like they reinvented the wheel on a lot of things (especially on the monitoring side). I also wish there was some more detail about the caching/endpoint side of things. We rely heavily on Akamai's services for their edge servers, caching, and site acceleration. I don't see anything like that here, but maybe they're just not mentioning it?

Re: FIFA 2014 World Cup live stream architecture

#18

Post by alexkappa »

Thanks for sharing this! How long was your team given to complete this project?

  Cassandra response time was increasing with load to a certain point where clients started to timeout and the video playback completely stopped.
and

  After these changes, we were able to achieve a latency in the order of 10ms for our 99% percentile.
How did you test this? Are there any benchmarking/load testing tools for cases such as streaming?

Re: FIFA 2014 World Cup live stream architecture

#20

Post by dreampeppers99 »

emilsedgh wrote: post #15

Just like everyone else, I really liked the article. And of course I have some questions. What is the benefit of EvoStream over nginx-rtmp? I use nginx-rtmp as well and it creates HLS files just fine. Also, how did you handle the network load? How many edge servers were actually delivering the content to users? Did you use 10G ethernets, link bonding, or did you use cloud services (eg cloudfront) to deliver the conte…

* We initially picked EMS because of its API, but we're moving to nginx-rtmp.

* We used 20G ethernet, we could get 19G from each machine.

* All the content is hosted by us.

Post reply on HN