Wonderful job, guys. Will you broadcast Olympics? What will change for this one?
Yes, for sure. :) Probably, there will be some changes but we're going to discovery some of them ongoing.
FIFA 2014 World Cup live stream architecture
21–30 of 30 posts
Re: FIFA 2014 World Cup live stream architecture
#22Thanks for sharing this! How long was your team given to complete this project? Cassandra response time was increasing with load to a certain point where clients started to timeout and the video playback completely stopped. and After these changes, we were able to achieve a latency in the order of 10ms for our 99% percentile. How did you test this? Are there any benchmarking/load testing tools for cases such as strea…
If I'm not wrong, we start to access that around 1y but I think we really spent 7-10m on it.
You can send metrics from Cassandra to graphite http://www.datastax.com/dev/blog/pluggable-metrics-reporting...
Since the streaming is similar to HTTP page flow, it's not that hard.
Ex: http://blazemeter.com/blog/how-load-test-http-live-media-str...
Re: FIFA 2014 World Cup live stream architecture
#23It feels like they reinvented the wheel on a lot of things (especially on the monitoring side). I also wish there was some more detail about the caching/endpoint side of things. We rely heavily on Akamai's services for their edge servers, caching, and site acceleration. I don't see anything like that here, but maybe they're just not mentioning it?
We only implemented the dashboard (getting data from all kinds of source) on monitoring side, we relied on: logtash, graphite, elasticsearch...
The architecture up from what was described is almost purely to provide caching capabilities, aka bunch of edge servers caching content to final users.
Re: FIFA 2014 World Cup live stream architecture
#24Just like everyone else, I really liked the article. And of course I have some questions. What is the benefit of EvoStream over nginx-rtmp? I use nginx-rtmp as well and it creates HLS files just fine. Also, how did you handle the network load? How many edge servers were actually delivering the content to users? Did you use 10G ethernets, link bonding, or did you use cloud services (eg cloudfront) to deliver the conte…
* We initially picked EMS because of its API, but we're moving to nginx-rtmp. * We used 20G ethernet, we could get 19G from each machine. * All the content is hosted by us.
Thank you for your answer. Appreciate it and best wishes.
Re: FIFA 2014 World Cup live stream architecture
#25Seems like quite a complex solution for 500,000 concurrent users.
Why not just use Akamai for RTMP ingest and delivery? From my experience they can do all that was described and at scale.
Re: FIFA 2014 World Cup live stream architecture
#26Seems like quite a complex solution for 500,000 concurrent users. Why not just use Akamai for RTMP ingest and delivery? From my experience they can do all that was described and at scale.
Please keep in mind that FIFA14 was one event, we still broadcast many other events and channels simultaneously ( private and public).
About why chose HLS over RTMP, in our experiments HLS showed to be easier to scale than RTMP (maybe because the latter is stateful + the current players don't have an optimized adaptive bitrate algorithm + it's easier to scale http over rtmp)
Re: FIFA 2014 World Cup live stream architecture
#27Seems like quite a complex solution for 500,000 concurrent users. Why not just use Akamai for RTMP ingest and delivery? From my experience they can do all that was described and at scale.
Please keep in mind that FIFA14 was one event, we still broadcast many other events and channels simultaneously ( private and public). About why chose HLS over RTMP, in our experiments HLS showed to be easier to scale than RTMP (maybe because the latter is stateful + the current players don't have an optimized adaptive bitrate algorithm + it's easier to scale http over rtmp)
+ iOS devices won't play RTMP.
Re: FIFA 2014 World Cup live stream architecture
#28Seems like quite a complex solution for 500,000 concurrent users. Why not just use Akamai for RTMP ingest and delivery? From my experience they can do all that was described and at scale.
Please keep in mind that FIFA14 was one event, we still broadcast many other events and channels simultaneously ( private and public). About why chose HLS over RTMP, in our experiments HLS showed to be easier to scale than RTMP (maybe because the latter is stateful + the current players don't have an optimized adaptive bitrate algorithm + it's easier to scale http over rtmp)
Yes agree, but Akamai can also transcode and repackage to HLS. We've used this at SportsLabs[1] to stream tens of thousands of sports events.
Re: FIFA 2014 World Cup live stream architecture
#29Seems like quite a complex solution for 500,000 concurrent users. Why not just use Akamai for RTMP ingest and delivery? From my experience they can do all that was described and at scale.
NIH syndrome.
Re: FIFA 2014 World Cup live stream architecture
#30Seems like quite a complex solution for 500,000 concurrent users. Why not just use Akamai for RTMP ingest and delivery? From my experience they can do all that was described and at scale.
NIH syndrome.
I disagree (maybe with a little context you will do understand):
* We were only allowed to broadcast to Brazil.
* We already have all the servers and needed bandwidth, why would you pay when you already have all the servers?