Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Video streaming at scale with Kubernetes and RabbitMQ

alexandreolive.medium.com

Post reply on HN
Print view View original

61–70 of 100 posts

Re: Video streaming at scale with Kubernetes and RabbitMQ

#61

Post by hotnfresh »

mikeryan wrote: post #58
hotnfresh wrote: post #20

Earlier quoted context omitted.

CF meaning Cloudflare? If you’re serving video through them, then you’re in “enterprise plan” territory. You can’t do that on the free or “self-serve” paid plans. $5k+/m depending on bandwidth needs (and if you just need a cdn to push bits, CF won’t be competitive on price—their enterprise prices are tailored for companies that want all sorts of managed services and private networking stuff)

Um. Cloudflare stream starts at $5 per month and you don’t pay for encoding only storage and bandwidth. You can serve a decent video library for $500 per month. https://www.cloudflare.com/products/cloudflare-stream/

Ah, must’ve changed up their billing structure to provide more add-ons for the self-serve plans since last time I was dealing them them. That’s good.

Re: Video streaming at scale with Kubernetes and RabbitMQ

#62

Post by bmurphy1976 »

_joel wrote: post #47
mannyv wrote: post #25

Fuck K8. You literally don't need it. Maybe he needs it because he's building on google cloud. AWS is easier, but you can do it with anything. The basic steps are: 1. Upload the file somewhere 2. Transcode it 3. Put the parts somewhere 4. Serve the parts You should really transcode everything into HLS. It's 2023, and everything that matters supports it. If you want 4k you can use HLS or the other thing (which I keep…

Not one mention of MPEG-DASH

You need to use MPEG DASH if you are contractually obligated to use DRM. You can create an M3U8 and an MPEG DASH manifest that share the same encrypted segment files.

Re: Video streaming at scale with Kubernetes and RabbitMQ

#63

Post by dilyevsky »

jiggawatts wrote: post #31
mannyv wrote: post #25

Fuck K8. You literally don't need it. Maybe he needs it because he's building on google cloud. AWS is easier, but you can do it with anything. The basic steps are: 1. Upload the file somewhere 2. Transcode it 3. Put the parts somewhere 4. Serve the parts You should really transcode everything into HLS. It's 2023, and everything that matters supports it. If you want 4k you can use HLS or the other thing (which I keep…

This post is somewhat unfairly voted down. Cloud services like S3 and Azure Storage were invented specifically for hosting images and video. That’s their origin story, their foundation, their very reason for being. Similarly, cloud functions / lambda were invented for background processing of blobs. The first demos were always of resizing images! Building out this infrastructure yourself is a little insane. Unless yo…

I actually worked on system just like OP is describing and we ran everything ourselves including DBs (on k8s) so I can offer some perspective. For egress heavy services such as video minimizing outrageous cloud egress fees is key. Having an ability to tell your cloud account manager you can easily run multi-cloud or on-prem makes it a lot easier to negotiate sane rates.

Ultimately you’re competing with the likes of Twitch (IVS), youtube and cloudflare on price and they ALL run their own compute so at certain size you will have to run your own hardware to stay competitive especially now that zirp is in rear view mirror.

See Dropbox as another example of this but in documents/storage space

Re: Video streaming at scale with Kubernetes and RabbitMQ

#64

Post by dilyevsky »

I used to work on system like this and even built the logic to use preemptible pool effectively just like OP. If I had to design it from scratch today I would use Temporal for job scheduling - their durable compute concept is perfect fit for this and we had a lot of trouble maturing the equivalent scheduling system trying to keep up with rapidly growing scale

Re: Video streaming at scale with Kubernetes and RabbitMQ

#65

Post by chunkymilk »

dbrueck wrote: post #13
com2kid wrote: post #9

This is nice if you only have to deliver in one format, but as soon as you want to show up on TVs you are stuck delivering in a lot of formats, and life gets complicated quickly. Throw subtitles in multiple languages, and different audio tracks, into the mix, and all of a sudden streaming video becomes a nightmare. Finally, if you are dealing with copyrighted materials, you have to be aware as to what country your us…

Oh, this is just the tip of the iceberg. Many parts of on-demand video streaming are largely commoditized at this point. Add in support for linear (live) streaming and ad insertion and things start to get really interesting. :)

My background is in linear (title VI and OTT) and ad-insertion. A big chunk of my job is explaining to folks that just because you solved VOD that doesn't mean you've solved linear. It's almost best to think of them as two distinct problems.

Re: Video streaming at scale with Kubernetes and RabbitMQ

#66

Post by dilyevsky »

goeiedaggoeie wrote: post #22
andrewstuart wrote: post #11

Earlier quoted context omitted.

Slap one of these puppies in…. AMD Alveo MA35D Media Accelerator https://www.xilinx.com/applications/data-center/video-imagin...

Ive used xilinx a fair bit for encoding. once you get past the pain of compiling your tooling for it it does speed up VOD encode significantly.

How’s the quality? I heard it was so-so and i think you can’t close your own presets

Re: Video streaming at scale with Kubernetes and RabbitMQ

#67

Post by dilyevsky »

klaussilveira wrote: post #7
andrewstuart wrote: post #2

Must be expensive to run on Google Cloud. Also looks pretty complex. The stabilization step presumably does a video encode …. that’s extremely expensive in terms of time, compute and money I wonder why it’s necessary.

I wonder if it wouldn't be cheaper to run an on-prem farm of BestBuy-grade "gamer PC" for smaller scale networks like that.

Twitch used to use cheap cores with Intel quicksync and maybe still does

Re: Video streaming at scale with Kubernetes and RabbitMQ

#69

Post by hckr1292 »

mikeryan wrote: post #50
mannyv wrote: post #25

Fuck K8. You literally don't need it. Maybe he needs it because he's building on google cloud. AWS is easier, but you can do it with anything. The basic steps are: 1. Upload the file somewhere 2. Transcode it 3. Put the parts somewhere 4. Serve the parts You should really transcode everything into HLS. It's 2023, and everything that matters supports it. If you want 4k you can use HLS or the other thing (which I keep…

I would like to know what the costs come out to per minute of video encoded and how many outputs they're getting in order to compare this to something like Media Convert (AWS), Google Transcode or something dedicated like Mux. For reference Google Transcoder is about $0.13 per minute of encoding (at four resolutions). Mux is at $0.032 / min, AWS Media Convert $0.0188. I should note I know Mux's pricing well, use them…

Full disclosure, I currently work at Mux on the video product. Previously though, I worked at an education startup with user generated video content. Like many others commenting on this thread, I built a simple queuing system using RabbitMQ and celery, transcoding on EC2 with ffmpeg. While we might have saved some money by doing this in house, we almost certainly discouraged users from uploading content because the entire video needed to transcoded before it could be viewed. For use cases like the breaking news or high traffic user generated content, you really want to minimize wait time, and that requires some kind of special sauce. At Mux, we encode content just in time for very fast publish times. It’s very challenging to do this on your own.

Re: Video streaming at scale with Kubernetes and RabbitMQ

#70

Post by jackcviers3 »

dbrueck wrote: post #13
com2kid wrote: post #9

This is nice if you only have to deliver in one format, but as soon as you want to show up on TVs you are stuck delivering in a lot of formats, and life gets complicated quickly. Throw subtitles in multiple languages, and different audio tracks, into the mix, and all of a sudden streaming video becomes a nightmare. Finally, if you are dealing with copyrighted materials, you have to be aware as to what country your us…

Oh, this is just the tip of the iceberg. Many parts of on-demand video streaming are largely commoditized at this point. Add in support for linear (live) streaming and ad insertion and things start to get really interesting. :)

Article made me nostalgic.

My very first job was writing streaming media services for radio. Integrating the ad services over events embedded into the stream identifying content was a pretty nifty solution. You send the metadata to your ad server, it selects the ads based on your user and media metadata, you swap out the player's media url to play the ad(s), and keep the stream going in the background. When the commercial break is up, you kill the ad stream and swap the media stream player back in. Presto changeo, seamless ads in 2007.

You also track media playback through the stream embedded events.

The hard part back then was live video/audio encoding. These cards lived at the stations, and streamed to our cdn. Geocoding for American service bases in foreign countries was interesting too, the streaming rights for stuff limits the countries you can stream in.

Monitoring the live encoders was an interesting problem - I setup an event-based solution so I could capture the audio waveform and analyze the frequency buckets on dozens of streams at once. Could tell if a stream was silent pretty quickly, and alert the station engineer via email.

Back then, a lot of this stuff wasn't off the shelf like it was today. Even on the frontend,working with Flash, window's media player sdk, and the QuickTime sdk and integrating it with Javascript was challenging. And the backends were all in php and Java 1.4, and mysql, and a whole lot of memcache.

Was a really great experience for a new professional programmer to write all that.

Post reply on HN