Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Video streaming at scale with Kubernetes and RabbitMQ

alexandreolive.medium.com

Post reply on HN
Print view View original

81–90 of 100 posts

Re: Video streaming at scale with Kubernetes and RabbitMQ

#81

Post by TheFragenTaken »

mannyv wrote: post #25

Fuck K8. You literally don't need it. Maybe he needs it because he's building on google cloud. AWS is easier, but you can do it with anything. The basic steps are: 1. Upload the file somewhere 2. Transcode it 3. Put the parts somewhere 4. Serve the parts You should really transcode everything into HLS. It's 2023, and everything that matters supports it. If you want 4k you can use HLS or the other thing (which I keep…

Recently got burnt by Microsoft deprecating Azure Media Services, so I can understand if the author doesn't want to use a cloud offering for encoding.

Re: Video streaming at scale with Kubernetes and RabbitMQ

#83

Post by mrd3v0 »

mannyv wrote: post #25

Fuck K8. You literally don't need it. Maybe he needs it because he's building on google cloud. AWS is easier, but you can do it with anything. The basic steps are: 1. Upload the file somewhere 2. Transcode it 3. Put the parts somewhere 4. Serve the parts You should really transcode everything into HLS. It's 2023, and everything that matters supports it. If you want 4k you can use HLS or the other thing (which I keep…

Depending on proprietary software such as cloud offerings for something as essential to its requirements such as encoding is not sustainable, and will create technical debt as your software/company will rely on the profitable success of the cloud service.

Re: Video streaming at scale with Kubernetes and RabbitMQ

#85

Post by robertlagrant »

malux85 wrote: post #75
codenesium wrote: post #60

Earlier quoted context omitted.

Rabbit and most databases have their own failover strategy. Putting it all on k8s is fine for a toy app but idk why anyone would deploy a real system like that.

OK, I can only speak to my personal projects and 20+ years experience at work. We run all of our stateful and stateless workloads on 10+ kubernetes clusters at work in multiple datacenters in multiple continents, and we serve 500 million users a month with it. I wrote the first BORG version of DFP backend systems at Google, where we served billions of users billions of ads a day, and we used stateful infrastructure m…

There are layers to this. At the simplest level, you only have K8s people (and aren't willing to use cloud services). So you install the RabbitMQ Helm chart, hope for the best, and fix any issues that come up.

Then you get a bit worried that the Postgres Helm chart, while good, doesn't do what you want. So you update to use a dedicated clustered Postgres, using some Postgres clustering tech.

Finally, you're at so much scale you can throw giant wads of advertising cash at the problem, and you can use anything you like and it'll work. You just need to choose the best thing for your particular problem.

Re: Video streaming at scale with Kubernetes and RabbitMQ

#86

Post by robertlagrant »

malux85 wrote: post #45
robertlagrant wrote: post #44

Earlier quoted context omitted.

What happens when your storage detaches from your k8s cluster? Your services start 503ing, hopefully, because you didn't design your system thinking that k8s == 100% uptime.

Anybody can invent random problems ad nauseam - that doesn’t prove anything. I’m not claiming that it’s totally bullet proof, I never said that - I’m saying that if you had a kubernetes cluster anyway why not benefit from its abilities? Especially when the alternative is single node, single points of failure, which is clearly inferior. The "what if the storage detaches" argument could easily apply to the single node…

> Anybody can invent random problems ad nauseam

I agree; I was replying to this invented problem:

> What happens in the article infra when the rabbitmq or database nodes fail?

It makes sense if you read the reply as a reply.

Re: Video streaming at scale with Kubernetes and RabbitMQ

#87

Post by robertlagrant »

jrockway wrote: post #54
malux85 wrote: post #45

Earlier quoted context omitted.

Anybody can invent random problems ad nauseam - that doesn’t prove anything. I’m not claiming that it’s totally bullet proof, I never said that - I’m saying that if you had a kubernetes cluster anyway why not benefit from its abilities? Especially when the alternative is single node, single points of failure, which is clearly inferior. The "what if the storage detaches" argument could easily apply to the single node…

Yeah, what happens when someone in AWS clicks "delete database" accidentally? It's the same thing that happens when K8s blows up in some weird way. You restore from your backup. (Fun fact: deleting the instance deletes the backups!)

Again, I was replying to:

> What happens in the article infra when the rabbitmq or database nodes fail?

Re: Video streaming at scale with Kubernetes and RabbitMQ

#88

Post by goeiedaggoeie »

c7DJTLrn wrote: post #52
mannyv wrote: post #25

Fuck K8. You literally don't need it. Maybe he needs it because he's building on google cloud. AWS is easier, but you can do it with anything. The basic steps are: 1. Upload the file somewhere 2. Transcode it 3. Put the parts somewhere 4. Serve the parts You should really transcode everything into HLS. It's 2023, and everything that matters supports it. If you want 4k you can use HLS or the other thing (which I keep…

If I'm understanding correctly you're suggesting using Lambda for processing. Is that a good idea? Lambda is actually expensive for heavy workloads.

If your video is short enough to encode in the lambda limit it is worth considering that https://aws.amazon.com/blogs/media/processing-user-generated... mediaconvert is expensive, generally AWS video stack is expensive (MediaConvert, Elemental, Media Connect)

Re: Video streaming at scale with Kubernetes and RabbitMQ

#89

Post by irksomehails »

whalesalad wrote: post #59
mannyv wrote: post #25

Fuck K8. You literally don't need it. Maybe he needs it because he's building on google cloud. AWS is easier, but you can do it with anything. The basic steps are: 1. Upload the file somewhere 2. Transcode it 3. Put the parts somewhere 4. Serve the parts You should really transcode everything into HLS. It's 2023, and everything that matters supports it. If you want 4k you can use HLS or the other thing (which I keep…

serverless on aws is underrated if your workload can fit on it. i have an app that hasn't received a single ounce of maintenance in like 3 years that still "just works", collects revenue from stripe, does all the business logic on lambdas, generates downloadable print-friendly pdf's on lambdas, etc. the supporting tech is dynamo + triggers for lambda, s3 and related triggers for lambda. but it would be hard for a non…

Hey, just curios, can you link your app.

Re: Video streaming at scale with Kubernetes and RabbitMQ

#90

Post by afvid »

jonnycoder wrote: post #27
mannyv wrote: post #25

Fuck K8. You literally don't need it. Maybe he needs it because he's building on google cloud. AWS is easier, but you can do it with anything. The basic steps are: 1. Upload the file somewhere 2. Transcode it 3. Put the parts somewhere 4. Serve the parts You should really transcode everything into HLS. It's 2023, and everything that matters supports it. If you want 4k you can use HLS or the other thing (which I keep…

What would you recommend using as an alternative to being locked into AWS?

Full disclosure I currently work for Bitmovin, but we work well on AWS, Google and Azure. We have some migration tools and are one of Microsoft's recommended alternatives since they are sunsetting Azure Media Services. Our Streams product is a simple, one API call solution that just works, similar to some of the things mentioned already, but we also have a pretty advanced API that can do whatever you need. We're also about to wrap up a beta for GPU acceleration with NVIDIA and make that publicly available.
Post reply on HN