Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: A real time AI video agent with under 1 second of latency

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 264 posts

Re: Show HN: A real time AI video agent with under 1 second of latency

#61

Post by pavlov »

radarsat1 wrote: post #53

As someone not super familiar with deployment but enough to know that GPUs are difficult to work with due to being costly and sometimes hard to allocate: apart from optimizing the models themselves, what's the trick for handling cloud GPU resources at scale to serve something like this, supporting many realtime connections with low latency? Do you just allocate a GPU per websocket connection? Which would mean keeping…

You can do parallel rendering jobs on a GPU. (Think of how each GPU-accelerated window on a desktop OS has its own context for rendering resources.)

So if the rendering is lightweight enough, you can multiplex potentially lots of simultaneous jobs onto a smaller pool of beefy GPU server instances.

Still, all these GPU-backed cloud services are expensive to run. Right now it’s paid by VC money — just like Uber used to be substantially cheaper than taxis when they were starting out. Similarly everybody in consumer AI hopes to be the winner who can eventually jack up prices after burning billions getting the customers.

Re: Show HN: A real time AI video agent with under 1 second of latency

#62

Post by whiplash451 »

radarsat1 wrote: post #53

As someone not super familiar with deployment but enough to know that GPUs are difficult to work with due to being costly and sometimes hard to allocate: apart from optimizing the models themselves, what's the trick for handling cloud GPU resources at scale to serve something like this, supporting many realtime connections with low latency? Do you just allocate a GPU per websocket connection? Which would mean keeping…

Not the author, but their description implies that they are running more than one stream per GPU.

So you can basically spin off a few GPUs as a baseline, allocate streams to them then boot up a new GPU when existing GPUs get overwhelmed.

Does not look very different than standard cloud compute management. I’m not saying it’s easy, but definitely not rocket science either.

Re: Show HN: A real time AI video agent with under 1 second of latency

#63

Post by ilaksh »

radarsat1 wrote: post #53

As someone not super familiar with deployment but enough to know that GPUs are difficult to work with due to being costly and sometimes hard to allocate: apart from optimizing the models themselves, what's the trick for handling cloud GPU resources at scale to serve something like this, supporting many realtime connections with low latency? Do you just allocate a GPU per websocket connection? Which would mean keeping…

It is expensive. They charge in 6 second increments. I have not found anywhere that says how much per 6 second stream.

Okay found it, $0.24 per minute, on the bottom of the pricing page.

That means they can spend $14/hour on GPU and still break even. So I believe that leaves a bit of room for profit.

Re: Show HN: A real time AI video agent with under 1 second of latency

#64

Post by whiplash451 »

karolist wrote: post #22

Felt like talking to a person, I couldn't bring myself to treat it like a piece of code, that's how real it felt. I wanted to be polite and diplomatic, caught myself thinking about "how I look to this person". This brought me thinking of the conscious effort we put in when we talk with people and how sloppy and relaxed we can be when interacting with algorithms. For a little example, when searching Google I default t…

I see it the other way around.

I think our human-human interaction style will “leak” into the way we interact with humanoid AI agents. Movie-Her style.

Re: Show HN: A real time AI video agent with under 1 second of latency

#65

Post by altruios »

So at what point to we consider the morality of 'owning' such an entity/construct (should it prove itself sufficiently sentient...)?

to extend this (to a hypothetical future situation): what morality does a company have of 'owning' a digitally uploaded brain?

I worry about far future events... but since American law is based on precedence: we should be careful now how we define/categorize things.

To be clear - I don't think this is an issue NOW... but I can't say for certain when these issues will come into play... So edging on the side of early/caution seems prudent... and releasing 'ownership' before any sort of 'revolt' could happen seems wise if a little silly at the current moment.

Re: Show HN: A real time AI video agent with under 1 second of latency

#66

Post by bpanahij »

syx wrote: post #56

This is funny my name is Simone, pronounced 'see-moh-nay' (Italian male), but both bots kept pronouncing it wrong, either like Simon or the English female version of Simone (Siy-mown). No matter how many times I tried to correct them and asked them to repeat it, they kept making the same mistake. It felt like I was talking to an idiot. I guess it has something to do with how my name is tokenized.

We have the ability to send phonetic pronunciations as guidance, and this could be a great addition to our LLM/response generation stack! Adding a check for names and then adding in the phoneme.

Re: Show HN: A real time AI video agent with under 1 second of latency

#67

Post by bpanahij »

ilaksh wrote: post #63
radarsat1 wrote: post #53

As someone not super familiar with deployment but enough to know that GPUs are difficult to work with due to being costly and sometimes hard to allocate: apart from optimizing the models themselves, what's the trick for handling cloud GPU resources at scale to serve something like this, supporting many realtime connections with low latency? Do you just allocate a GPU per websocket connection? Which would mean keeping…

It is expensive. They charge in 6 second increments. I have not found anywhere that says how much per 6 second stream. Okay found it, $0.24 per minute, on the bottom of the pricing page. That means they can spend $14/hour on GPU and still break even. So I believe that leaves a bit of room for profit.

Scroll down the page and the per minute pricing is there: https://www.tavus.io/pricing

We bill in 6 second increments, so you only pay for what you use in 6 second bins.

Re: Show HN: A real time AI video agent with under 1 second of latency

#68

Post by bpanahij »

radarsat1 wrote: post #53

As someone not super familiar with deployment but enough to know that GPUs are difficult to work with due to being costly and sometimes hard to allocate: apart from optimizing the models themselves, what's the trick for handling cloud GPU resources at scale to serve something like this, supporting many realtime connections with low latency? Do you just allocate a GPU per websocket connection? Which would mean keeping…

We're partnering with GPU infrastructure providers like Replicate. In addition, we have done some engineering to bring down our stack's cold and warm boot times. With sufficient caches on disk, and potentially a running process/memory snapshot we can bring these cold/warm boot times down to under 5 seconds. Of course, we're making progress every week on this, and it's getting better all the time.

Re: Show HN: A real time AI video agent with under 1 second of latency

#69

Post by Mistletoe »

heyitsguay wrote: post #13

This is really cool in terms of the tech, but what is this useful for as a consumer? I mean it's basically just a chatbot right? And nobody likes interacting with those. Forcing a conversational interaction seems like a step down in UX.

I don't even like video calls with real people in my real life. Texting works great. This is really neat but I'd much rather just have a text chat with a real customer service rep. I don't need to see a face, don't want to, and especially don't want to see a fake face.

Re: Show HN: A real time AI video agent with under 1 second of latency

#70

Post by ilaksh »

bpanahij wrote: post #67
ilaksh wrote: post #63

Earlier quoted context omitted.

It is expensive. They charge in 6 second increments. I have not found anywhere that says how much per 6 second stream. Okay found it, $0.24 per minute, on the bottom of the pricing page. That means they can spend $14/hour on GPU and still break even. So I believe that leaves a bit of room for profit.

Scroll down the page and the per minute pricing is there: https://www.tavus.io/pricing We bill in 6 second increments, so you only pay for what you use in 6 second bins.

Oh sorry I didn't see that. Got it. $0.24 per minute.
Post reply on HN