Show HN: A real time AI video agent with under 1 second of latency

11/10 creepiness, but well done. The hardest part of this for me was to hang up lol. Felt weird just closing the tab haha.

Sure but to me this sounds paranoid and as pointless as the movie industry trying to create non piratable technology... As in, worried about things out of their control. You cannot go about your life without using your voice unless you're a mute by choice or physically, and all a company needs is a few seconds of your voice to recreate it. If a company is hell bent on getting a voice, they can get it. If you're not widely known, or hold some kind of power, no one likely cares about your voice, and if you are, its likely there's already lots of audio sources of you out there... Even if you're not widely known, if you've ever made an instagram post, a reel, a tiktok, vine, youtube vid, etc, you're out there. Probably makes more sense to go on about your life and resort to legal means if your voice is used without your consent.

Same with your face... You leave your home, other humans see your face, cameras see your face. You do not get to control who sees your face or even who captures your face when you're in public, but you can decide whether or not you consent to your face being used by an entity for profit.

We make the distinction between humans consuming information and machines because humans can't typically reproduce the original material. So like, you can go see a movie, but you can't record it with a device which would allow you to reproduce it. But what if human brains could reproduce it? Then what? Then humans could replay it to themselves all they want, and to those near them, but wouldn't be allowed to reproduce it in mass for profit, or they'd get sued. I think the same stuff applies to data ingested by AI models. People care so much about what is fed in when the same information is fed in to humans around the world which increases their knowledge and informs their future decisions, their art, their thoughts. Humans don't have to pay to see a picture of the Mona Lisa, or pictures or any other art out there, even if it'll influence their own art later on. But somehow we want to limit what is fed to models based on it having gotten the permission to be influenced by its existence. I agree, we can't feed protected IP, or secret recipes, formulas for things that are not in the public sphere.. etc.. But other than that, not sure how people expect to limit what is fed into it that it can draw inspiration from.. As long as it doesn't copy verbatim... I get that images have been generated where original material has come out, but if its sections of, or concepts of, then its the same as a human being influenced by it, I honestly don't think that matters.

Then comes the idea that this is owned by a private company who's profiting from it all... Thats true... But there's also open source models that compete with them. Not sure what the best answers to it all is.. But to go back to the original point, if your unique voice, or image isn't copied precisely for profit, then whatever... It'll get used by models, or humans in their thoughts, you can't control what your existence affects in the world, just who gets to profit off of it.

E_PROGRAMMER_WAS_INSUFFICIENTLY_POLITE? With a corresponding E_PROGRAMMER_WAS_EXCEEDINGLY_POLITE?

https://en.wikipedia.org/wiki/INTERCAL for those who don't know.

That's a cool tech demo, I really like it. I thought about something similar with only open sourced components:

1. Audio Generation: styletts2 xttsv2 or similar for and fine tuning 5min of audio for voice cloning

2. Voice Recognition: Voice Activity Detection with Silero-VAD + Speech to Text with Faster-Whisper, to let users interrupt

3. Talking head animation: some flavor of wav2lip, diff2lip or LivePortrait

4. Text inference: Any grok hosted model that is fast enough to do near real time responses (llama3.1 70b or even 8b) or local inference of a quantized SML like a 3B model on a 4090 via vLLM

5. Visual understanding of users webcam: either gpt-4o with vision (expensive) or a cheap and fast Vision Language Model like Phi3-vision, LLaVA-NeXT, etc. on a second 4090

6. Prompt:

You are in a video conference with a user. You will get the user's message tagged with #Message: and the user's webcam scene described within #Scene: . Only reply to what is described in when the user asks what you see. Reply casual and natural. Your name is xxx, employed at yyy, currently in zzz, I'm wearing ... Never state pricing, respond in another language etc...

This has been addressed in fictional plots for ages. How well do you know the person you're talking to? There should be something only the two of you could possibly know. In Rick and Morty, Morty tells a judge the last words her husband spoke to her before dying to convince her he was really communicating with her dead husband (which was actually not true, but his tech was way beyond anything real AI will ever be able to do). Some digital cloning company like this might get your face and the basic shape of your body, but what about scars usually covered by clothing? Genitalia if you're willing to go there?

If it's a person you don't know, first ask if it matters. Is the point to get information or talk to a real person? If it's prospective romance or something, real people can still catfish and otherwise scam you. If, for whatever reason, it really matters, ask them to do a bunch of athletic tasks. Handstand. Broad jump. Throw a ball across the room. They're probably not going to scan people they digitally clone to see how they do these things, so chances are good with the techniques that exist today the vast majority of training data will be from elite athletes doing these things on television. No real person would actually be good at all tasks and will either be totally unable to do some of them or can do them but very clunkily. Do they warm up? Chances are good training data won't show that and AI clones trained by ML might not bother, but a real person would have to.