Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Live-caption glasses let deaf people read conversations [video]

youtube.com

Post reply on HN
Print view View original

141–150 of 181 posts

Re: Live-caption glasses let deaf people read conversations [video]

#141

Post by quantumquetzal »

ape4 wrote: post #32
quantumquetzal wrote: post #15

Background: researched this space for a graduate degree. There are a few issues that are unanswered by this video (which isn't intended to be a technical deep dive, but I don't see any related links in the video description): 1. How do these glasses handle multiple simultaneous speakers? Based on the display I saw, it shows the speakers' words sequentially, which starts to fall apart in real-world environments, espec…

It would be nice if it put the captions over the speaker (in a speech bubble?)

There’s a significant amount of evidence pointing to the fact that “anchored” captions (a la speech bubbles, like a comic book) is a preferred way to render captions, assuming the bubbles are at the same focal distance as the speaker. This is solvable with “true” AR, but getting that compute into the lightweight form factor of the original video is an unsolved problem, and is a ways off.

Re: Live-caption glasses let deaf people read conversations [video]

#142

Post by quantumquetzal »

DrKeithDuggar wrote: post #23
quantumquetzal wrote: post #15

Background: researched this space for a graduate degree. There are a few issues that are unanswered by this video (which isn't intended to be a technical deep dive, but I don't see any related links in the video description): 1. How do these glasses handle multiple simultaneous speakers? Based on the display I saw, it shows the speakers' words sequentially, which starts to fall apart in real-world environments, espec…

Thank you for looking at XRAI Glass! 1. For multiple simultaneous speakers of comparable volume, it’s only as good as the underlying speech-to-text engines we’ve implemented/integrated, which is currently not very good. It’s active area of research and engineering for us and we believe we’ll make strides to improve things; but, as you rightly point out, solving the crosstalk problem is very difficult. For the more ge…

Wow! Thanks for the response. This is exciting work, and I’m pleased to see it being iterated on.

Re: #2. I’m assuming the varying depth is manually-controlled? Or is it automated by some method? If it’s manual, can the adjustment be made while transcription is active? In other words, can I change the focal distance to match the speaker without interrupting the speaker?

All in all, cool stuff! Best of luck with the work.

Re: Live-caption glasses let deaf people read conversations [video]

#143

Post by quantumquetzal »

jackblemming wrote: post #31
elil17 wrote: post #28

I'm a hearing person and I've spent a summer interning in a 50/50 mixed Deaf and hearing research group. My take is that this is a huge UI improvement for AI speech to text, which a lot of Deaf people are already using to listen to conversations. It seems particularity great because it allows this technology to provide situational awareness while, for example, walking. It's important to remember, though that, for con…

Why can’t they also make glasses that translate sign language into English audio or text then?

Sign language (speaking with ASL in mind, unfamiliar with others) uses a different grammatical structure from American English. To boot, a signer can adopt different body languages to imbue their message with different tonality. Current ML models (as far as I’m aware) struggle with capturing that tonality in speech-to-text, and there’s _far_ less data for sign language recognition, regrettably.

Re: Live-caption glasses let deaf people read conversations [video]

#144

Post by lern_too_spel »

dymk wrote: post #46
Blackthorn wrote: post #24

This is the sort of thing I expected Google Glass to be able to do. But Google apparently lacks sufficient imagination, so they just unceremoniously canned it. Really hope this takes off.

Google literally announced this feature 10 months ago at I/O 2022

Not with Google Glass but with some new vaporware glasses. https://www.theverge.com/2022/5/13/23068759/google-io-2022-a...

Re: Live-caption glasses let deaf people read conversations [video]

#145

Post by flangola7 »

quantumquetzal wrote: post #15

Background: researched this space for a graduate degree. There are a few issues that are unanswered by this video (which isn't intended to be a technical deep dive, but I don't see any related links in the video description): 1. How do these glasses handle multiple simultaneous speakers? Based on the display I saw, it shows the speakers' words sequentially, which starts to fall apart in real-world environments, espec…

The Android voice recording app can already write a transcript in real time that indicates different speakers.

Re: Live-caption glasses let deaf people read conversations [video]

#146

Post by baseline-shift »

danscarfe wrote: post #135
raldi wrote: post #104

Earlier quoted context omitted.

The video gives the general idea, but it only shows a graphical simulation of what the wearer sees. I'm curious to see an actual photo or video of the text appearing on the lens of the glasses, in reality, with no special effects.

It doesn't appear on the glasses. It appears in 3D space infront of you. It's almost impossible to show via video.

I don't understand this page https://www.nreal.ai/compatibility-list/ What do the glasses need to be compatible with? I don't recognize or own any of these products listed.

I have an iPhone. How does it work with a device as part of the deal?

Re: Live-caption glasses let deaf people read conversations [video]

#147

Post by justinator »

akira2501 wrote: post #128
justinator wrote: post #107

Earlier quoted context omitted.

>3. As mentioned by another commenter, this is a useful idea for people who lose their hearing later in life. That said, this is less (although certainly still) useful for people who have congenital hearing loss and primarily communicate via ASL.> Someone primarily communicates with ASL and then there's me that doesn't know ASL. I can speak to them, and they can read what I've spoken. That works pretty well. They com…

ASL is heavily inspired by English, so it's usually very easy for people to become basically conversant in it rather quickly. For signs you don't know, there's literal "finger spelling" that's part of the language, so conversational learning is greatly aided by this. Which.. aside from that, you could do this anyways. When I first started living with a deaf person, I just wrote things down on paper, and they wrote ba…

>This isn't adding much.

We can communicate without seeing each other.

Re: Live-caption glasses let deaf people read conversations [video]

#148

Post by danscarfe »

baseline-shift wrote: post #146
danscarfe wrote: post #135

Earlier quoted context omitted.

It doesn't appear on the glasses. It appears in 3D space infront of you. It's almost impossible to show via video.

I don't understand this page https://www.nreal.ai/compatibility-list/ What do the glasses need to be compatible with? I don't recognize or own any of these products listed. I have an iPhone. How does it work with a device as part of the deal?

Unfortunately Apple users will have to wait for the Apple glasses which are being released some point after 2025/2026. The specific things the glasses use are HDMI over DisplayPort Alt Mode (USB-C) and Qualcomm chips in the phone.

Re: Live-caption glasses let deaf people read conversations [video]

#149

Post by danscarfe »

raldi wrote: post #138
danscarfe wrote: post #135

Earlier quoted context omitted.

It doesn't appear on the glasses. It appears in 3D space infront of you. It's almost impossible to show via video.

Oh, interesting. It's not being projected onto the lenses, but straight into the eye or something?

The light is bounced straight into your eyes using something called a birdbath lens

Re: Live-caption glasses let deaf people read conversations [video]

#150

Post by danscarfe »

quantumquetzal wrote: post #142
DrKeithDuggar wrote: post #23

Earlier quoted context omitted.

Thank you for looking at XRAI Glass! 1. For multiple simultaneous speakers of comparable volume, it’s only as good as the underlying speech-to-text engines we’ve implemented/integrated, which is currently not very good. It’s active area of research and engineering for us and we believe we’ll make strides to improve things; but, as you rightly point out, solving the crosstalk problem is very difficult. For the more ge…

Wow! Thanks for the response. This is exciting work, and I’m pleased to see it being iterated on. Re: #2. I’m assuming the varying depth is manually-controlled? Or is it automated by some method? If it’s manual, can the adjustment be made while transcription is active? In other words, can I change the focal distance to match the speaker without interrupting the speaker? All in all, cool stuff! Best of luck with the w…

The depth is set manually but can be changed at any point, even mid conversation
Post reply on HN