Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon device recorded private conversation, sent it out to random contact

kiro7.com

Post reply on HN
Print view View original

691–700 of 734 posts

Re: Amazon device recorded private conversation, sent it out to random contact

#691

Post by andreyk »

titzer wrote: post #690
andreyk wrote: post #686

Earlier quoted context omitted.

It's a 1-in-a-million edge case.... I would hope people on HN at least would understand this stuff happens in software.

1 in a million happens about 4000 times a second on a single CPU running at 4ghz... Did you really want to talk concrete numbers? Because if so, I am wondering about the probability that Amazon's voice recognition mistakes random conversation as a valid entry in someone's contact list, as well as the other terms in this equation. I don't know if you work for Amazon, and if so I don't want to single you out specifical…

I do not work in Amazon. I side with them on this because look at this protocol:

1 Wake up w “Alexa.” 2 respond to “send message” 3 respond to “To whom?” 4 respond to “[contact name], right?”

As an engineer (well, an AI researcher who used to engineer), that looks to me like they were not negligent and it was hard to predict background conversation would produce this unlikely set of inputs - it would be nice to see stats but this is the first time it is covered in the media to my knowledge. And as with the laughter story, they will now change the inputs to make it likely 1-in-a-trillion this will happen.

However, it does seem like Alexa etc. will have to be better about recognizing audio from TVs/conversations and stuff directed at it - and I am sure they are working on it.

PS comparison to CPU is not great obvs since it's about number of instances (how many times does "Alexa" get woken up by background audio - not 4ghz)

Re: Amazon device recorded private conversation, sent it out to random contact

#692

Post by toss1 »

strken wrote: post #499
toss1 wrote: post #198

Earlier quoted context omitted.

And this is where "software" diverges from "engineering". Bugs happen in architecture, aircraft, etc. too. the difference is that the actual engineers are paid to have a precautionary approach and spend significant resources to actively prevent bugs from making it into the final product. In contrast, software is often written to "ship first", be "agile", and "move fast and break things". Yet when it causes problems,…

It's worth observing that, firstly, aircraft still crash and bridges still collapse, amd secondly, that a culture of blameless analysis has gone a long way towards making them crash and collapse less.

Absolutely!

But note the blameless analysis is NOT the same as saying "meh" errors happen.

It is a culture and deliberate practice of allocating resources to seeking and classifying risks, and designing, implementing, and testing engineering and procedural mitigation strategies (vs. development as usual).

Re: Amazon device recorded private conversation, sent it out to random contact

#693

Post by titzer »

saas_sam wrote: post #287
bmans94 wrote: post #259

It's absurd to me that someone would wire up "every room in their house" with internet connected microphones (in a technology that is still in it's infancy!) and then claim you felt like your privacy was "invaded". At best, it's a Trojan Horse, but the naivety here is astounding. Of course your Echo shouldn't send out private conversations, and it looks like a legitimate bug occurred. But man, the buy-now think-later…

You literally carry an audio & video recording device with a GPS chip and multiple wireless attack vectors on you for nearly every waking hour of your life.

I don't. I use my phone less and less. And I disable location every chance I get, leave it at home when I go out, on my desk when I step away, it stopped inhabiting my pocket, stays in the other room when I am concentrating, etc. I don't trust my phone, but occasionally it's useful. Why is this hard for people?

Re: Amazon device recorded private conversation, sent it out to random contact

#694

Post by amzn_engineer1 »

amzn_engineer1 wrote: post #66

What actually happened: Alexa misinterpreted some voice commands and activated a "call" skill. The people involved and local news got very excited and escalated this into a conspiracy story. Amazon takes customer privacy EXTREMELY seriously. There's no way a team would get the "ok" to build a skill that randomly records private conversations then sends them to a random contact. It also doesn't make any logical sense…

I shouldn't have called this a "conspiracy story" and I should have given more respect to the parties involved who experienced this. While I think the issue is similar to a "butt dial", the customer felt violated and more precautions should have been taken to prevent this.

What I should have said is this report makes it difficult to understand what actually happened. It seems clear this article favors click-bait quotes that insinuate a "big brother" vibe such as:

> "'unplug your Alexa devices right now,' she said. 'You're being hacked.'"

If she had instead said "Alexa butt-dialed me!", would you still be interested in this report?

Re: Amazon device recorded private conversation, sent it out to random contact

#695

Post by titzer »

andreyk wrote: post #691
titzer wrote: post #690

Earlier quoted context omitted.

1 in a million happens about 4000 times a second on a single CPU running at 4ghz... Did you really want to talk concrete numbers? Because if so, I am wondering about the probability that Amazon's voice recognition mistakes random conversation as a valid entry in someone's contact list, as well as the other terms in this equation. I don't know if you work for Amazon, and if so I don't want to single you out specifical…

I do not work in Amazon. I side with them on this because look at this protocol: 1 Wake up w “Alexa.” 2 respond to “send message” 3 respond to “To whom?” 4 respond to “[contact name], right?” As an engineer (well, an AI researcher who used to engineer), that looks to me like they were not negligent and it was hard to predict background conversation would produce this unlikely set of inputs - it would be nice to see s…

They would like to put this in a billion homes no doubt, and suppose that people are home, talking, a couple hours a day. Now we are talking real numbers. A few billion conversation-hours per day, 365 days in a year--suddenly one in a trillion is starting to look like it's gonna happen a couple times a year. Now if Amazon knows these probabilities--which they don't, because they clearly have not done due diligence in understanding their rapidly evolving, inscrutable voice models--they are now knowingly violating eavesdropping laws, probabilistically.

This is part of the problem. People want to handwave away small probabilities when they should be busting their asses to make probabilities actually 0--solutions like not having this crap in their house at all.

Re: Amazon device recorded private conversation, sent it out to random contact

#696

Post by wilsonnb »

rectang wrote: post #399
wilsonnb wrote: post #375

Earlier quoted context omitted.

That seems like an extreme perspective. Kind of like the tech version of abstinence only sex ed. Sure, it's the only 100% safe way, but it's not useful for most people. It doesn't weigh the benefits against the potential cons, or even take into consideration the actually likelihood of data being leaked.

The alternative perspective you're presenting sounds to me utterly cavalier about the prospect of ruining people's lives. It's like Equifax's attitude towards identity theft: it doesn't affect their profitability, so why care? It's because such blithe dismissal of the damage caused by data gathering is so prevalent in the industry that the likelihood of devastating compromise is so high and the costs borne by the pop…

The problem I have is that you are conflating potential damage and actual damage as the same thing, which is not how you accurately measure risk.

I am honestly confused as to how you interpreted my last comment as "utterly cavalier about the prospect of ruining people's lives", when all I said was that your assumption doesn't take into account the actual probability of data being leaked and it doesn't weigh any of the benefits of data collection against that risk.

Re: Amazon device recorded private conversation, sent it out to random contact

#697

Post by amzn_engineer1 »

toss1 wrote: post #233
amzn_engineer1 wrote: post #66

What actually happened: Alexa misinterpreted some voice commands and activated a "call" skill. The people involved and local news got very excited and escalated this into a conspiracy story. Amazon takes customer privacy EXTREMELY seriously. There's no way a team would get the "ok" to build a skill that randomly records private conversations then sends them to a random contact. It also doesn't make any logical sense…

And this is where "software" diverges from "engineering". It's not a conspiracy, it's negligence. Bugs happen in architecture, aircraft, etc. too. the difference is that the actual engineers are paid to have a precautionary approach -- and spend significant resources -- to actively prevent bugs from making it into the final product. Amazon and your team has built a great product (I have one and make moderate use of i…

I agree with you and my response to another comment elaborates: https://news.ycombinator.com/item?id=17154679

I'm all for doing away with the "it's gotta ship yesterday" mentality, what your thoughts are on how developers can help stop it?

Re: Amazon device recorded private conversation, sent it out to random contact

#699

Post by [deleted] »

flukus wrote: post #574
HankB99 wrote: post #572

Earlier quoted context omitted.

That's good but I think I prefer http://dilbert.com/strip/1994-04-24

This is even better because of the date, we had voice operated computers 24 years ago that ran on machines that wouldn't even be called a potato these days and they didn't need to send the voice off for cloud processing. And voice control didn't fail way back then because it wasn't good, it failed because it was a terrible input mechanism and that is why it will fail again.

[deleted]

Re: Amazon device recorded private conversation, sent it out to random contact

#700

Post by sqlacid »

ajross wrote: post #250
btgeekboy wrote: post #34

Earlier quoted context omitted.

It sounds like she had the voice-activated equivalent of butt-dialing.

Yeah, pretty much. It's likely that the device heard something like "Alexa, send a note to Enrique" and then recorded and sent the subsequent note to the matching contact that it found. The microfailure is that it either failed to respond to the user in a way that was clear, or did it in a way that the user didn't notice (I dunno, volume all the way down? Output to a bluetooth speaker or headphone jack that wasn't au…

>Output to a bluetooth speaker or headphone jack that wasn't audible?). And that can be fixed.

Not sure how to fix that, if you use the aux out it has no way of knowing what happens on the other end; a confirmation prompt that isn't heard could still pick up an errant "Ok" response. Perhaps require a PIN like the shopping confirmation that would make it much much more unlikely? Or better yet, disable messaging as an option.

Post reply on HN