Live data from Hacker News

The day my script killed 10k phones in South America

new.pythonforengineers.com

151–160 of 164 posts

Re: The day my script killed 10k phones in South America

#151

Author here. I shared this yesterday, didnt get much upvotes and quickly vanished. Thanks to /u/nixcraft for sharing it again! Looks like 2nd time lucky. Im seeing the same type of comments here and on Reddit, I'll try to answer a few common ones. 1. Yes, I know it was stupid testing on production. But we'd been told we needed to release the product on Monday (this was Friday), no objections. A previous project manag…

Just replying here so you can see it, but I would caution against continuing to share this story as it's written. If I was a hiring manager and I saw this, I wouldn't hire you, regardless of how well you did during interviews.

These are my options, some shared by others in the threads here, some opposite.

> It would lock the low-level features that allowed you to make calls, use Wifi, or even post pictures on Instagram/Facebook (the horror!) until you paid up. ... > Except for the poor sod in Peru who couldn't make phone calls or post duck-face pictures to Instagram, but no one asked him.

Both of these quotes seems like you don't quite understand the impact on your end users, or just don't care. Neither are good options as you grow your career.

> Our own internal tool worked well (I would like to say it was all due to me, which would be an arrogant thing to say, but also 100% true, so I will say it: It was all or mostly due to me).

Your internal tool, which you are taking complete credit for, is what didn't have protections in place that ended up harming users. Also, it comes across as bragging or something. People that are too obsessed about credit usually are not ones I'd like to work with. Team efforts are hard to define and split up and people that over index on that usually are just bad team members.

> But one mobile phone manufacturer (that I will call the K-Pop Phone Manufacturing Company), based in Korea (yes, that one)

Say it or don't. These games where you can act all cheeky about how you didn't 'actually' say it is just juvenile IMHO.

> Because of time pressures, there was no time (or political will) to check the script was well written. As soon as I banged it out, it was live. And I mean literally, 10 seconds after I pressed save, the script was running on live production servers.

This is beyond just the org's failure. As engineers, we're responsible for what happens on our watch. We hold the line and try to enforce engineering practices, not shrug and yolo and then blame the org.

> I could have tested it better, but that would have meant working late into the night. No thanks. I had already worked a few late nights/weekends and I was done.

This comes across as again, you don't actually care at all. Not a hiring quality IMHO

> My colleague was helping me, so we got it all done in about an hour. By 18:30, we had unlocked all phones.

Oh no! You worked an entire hour between 5:30 pm and 6:30 pm! The horror! Most people work much much longer hours, later into the evening or night when things fail. The difference is they don't complain about it and care.

> A project manager confirmed it, and I took a breath of relief and quickly logged off before more fuckups could be found.

Great! You fixed it, but rather then trying to verify you fixed everything, you logged off at 6:30 and screw anyone else effected. Not a great look.

> The Lessons we learnt (or more likely, didn't learn)

If you didn't learn the lessons, why are you writing about it? Just as a war story? Or do you think this comes across as a positive for your career?

> The lesson here is: Do some f'king checks before you lock someone's phone, but I don't know if that lesson applied to us or K-pop (most likely, both of us).

You know. You're in control of what you can do, so you do what you can to prevent issues from happening under your watch.

> This is not just good QA, but good security as well. But bah! None of that silliness for us, thanks.

Again, you come across as uncaring. Why are you working in this industry?

> We assumed that because our own software checked for this sort of thing, that the K-pop phone's software would do as well.

Great. Your software failed. Don't blame downstream if you can't even get your first layer correct.

> Everyone pretended it was an honest mistake.

It was an honest mistake. The difference is if you learn and grow, or if you just shrug and make the mistakes over and over again.

Honestly, this is just my two cents. Take it all with grains of salt, and ignore as you wish. I don't intend to be personally attacking you or any of the sort. Just expressing where my mind is reading it.

If I was you, I'd take this down and seriously consider if engineering is the right career path for you. It doesn't seem like you actually enjoy it or care about it and just working for the easy money doesn't lead to a fulfilled life at the end. And living a fulfilled, happy life, is the most important thing after all.

Best of luck in the future.

Re: The day my script killed 10k phones in South America

#152
post #141
post #94

Earlier quoted context omitted.

1) You don't know that. And there may be emergency situations where you're on the phone with someone other than from the emergency number. 2) How rental cars do it doesn't seem to have anything to do with this. >By the way, if someone dies because of it, the only person whose conscience it is on is the person who bought a phone they couldn't afford and then didn't pay for it. Just like if someone died because the car…

Is not donating malaria nets to poor africans more or less defensible than choosing to sell them phones which deactivate when unpaid? You seem to be suggesting that it would be fucked up to not donate these phones to the poor people who aren't paying their bills. Does that mean that it's also fucked up to not donate them other things?

Nobody's saying they have to give out phones for free. We're saying that somehow those people ended up with the phones in their possession and the company's right to get paid DOES NOT mean it can lock out emergency functions on the phones. They can go ahead and cut off the normal phone service but if they lock out the dialer app and even one person fails to call an ambulance or police or firemen or whatever because of that lock that's one hundred percent on them. This goes double, triple if the lockout happened by mistake to innocent people.

Honestly, it's bad enough that these companies think it's even acceptable to remotely control computers. Locking out emergency services is just on a whole new level. Didn't get paid? Seek redress in the justice system like everyone else.

Re: The day my script killed 10k phones in South America

#153
post #116

Earlier quoted context omitted.

> If you get a phone on contract, you don't actually own it till you pay it off, and the company is entitled to lock it. No. You don't turn off somebody's phone while they're using it. You have no idea what they could be doing. Someone could be literally dying in front of your locked out consumer who is now unable to call for help. This is vital infrastructure and you're denying them access. If somebody dies because…

You know there is literally a 0% chance of the company that owns the phone getting it back after having it stolen from them. Having a system like this is the only reliable way of making sure people in low trust countries will pay what they bought.

So what? Don't loan out phones then. If you do, then don't lock out emergency functions. Incorporate the risk of not getting paid into the product's price or something.

In my country, if someone's life is in danger and someone else sees that, they are legally obligated to help. Failing to do that is criminal negligence. When those people are in front of a judge being asked why they didn't help, what will they answer? "Oh, I tried but the phone company locked me out." Will the judge care about the fact that the company locked them out because they owe them money? No, the judge is going to make it even more expensive for them at the very least and that's honestly a pretty generous outcome.

Failing to call for help because you don't have a phone? That's tough, a difficulty of life. Having one but being unable to because some idiot company locked you out? Yeah, that's extremely fucked up. Emergency services are rights and no company on earth gets to deny people those rights. Nobody really cares that the user didn't pay for the phone. What matters is that the phone was in the user's possession and could have saved a life had the company not locked the user out. Nobody deserves to die because of some company's entitlement.

Re: The day my script killed 10k phones in South America

#154
post #71

Earlier quoted context omitted.

“To the tens of thousands of people whose phones were suddenly and inexplicably bricked: my bad. You see, it was almost the weekend and although I had 3 days to responsibly approach this problem, I had already pulled several late nights, I’m sure you understand”

That's not the point. It's just not sensible to blame this one developer for a horrible environment in which this was the only option aside from getting fired, which is quite onerous a responsibility to put on a single person. This falls squarely on management.

>in which this was the only option aside from getting fired

I don’t believe an ill conceived script was his only option. And the rub for most people in these comments is that the code he wrote should have never made it to his fingers—he *knew* it was going to be rushed into prod, but he didn’t stop and consider that randomly generating numbers in a phone number-like schema to the scale of tens of thousands would actually semi-regularly produce a real phone number?! Seriously?

Re: The day my script killed 10k phones in South America

#155

Earlier quoted context omitted.

> If you get a phone on contract, you don't actually own it till you pay it off, and the company is entitled to lock it. No. You don't turn off somebody's phone while they're using it. You have no idea what they could be doing. Someone could be literally dying in front of your locked out consumer who is now unable to call for help. This is vital infrastructure and you're denying them access. If somebody dies because…

1) I highly doubt that the app disables emergency calling. 2) Some (rental, probably also lease) vehicles actually do have disablers. They prevent the vehicle from being started, not stop them in the middle of the road. 3) The police don't repo a thing. That's a civil matter and generally outsourced to the lowest bidder with a tow truck. By the way, if someone dies because of it, the only person whose conscience it i…

> I highly doubt that the app disables emergency calling.

Well, I highly doubt that it doesn't. Given how little attention the company paid to these important details, who's to say they didn't simply brick the user's phone?

> They prevent the vehicle from being started, not stop them in the middle of the road.

Like I said, nobody will randomly disable somebody's car engine. These people at least have enough foresight to wait until it's safe.

> The police don't repo a thing.

They do where I live. Bank loans money with car as collateral, consumer defaults, bank seeks justice, judge authorizes police to repossess the car so the bank can sell it and minimize losses. Police can enter private property and even use force if necessary.

What is even the point of police if they aren't gonna do things like this? Outsourced to tow trucks? Really? When some random citizen reposseses property, it's robbery.

> if someone dies because of it, the only person whose conscience it is on is the person who bought a phone they couldn't afford and then didn't pay for it

No. The fact that person didn't pay for the phone doesn't matter matter at all. The company can always negotiate new terms or seek justice later.

What matters is the fact that in an emergency the only thing standing between the consumer and a call for help was the phone company. Worse, they got in the way on purpose, as if it was their job to punish the consumer. There is no defending this.

> Just like if someone died because the car got repo'd and they couldn't drive to a hospital.

Not really. If you don't have a car, you can still call emergency services. At least you would have been able to if some stupid company had not locked your phone.

Re: The day my script killed 10k phones in South America

#156
post #141

Earlier quoted context omitted.

Is not donating malaria nets to poor africans more or less defensible than choosing to sell them phones which deactivate when unpaid? You seem to be suggesting that it would be fucked up to not donate these phones to the poor people who aren't paying their bills. Does that mean that it's also fucked up to not donate them other things?

Nobody's saying they have to give out phones for free. We're saying that somehow those people ended up with the phones in their possession and the company's right to get paid DOES NOT mean it can lock out emergency functions on the phones. They can go ahead and cut off the normal phone service but if they lock out the dialer app and even one person fails to call an ambulance or police or firemen or whatever because o…

Okay, please go ahead and show me where the article claims that they "lock[ed] out emergency functions on the phones", thanks.

Re: The day my script killed 10k phones in South America

#157
post #140

Earlier quoted context omitted.

If everyone saw things like you, there would probably not be nearly as many poor people with phones. It is these very consequences (repo, deactivation) that make it possible for these phones to be offered at affordable prices. Sadly we don't live in a utopia, especially the poor people of this world don't. I'm sure they'd rather have the phone that stops working and possibly indirectly kills their family when unpaid…

Curious, as I am from a poor country where: 1. Lots of poor people have phones, sometimes very expensive ones (as these are sought as a symbol status) 2. There is no such thing as repossession. And living abroad makes me believe that it is an phenomenon in United States only. It is illegal in many parts of the world. *edit: word

> There is no such thing as repossession

sure there isn't. Guess there's no financing, either, then, huh?

Re: The day my script killed 10k phones in South America

#158

Earlier quoted context omitted.

Nobody's saying they have to give out phones for free. We're saying that somehow those people ended up with the phones in their possession and the company's right to get paid DOES NOT mean it can lock out emergency functions on the phones. They can go ahead and cut off the normal phone service but if they lock out the dialer app and even one person fails to call an ambulance or police or firemen or whatever because o…

Okay, please go ahead and show me where the article claims that they "lock[ed] out emergency functions on the phones", thanks.

Okay. It's right there on the title: "my script killed 10k phones". Do you have any reason to believe "killed" means anything lesser than "rendered completely useless, bricked"? Also, there's the fact the software "would lock the low-level features that allowed you to make calls". There is no reason to believe calls to emergency services were exempted.

I don't really want to comment on the rest of the article. Actually, as one of the "poor sods from places like Peru and Chile" I do have quite a few things to say about the author's gross negligence and his endless rationalizations. I just don't want to get permabanned for saying them. Suffice it to say that negligence of this magnitude is actually a crime where I live.

Re: The day my script killed 10k phones in South America

#159

Earlier quoted context omitted.

Yes, just like you still own the phone if you take out a bank loan, use it to buy the phone, and then default on the loan.

But you don't, not if you secured the bank loan using the phone. Which you did if you financed it through a carrier. Just like if I go to a bank, get a loan for a car, buy that car, and stop making payments... THE BANK OWNS THE CAR.

I assume that isn't done because the value of the good is too low to justify the effort of repo'ing it.

Re: The day my script killed 10k phones in South America

#160

Earlier quoted context omitted.

Curious, as I am from a poor country where: 1. Lots of poor people have phones, sometimes very expensive ones (as these are sought as a symbol status) 2. There is no such thing as repossession. And living abroad makes me believe that it is an phenomenon in United States only. It is illegal in many parts of the world. *edit: word

> There is no such thing as repossession sure there isn't. Guess there's no financing, either, then, huh?

Not only there is no repossession but almost everything is sold in installments. You can pay in installments even in supermarkets.

In that case what makes the system works is the credit system. When almost everything you buy you must use installments then losing your credit is a terrible situation and the only solution is to pay (or try to).

In the beginning of my career I worked in a small local ISP and usually the middle/high class were a bigger part of the people with overdue payments, while people living in worse neighbourhoods were almost always paying in time. They can't afford to not pay.

Sometimes we live in a bubble that makes we think the way we live and the way things are are the only way things should be. We must be careful to not extend our vision of things to everyone and everything, the world is much bigger than our backyard.

*edit: a paragraph

Post reply on HN