Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The day my script killed 10k phones in South America

new.pythonforengineers.com

Post reply on HN
Print view View original

21–30 of 164 posts

Re: The day my script killed 10k phones in South America

#21

Post by tcskeptic »

leephillips wrote: post #18

“the main product was an app...that would lock the phone if it was stolen, or if the customer stopped paying. The app was built as part of the Android OS, so you couldn't uninstall it. It would lock the low-level features that allowed you to make calls, use Wifi...until you paid up. All good.” No, not all good. There’s the problem right there. If I stop paying you can stop providing me the service I’m paying for. Usi…

Why do you assume the users own the phones? There are plenty of business models where the company owns the phones and the user gets to use it by paying a monthly fee. One of the services being provided may well be the use of the phone itself.

Re: The day my script killed 10k phones in South America

#22

Post by shaan7 »

heavenlyblue wrote: post #13
quenix wrote: post #2

This feels incredibly negligent. It seems obvious that generating thousands of random phone numbers and locking them without any other checks would lock some real phones. I'm puzzled that the author did not consider this when writing the script, let alone testing it or running it on live prod servers.

I don't understand how people can defend this developer in the first place. I could understand if they were about 16 years old and actually abused by their employer, but generating random numbers to test the locking of phones? Basic maths should tell you that the probability of matching some real phone number is actually quite high. Moreover, what exactly were they testing? i.e. if they were testing nonexistent numbe…

> Basic maths should tell you that the probability of matching some real phone number is actually quite high.

Yeah well, the thing about pressure (of the sort the author has describing) is that you make mistakes - the kind where some times 3+3 = 9.

Source: I queued jobs to nuke the whole production database (relax, we had automated backups+verification so we recovered fast enough; and then added some gating so make this more difficult in the future)

Re: The day my script killed 10k phones in South America

#23

Post by treesknees »

quenix wrote: post #2

This feels incredibly negligent. It seems obvious that generating thousands of random phone numbers and locking them without any other checks would lock some real phones. I'm puzzled that the author did not consider this when writing the script, let alone testing it or running it on live prod servers.

Not defending him, but he does state that unlike the other mobile providers he's worked with, this K-Pop company's backend system would lock _any_ phone number, not just the ones they're responsible for. Perhaps he didn't know this. I bet normally he'd generate random numbers not associated with the provider, check they show up as "locked" in the database, under the assumption that the backend actually doing the locking signal would reject the numbers due to them being under control of another operator.

It's similar to our networking lab where we generate bogus traffic using random src/dst IP addresses under the assumption that our null routes and firewall will block them from the real Internet. In this case, the provider just decided no firewall was needed. We can't use "fake" IP addresses (yes there are IP ranges for internal use but that doesn't work for certain testing scenarios.)

Re: The day my script killed 10k phones in South America

#24

Post by shantnutiwari »

leephillips wrote: post #18

“the main product was an app...that would lock the phone if it was stolen, or if the customer stopped paying. The app was built as part of the Android OS, so you couldn't uninstall it. It would lock the low-level features that allowed you to make calls, use Wifi...until you paid up. All good.” No, not all good. There’s the problem right there. If I stop paying you can stop providing me the service I’m paying for. Usi…

The product was for phones that had not been paid off-- if you get a phone on a 2 year contract, you dont "own" the phone until it is paid off

Re: The day my script killed 10k phones in South America

#25

Post by thih9 »

> Except for the poor sod in Peru who couldn't make phone calls or post duck-face pictures to Instagram, but no one asked him.

There are countless situations where suddenly not having a working phone means a serious problem. The article presents an extremely negligent look at the issue.

Re: The day my script killed 10k phones in South America

#26

Post by cyxxon »

tcskeptic wrote: post #21
leephillips wrote: post #18

“the main product was an app...that would lock the phone if it was stolen, or if the customer stopped paying. The app was built as part of the Android OS, so you couldn't uninstall it. It would lock the low-level features that allowed you to make calls, use Wifi...until you paid up. All good.” No, not all good. There’s the problem right there. If I stop paying you can stop providing me the service I’m paying for. Usi…

Why do you assume the users own the phones? There are plenty of business models where the company owns the phones and the user gets to use it by paying a monthly fee. One of the services being provided may well be the use of the phone itself.

That would be something to discuss with the user and the company that then might own the phone. In this case though, someone completely unrelated to the contract locked the phone. Someone in another country. Someone working for some completely unrelated telco company. Someone who just randomly locked some phones by banging out a script to generate phone numbers, without checking if the phones were owned by the user, their telco, or my grandmother.

Re: The day my script killed 10k phones in South America

#27

Post by donohoe »

stuff4ben wrote: post #7

If you haven't killed some production environment in your career, can you really call yourself a programmer? I kid of course, but I've been there, done that. It's lessons learned the hard way and I became a better developer because of it.

Yeah, but this one is particularly bad with a lot of real world consequences.

If I were that developer I would have never have blogged about it publicly. It is admitting to gross negligence IMHO.

Re: The day my script killed 10k phones in South America

#28

Post by scollet »

kingcharles wrote: post #20
stuff4ben wrote: post #7

If you haven't killed some production environment in your career, can you really call yourself a programmer? I kid of course, but I've been there, done that. It's lessons learned the hard way and I became a better developer because of it.

I once ran a query on a live production web site thinking I was doing a SELECT but somehow ran an UPDATE which set all the millions of users' gender to 'M', regardless of their selection. Restoring from backups would have been a nightmare. And telling my supervisor would have been a nightmare. So I used everyone's selected title (Mr, Ms, Mrs etc) to reset their gender setting, and then put the keyboard down and went…

I think you made the wrong call here.

Re: The day my script killed 10k phones in South America

#29

Post by throwdecro »

heavenlyblue wrote: post #13
quenix wrote: post #2

This feels incredibly negligent. It seems obvious that generating thousands of random phone numbers and locking them without any other checks would lock some real phones. I'm puzzled that the author did not consider this when writing the script, let alone testing it or running it on live prod servers.

I don't understand how people can defend this developer in the first place. I could understand if they were about 16 years old and actually abused by their employer, but generating random numbers to test the locking of phones? Basic maths should tell you that the probability of matching some real phone number is actually quite high. Moreover, what exactly were they testing? i.e. if they were testing nonexistent numbe…

Being somewhat generous, I don't think it's because they didn't understand the probabilities, it's because they didn't think about the impact of what they were doing at all, beyond the problem they needed to solve.

I.e., the only thing they were considering was whether the number entered at the start of the test would show up in the right places at the other end of their system. They simply didn't consider that an actual phone would be shut off along the way.

Post reply on HN