Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The day my script killed 10k phones in South America

new.pythonforengineers.com

Post reply on HN
Print view View original

121–130 of 164 posts

Re: The day my script killed 10k phones in South America

#121

Post by VortexDream »

musiccog wrote: post #118
innodb1 wrote: post #110

Importance of testing has been told multiple times,even though we all ignore it largely. I vividly remember an incident ( this was in the early 2000's ) we had a couple of Linux machines both remote and local. The testing and development happens in Local and the code is pushed in a short time over dialup connection. One of my team member had a instance of remote open. He was deleting some cache files in his local , h…

> It taught very valuable lessons, Especially not to type rm -rf without checking multiple times. "Measure twice, cut once." One of the many uses I have for a mindmap in my daily use (Freemind 0.9.0!), is as a place to write out sysadmin/sql commands in a structured way before executing them. If I can test it on dev before production - great. If I have to run it directly on production, following this method (religiou…

Can you maybe go into a bit more detail about how you use Freemind? It's not something I've heard somebody do before and it sounds interesting.

Re: The day my script killed 10k phones in South America

#122

Post by VortexDream »

polartx wrote: post #71
Iolaum wrote: post #46

Earlier quoted context omitted.

Author says in the article they had a 3 day deadline to finish the product and needed to test thousands of cases while already having pulled late nights to keep up with schedule.

“To the tens of thousands of people whose phones were suddenly and inexplicably bricked: my bad. You see, it was almost the weekend and although I had 3 days to responsibly approach this problem, I had already pulled several late nights, I’m sure you understand”

That's not the point. It's just not sensible to blame this one developer for a horrible environment in which this was the only option aside from getting fired, which is quite onerous a responsibility to put on a single person. This falls squarely on management.

Re: The day my script killed 10k phones in South America

#123

Post by vodkapump »

saithir wrote: post #115
whitexn--g28h wrote: post #56

+1(XXX)555-XXXX The 555 prefix for US phone numbers is reserved for fake phone numbers in the US and elsewhere. Use this block for testing.

Which would be excellent advice if he was using phone numbers. Which the phone's IMEI isn't.

> Remember I told you the phones were "randomly" generated? The python script would randomly generate an 11 letter "phone number".

Author says "phone number" and 11 letter(?) IMEI is 15 digits, so who even knows what the author means at this point.

Re: The day my script killed 10k phones in South America

#124

Post by hoseja »

shantnutiwari wrote: post #45

Author here. I shared this yesterday, didnt get much upvotes and quickly vanished. Thanks to /u/nixcraft for sharing it again! Looks like 2nd time lucky. Im seeing the same type of comments here and on Reddit, I'll try to answer a few common ones. 1. Yes, I know it was stupid testing on production. But we'd been told we needed to release the product on Monday (this was Friday), no objections. A previous project manag…

>I was just following orders!

Re: The day my script killed 10k phones in South America

#126

Post by sundvor »

thih9 wrote: post #25

> Except for the poor sod in Peru who couldn't make phone calls or post duck-face pictures to Instagram, but no one asked him. There are countless situations where suddenly not having a working phone means a serious problem. The article presents an extremely negligent look at the issue.

There are indeed large ranges of highly valid reasons why having phone/ Internet connectivity is highly critical.

But sure, "post duck photos". I understand it was perhaps meant as a paraphrase, yet it is indicative of an almost callous level of arrogance at the author's company.

Re: The day my script killed 10k phones in South America

#127

Post by felipelemos »

jfrunyon wrote: post #86
matheusmoreira wrote: post #76

Earlier quoted context omitted.

> If you get a phone on contract, you don't actually own it till you pay it off, and the company is entitled to lock it. No. You don't turn off somebody's phone while they're using it. You have no idea what they could be doing. Someone could be literally dying in front of your locked out consumer who is now unable to call for help. This is vital infrastructure and you're denying them access. If somebody dies because…

1) I highly doubt that the app disables emergency calling. 2) Some (rental, probably also lease) vehicles actually do have disablers. They prevent the vehicle from being started, not stop them in the middle of the road. 3) The police don't repo a thing. That's a civil matter and generally outsourced to the lowest bidder with a tow truck. By the way, if someone dies because of it, the only person whose conscience it i…

> By the way, if someone dies because of it, the only person whose conscience it is on is the person who bought a phone they couldn't afford and then didn't pay for it. Just like if someone died because the car got repo'd and they couldn't drive to a hospital.

Sure. What they were thinking trying to buy something without having money, right? And even worse, what they were thinking being poor. They should have bought a very shit phone, closer to what they can afford. And if they lost their job and then couldn't pay for the phone, it's their own fault. /s

Re: The day my script killed 10k phones in South America

#128

Post by lifthrasiir »

Back in time I was working on an online trading card game, the integration test was running in the live server. (Don't ask me why. All I can say is that the server had to be hastily built for some reason and it was known to be suboptimal in several aspects.) In order to avoid inference the test created tons of placeholder accounts and corresponding environments and it actually ran mostly well given the circumstances.

One day I received a report that several accounts were simply disappearing and had to be re-registered every few days or so. I had to go through lots of recorded logs to determine what was happening: one of these accounts was named "Card" and some test was using it as a placeholder account, so it got removed when the test finished. How apt it is, that someone would use "Card" or similarly rare handles in a TCG. The testing infrastructure was updated to ensure that placeholder accounts can't coincide with actual accounts and affected users were compensated.

Re: The day my script killed 10k phones in South America

#129

Post by dotancohen »

solarized wrote: post #125

I always love when "TIL things" like this getting popular. Many peoples still threat it as a disgrace. Whereas other peoples really need it as a priceless lesson learned.

I love learning from mistakes. Particularly from other peoples' mistakes.

Articles like this are invaluable in high-technology fields, where one cannot afford the time nor consequences of making every possible mistake.

Re: The day my script killed 10k phones in South America

#130

Post by Arnavion »

trutannus wrote: post #90
thih9 wrote: post #25

> Except for the poor sod in Peru who couldn't make phone calls or post duck-face pictures to Instagram, but no one asked him. There are countless situations where suddenly not having a working phone means a serious problem. The article presents an extremely negligent look at the issue.

> extremely negligent look at the issue Yes, the part about them overwriting the data from each cycle struck me as odd. I don't think that's good practice even if testing.

It doesn't seem unusual to me. You write a test script that generates random numbers and saves them to foo.txt, then reads them back from foo.txt for the purpose of running one kind of test. Then you wrap that in a loop for every kind of test. You end up testing everything, but foo.txt only has the numbers from the last test.

To be clear, I'm not saying it's not a bad practice to do that. But the context is that it was a one-off job to do the testing, not a proper test framework, for which this kind of half-assed approach is reasonable.

Post reply on HN