Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The day my script killed 10k phones in South America

new.pythonforengineers.com

Post reply on HN
Print view View original

101–110 of 164 posts

Re: The day my script killed 10k phones in South America

#101

Post by JanisL »

nso95 wrote: post #79
JanisL wrote: post #60

I've noticed more as time has gone on people are working on critical software without the skills or organizational structures required to actually do so safely. What I'm wondering is if this is a change that's happened more broadly or is just something that I didn't notice before? Overall the average years of experience in the industry has dropped in the last decade due to the influx of new developers but what's the…

This guy wasn't inexperienced though

It wasn't so clear but I was mostly thinking of the management when I wrote that comment. In this particular case I think people might get caught up on what an individual did but I think the management failing here is the far bigger deal.

Re: The day my script killed 10k phones in South America

#102

Post by tgsovlerkhgsel »

matheusmoreira wrote: post #76
shantnutiwari wrote: post #45

Author here. I shared this yesterday, didnt get much upvotes and quickly vanished. Thanks to /u/nixcraft for sharing it again! Looks like 2nd time lucky. Im seeing the same type of comments here and on Reddit, I'll try to answer a few common ones. 1. Yes, I know it was stupid testing on production. But we'd been told we needed to release the product on Monday (this was Friday), no objections. A previous project manag…

> If you get a phone on contract, you don't actually own it till you pay it off, and the company is entitled to lock it. No. You don't turn off somebody's phone while they're using it. You have no idea what they could be doing. Someone could be literally dying in front of your locked out consumer who is now unable to call for help. This is vital infrastructure and you're denying them access. If somebody dies because…

> Would you randomly disable somebody's car's engine because they didn't pay?

I believe that's exactly what some systems for subprime car loans do (they don't shut it down while running, but will prevent it from starting - if they're somewhat clever then only if the car has been off for a certain amount of time).

Re: The day my script killed 10k phones in South America

#103

Post by tgsovlerkhgsel »

jfrunyon wrote: post #91
tim333 wrote: post #36

Earlier quoted context omitted.

Not usually the case under UK law at any rate.

Huh? So you're telling me that I can walk into the phone shop, get them to give me a phone with financing, and then not pay it, and... still own it? Sweet deal!

Yes, just like you still own the phone if you take out a bank loan, use it to buy the phone, and then default on the loan.

Re: The day my script killed 10k phones in South America

#104

Post by RHSeeger »

cycomanic wrote: post #15
quenix wrote: post #2

This feels incredibly negligent. It seems obvious that generating thousands of random phone numbers and locking them without any other checks would lock some real phones. I'm puzzled that the author did not consider this when writing the script, let alone testing it or running it on live prod servers.

Too me this is another example showing how much behind "software engineering" is behind most otherl engineering disciplines. Processes in other disciplines are build very much into the practice that these things don't happen because people die if a plane crashes or a bridge collapses. Also that somehow software gets away with absolving itself of any liability (i.e. the usual "this software is provided as is...") is q…

People die all the time because things for buildings are changed "in production". It's common for the builder (or supplier... the people in charge of taking the engineers plans and actually building the thing) to swap out parts at their own discretion. And there's tons of stories of lots of people dying because of it.

There's also stories of architects designing buildings using "new and interesting techniques" with little to no testing. There's a story of one building that was lighting things across the street on fire because it was basically a big parabolic mirror.

And.. well.. the Boeing 737 Max.

So sure, engineers and related jobs are _better_ about it, but they're certainly not perfect. The still do stupid things. It's just less likely because the fields have been around for a long time and there's been time to learn the right ways to do things... and to learn the things that get people killed.

As the expression goes, regulations are written in blood... because new regulations come about when people die because of a gap.

Re: The day my script killed 10k phones in South America

#105

Post by marcosdumay »

aunty_helen wrote: post #96
fragmede wrote: post #74

Earlier quoted context omitted.

Because (pardon my language) fuck that business model, that's why. It's reprehensible on a number of different levels, and specific aspects of that have been made illegal, but companies keep trying to bring back the exploitation. Just because there's a business model doesn't mean that we have to support it, or help it be implemented. We can shame our fellow developers choosing to enable scams and write malware/comput…

Put frankly, you're out of touch with how south america works. Things you would never expect can be paid in installments here. People just don't have heaps of disposable cash. There's litterally lines at the atm 2 times a month, pay day. The rest of the time people are day to daying without cash in their pocket.

The things that you pay in installments do not suddenly and simply die if you miss a payment.

There are right and wrong ways to go collect payment, and this one is a wrong one.

Re: The day my script killed 10k phones in South America

#106

Post by marcosdumay »

tim333 wrote: post #36
shantnutiwari wrote: post #24

Earlier quoted context omitted.

The product was for phones that had not been paid off-- if you get a phone on a 2 year contract, you dont "own" the phone until it is paid off

Not usually the case under UK law at any rate.

Not under Brazilian law either. I wonder where in South America those phones are.

Re: The day my script killed 10k phones in South America

#107

Post by jayzalowitz »

zarzavat wrote: post #97
kingcharles wrote: post #20

Earlier quoted context omitted.

I once ran a query on a live production web site thinking I was doing a SELECT but somehow ran an UPDATE which set all the millions of users' gender to 'M', regardless of their selection. Restoring from backups would have been a nightmare. And telling my supervisor would have been a nightmare. So I used everyone's selected title (Mr, Ms, Mrs etc) to reset their gender setting, and then put the keyboard down and went…

What did you do about 'Dr's?

Doctors arent people, they are student loan debt... not neccesary.

Re: The day my script killed 10k phones in South America

#108

Post by systemvoltage »

macintux wrote: post #59
shantnutiwari wrote: post #45

Author here. I shared this yesterday, didnt get much upvotes and quickly vanished. Thanks to /u/nixcraft for sharing it again! Looks like 2nd time lucky. Im seeing the same type of comments here and on Reddit, I'll try to answer a few common ones. 1. Yes, I know it was stupid testing on production. But we'd been told we needed to release the product on Monday (this was Friday), no objections. A previous project manag…

> Including my boss, who was fired when he was attending his mother's funeral. And people wonder why big companies are generally loathed.

This is a false generalization. It’s simply not true that this happens everywhere in big companies. Seems like a rare case.

Re: The day my script killed 10k phones in South America

#109

Post by distantaidenn »

As others have pointed out, I wouldn't even have created this article, but to each their own.

I'm assuming this client company was Samsung, due to scale and the author calling it the "K-Pop" company. They are known for using idols in their marketing. Still a little cringe to call it that.

My take away from this article:

1. This act was extreme, bordering on criminal, negligence. Some peoples' livelihoods depend on access to their phone. I assume most people using this service are of a lower income. I wonder how many jobs may have been lost due to their not having access to their mobile for a day.

2. Samsung needs to hire better contractors -- but this is what happens when companies go with the lowest bidder, I suppose. I've seen it time and time again.

Re: The day my script killed 10k phones in South America

#110

Post by innodb1 »

Importance of testing has been told multiple times,even though we all ignore it largely.

I vividly remember an incident ( this was in the early 2000's )

we had a couple of Linux machines both remote and local. The testing and development happens in Local and the code is pushed in a short time over dialup connection.

One of my team member had a instance of remote open.

He was deleting some cache files in his local , he by mistake typed in rm -rf / instead of rm -rf * in the remote machine ,assuming it was his local development machine.

He didn't realize the mistake and finding it was taking much longer than expected he cancelled the command. went home happily. He was using a common login. Though he informed his lead , the tech lead also didn't realize the seriousness of the command that moment.

What actually happened was every file which had 777 permission ,got deleted till he pressed Crtl-C.

Now some of the files in production server was gone and some were still intact.

This led to lots of confusion , the person who issued the command didn't realize his mistake , till the server admin who was about to loose his job for faulty permissions some how convinced the management that someone had issued a wrong command , and that he was somehow not at fault.

Our Boss was super irritated when he found out what happened. The person who typed the wrong command , along with his tech lead left the job after a few months on their own.

It taught very valuable lessons, Especially not to type rm -rf without checking multiple times.

I still remember this incident every time I delete something.

Post reply on HN