Live data from Hacker News

The day my script killed 10k phones in South America

new.pythonforengineers.com

41–50 of 164 posts

Re: The day my script killed 10k phones in South America

#42
post #2

This feels incredibly negligent. It seems obvious that generating thousands of random phone numbers and locking them without any other checks would lock some real phones. I'm puzzled that the author did not consider this when writing the script, let alone testing it or running it on live prod servers.

I don't understand how people can defend this developer in the first place. I could understand if they were about 16 years old and actually abused by their employer, but generating random numbers to test the locking of phones? Basic maths should tell you that the probability of matching some real phone number is actually quite high. Moreover, what exactly were they testing? i.e. if they were testing nonexistent numbe…

I pass no judgment, but merely take the developer at his word: https://new.pythonforengineers.com/blog/confessions-of-a-1x-...

Re: The day my script killed 10k phones in South America

#43
Putting aside the issue of locking real phones this has an issue of state persisting between test runs. The previous execution of a test should not change state for subsequent execution.

This is a long way of asking why the phones weren't unlocked as part of a cleanup step in the same test. Although, at this company, if they did this, they might not notice they were locking and unlocking phones whenever this ran and then periodically disable and enable their customers phones.

Re: The day my script killed 10k phones in South America

#45
Author here. I shared this yesterday, didnt get much upvotes and quickly vanished. Thanks to /u/nixcraft for sharing it again! Looks like 2nd time lucky.

Im seeing the same type of comments here and on Reddit, I'll try to answer a few common ones.

1. Yes, I know it was stupid testing on production. But we'd been told we needed to release the product on Monday (this was Friday), no objections. A previous project manager had been fired for not being fast enough, and the whole test department was at risk of redundancy. Vice president level executives were asking for updates in daily standups.

Like I say in the post, sometimes, you do what you're told or you find another job. If it hadnt been the height of Covid , I might have chosen option 2.

2. Why didnt we test on a staging / dev server? We did have those, but they didnt work with the K-Pop companies servers. And we didnt have time (or enough people) to make it work. Lots of other engineers decided to just quit. Besides, we never thought K-pop would actually lock phones we didn't own.

3. Some people have asked about it: This type of app is perfectly legal. If you get a phone on contract, you don't actually own it till you pay it off, and the company is entitled to lock it. This is builtin on iPhones, Android need external apps to enforce this.

4. While it may seem stupid in retrospect (LOL, this guy calls himself an engineer, tests in production)-- at that time, it was just one firefight after another, and there was no time to think or plan. I can joke about it now, but at the time, it was one of the worst pressure cooker environments.

Some updates: After we released the 3 products, the whole QA/test department was made redundant. Including my boss, who was fired when he was attending his mother's funeral. Funnily enough, I stayed because I knew details of a legacy product. So far the last 3-4 weeks, I had no boss or anyone to report to.

Re: The day my script killed 10k phones in South America

#46
post #10

Earlier quoted context omitted.

More specifically for the deadline they are imposing without asking for feedback about feasibility and associated risks.

The deadline for generating a bunch of random phone numbers to block? The author would probably save some time by hard-coding some truly non-existent phone numbers instead of using a random number generator.

Author says in the article they had a 3 day deadline to finish the product and needed to test thousands of cases while already having pulled late nights to keep up with schedule.

Re: The day my script killed 10k phones in South America

#47

If you haven't killed some production environment in your career, can you really call yourself a programmer? I kid of course, but I've been there, done that. It's lessons learned the hard way and I became a better developer because of it.

Oh yes. I've accidentally run a SQL UPDATE command without a WHERE clause and blew away everyone's password. Our process to restore a backup was very manual and time consuming back then, so it was fun sitting around and waiting for that to happen while everybody was locked out of the application.

Re: The day my script killed 10k phones in South America

#48
post #37
post #28

Earlier quoted context omitted.

I think you made the wrong call here.

On the plus side, why are they storing gender info in the first place?

Why would you jump to the assumption that they don't need to store gender?

Re: The day my script killed 10k phones in South America

#49

If you haven't killed some production environment in your career, can you really call yourself a programmer? I kid of course, but I've been there, done that. It's lessons learned the hard way and I became a better developer because of it.

Ah, that feeling of dread in your stomach as the realization hits.
Post reply on HN