Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Weirdest hack that you ever saw in production?

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 289 posts

Re: Ask HN: Weirdest hack that you ever saw in production?

#81

Post by cagataygurturk »

Year 2006, we had a very high-traffic website running with 1 MySQL server and 1 web server (PHP). Maybe high availability or resillience terms were not coined yet, that's why we were comfortable with having one server per each function. Web server had two ethernet cards, one is with private IP and one is publicly accessible IP. After a while, the platform started to crash and I would be called by my loyal users before Pingdom alerts reach to me, then I would call the datacenter technicians to press restart button of the web server. Obviously it was a lengthy process for recovery, with a lot of human involved.

After a while, I discovered that the issue was about web server's ethernet card attached to internal network and used to connect MySQL server. When that ethernet card stops working, the platform would crash. On the other hand, it was also possible to connect to MySQL using the other ethernet card via public IP. It would reduce the performance of the platform, since all the bandwidth of that card (100 mbps!) is already eaten by HTTP traffic, but at least it would keep it running.

I ended up writing a script at my home computer, checking if the platform is up or not. Once the faulty ethernet card fails, it would connect to FTP, change PHP configuration to use the other ethernet interface to connect to MySQL server, and send an e-mail to datacenter technicians to press restart button.

This script successfully did its job during 3 months, until I eventually replaced the faulty ethernet card and fixed the issue.

Isn't it "Invent and Simplify" like Jeff Bezos says?

Re: Ask HN: Weirdest hack that you ever saw in production?

#82

Post by Kubi »

Big C codebase. To be more precise, they said it's C++, but as far as I could see, it was C compiled with g++.

Some code read xml data. Instead of choosing one of the xml-parsers available, author decided to write another one. Instead of using C++ features, atoi() used. For empty strings, atoi() got NULL and segfaulted. Signal 11 has been handled and suppressed in order to avoid crashes. Certainly, the code had other segmentation faults too, which could not been discovered this way. :)

Re: Ask HN: Weirdest hack that you ever saw in production?

#83

Post by nailer »

orf wrote: post #2

Finance needed to do end-of-year stuff a couple of days past end-of-year. The system couldn't handle this, bad things would happen and data would change once end-of-year passes. Solution? A bash script that does: while true: set date to 4pm end-of-year sleep 1

Why the loop, vs setting the date once?

Re: Ask HN: Weirdest hack that you ever saw in production?

#84

Post by mi3law »

In a consumer app, I would say Snapchat's early camera hack on Android takes the cake.

To be brief, their app ran the Android native camera app in the background and took a screenshot of the resulting feed for the image, bypassing actual integration with Android's camera apps. Having worked on an Android smartphone from the ground up, I can understand their reluctance to commit dev time to having to support so many Android versions and other variations on all the devices out there, but still a lazy weird hack.

https://android.gadgethacks.com/how-to/fyi-why-androids-snap...

https://www.reddit.com/r/GooglePixel/comments/64xqv0/snapcha...

Re: Ask HN: Weirdest hack that you ever saw in production?

#85

Post by orf »

nailer wrote: post #83
orf wrote: post #2

Finance needed to do end-of-year stuff a couple of days past end-of-year. The system couldn't handle this, bad things would happen and data would change once end-of-year passes. Solution? A bash script that does: while true: set date to 4pm end-of-year sleep 1

Why the loop, vs setting the date once?

NTP would resync it, and obviously after X hours it would no longer be end of year even if you set the date in the past.

It needed to be end of year day for 2 or 3 days.

Re: Ask HN: Weirdest hack that you ever saw in production?

#86

Post by sokoloff »

Gas boiler in an old building with a safety mechanism that would trip and latch/lockout sometimes in high wind conditions, requiring a manual power cycle to restore heat.

Solution: Arduino and a relay in-line to cut the power for 1 minute every 120 minutes.

Planned to add a DS1820 to only cut power when required, but never got around to it.

Re: Ask HN: Weirdest hack that you ever saw in production?

#88

Post by ComputerGuru »

Kubi wrote: post #82

Big C codebase. To be more precise, they said it's C++, but as far as I could see, it was C compiled with g++. Some code read xml data. Instead of choosing one of the xml-parsers available, author decided to write another one. Instead of using C++ features, atoi() used. For empty strings, atoi() got NULL and segfaulted. Signal 11 has been handled and suppressed in order to avoid crashes. Certainly, the code had other…

You mean, instead of fixing _just_ the atoi() crash, that developer fixed all crashes with his patch? Quite the clever bastard!

Re: Ask HN: Weirdest hack that you ever saw in production?

#89

Post by technofiend »

mattzito wrote: post #19

In my startup days, we were working on a proof of concept with a really big bank. Because of their security rules, we couldn't have direct access to their systems - so if we wanted to do something remotely, we would have to start a webex, they would join and share their screen, and give us remote control. This worked great, except if we wanted to work over the weekend, since if we left the screen alone for more than…

This seems to be a very common problem as evinced by the many solutions below. A friend of mine asked me solve the problem for him because he was remotely accessing a database and the company-issued laptop, VPN client, remote server and database server all had aggressive timeouts. Getting a cup of coffee meant logging in to everything again using long and complex company-issued passwords.

We used one of the common Raspberry PI Human Interface Device (HID) Python packages to send a harmless keyboard or mouse event once every 5 minutes.

Re: Ask HN: Weirdest hack that you ever saw in production?

#90

Post by evgen »

nailer wrote: post #83
orf wrote: post #2

Finance needed to do end-of-year stuff a couple of days past end-of-year. The system couldn't handle this, bad things would happen and data would change once end-of-year passes. Solution? A bash script that does: while true: set date to 4pm end-of-year sleep 1

Why the loop, vs setting the date once?

I would guess the system was probably running ntpd or some other time sync service that they were unwilling or unable to turn off.
Post reply on HN