Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is your failure story?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 48 posts

Re: Ask HN: What is your failure story?

#21

Post by crimsonalucard »

kadendogthing wrote: post #11
ideonexus wrote: post #8

I was responsible for a software bug that forced the entire United States Coast Guard to revert to using paper records to track all of their Aviation Logistics for an entire month. The bug was in an aircraft maintenance dashboard that had over a dozen filters on it. It passed a thorough round of QAing, but once in production it was found that a specific selection of filters would create an infinite loop in the report…

Huge kudos to your manager for standing up like that. Companies often look for a single point of failure (it's only human), instead of admitting that the organization as a whole systematically failed to prevent the error, let alone discover it "in time." I hate to use platitudes but companies are only really as strong as their weakest link (not to impugn your skills or anything like that). Letting a single failure si…

You also need to be a person worth fighting for. It goes both ways.

Re: Ask HN: What is your failure story?

#22

Post by tabtab »

lelima wrote: post #7

Once I scale up Azure analysis services(main data source for company Marketing and decision making) cause we were running low memory in peak hours, only to find out that I couldn't scale down, increasing cost by 300%. I had to recreate all the models again in the old tier increasing even more the cost cause we have to maintain both data sources till I finish with the creation of the models. My boss was mad.~azure lvl…

Here's my most memorable Microsoft snafu: My desktop was just upgraded to Windows 10. The "Favorites" links in File Explorer were converted into "Quick Access" links by the Microsoft migration process, but the aliases (titles) of the links had changed, or so I thought. I renamed the aliases back to what they were under Windows 8.

The phone started ringing with complaints, and to my horror I learned that I inadvertently renamed widely-used WAN and intranet folders. Quick-Access had no aliases: I had inadvertently changed the actual folder names. A colleague walked by and asked, "Why are you sweating and white like a ghost?" Fortunately I renamed them back before too much damage was done.

Re: Ask HN: What is your failure story?

#23

Post by deanmoriarty »

tracer4201 wrote: post #20

Not a complete failure story, but I joined a FANG company as an entry level engineer. In about three and a half years, I’ve gone through two promotions. My most recent team where I got promoted to senior engineer is demoralizing to me and gives me anxiety even thinking about work. I’m feeling a mixture of burnout and just a general disinterest in my teams domain area. My current team has also grown in ways I didn’t e…

Thanks for your post.

I am currently at a startup and have been here a few years. I came to really hate the environment, mostly because the people this company can hire are not what I would personally consider talented (due to a mix of non-exponential company growth and lower compensations than FAANG companies). The result of hiring low quality professionals is that the engineering standards are so low that it really depresses me.

For this reason, I've interviewed and got offers with 3 FAANGs, and I am contemplating about moving to one of them. After reading your post, I recognize I am perhaps romanticizing FAANGs a little bit more than what's fair, but the thing that intrigues me the most is the ability to work with and learn from other very talented engineers (so I can grow), that I have a hard time finding in startups that don't have big pockets as them.

And of course, also the financial aspects, since financial freedom is relatively high in my priority list.

Re: Ask HN: What is your failure story?

#24

Post by allwynpfr »

muzani wrote: post #13

0. Graduated from a top CS uni. Starting salaries were lower than KFC assistant manager. Started looking outside tech to for enough income to sustain being a programmer. 1. Put down $6000 on a coffee kiosk. Made real good money every day. But partners were all dishonest and manipulative. 2. Built an app for a guy. He paid well. Built another app for him. Contract sucked, one clause meant we couldn't get paid even tho…

Duuudeeeeee this was like watching a movie! You should totally turn this into a blog, perhaps even a book I bet you learnt lots along the way, you're literally the poster boy/girl for try, fall down, stand up, try Thanks for this :)

Re: Ask HN: What is your failure story?

#25

Post by throwawaymlc »

tracer4201 wrote: post #20

Not a complete failure story, but I joined a FANG company as an entry level engineer. In about three and a half years, I’ve gone through two promotions. My most recent team where I got promoted to senior engineer is demoralizing to me and gives me anxiety even thinking about work. I’m feeling a mixture of burnout and just a general disinterest in my teams domain area. My current team has also grown in ways I didn’t e…

I am in a very similar situation, 4.5 years in at a large / prestigious organization where I was thinking there was a lot of scope for progression and I could stay long term.

First 3 years were great, good work and appraisals, I learnt a lot. After that, things deteriorated quite suddenly, a combination of some urgent but poorly conceived projects, passive management, too much time in meetings, a few layers of extra paperwork / approvals introduced.

Basically, the work environment really did go downhill, but in hindsight I didn't deal with that very well or put together a plan to move.

I would suggest you come up with a plan to bring this to a head sooner rather than later. Try to do an excellent job with your status reports and pointless meetings, so that you can transfer or get a good reference from your manager, and leave on a high note. At the same, research the market for your next job, pick up any missing skills, and prepare for a move on your own terms and schedule. Don't let the situation fester without doing anything about it.

Re: Ask HN: What is your failure story?

#26

Post by tracker1 »

Once worked for a company that had a subscription based service. They wanted a monthly report of expired accounts so that they could warm-call for renewals. The bug was that some accounts had paid for 2 years. After a couple years, someone noticed that the reports hadn't been including the 2-years subscriptions and it was effectively a loss of a couple million in revenue.

While I did code it to the specifications I was given, and was really clear on understanding it was still a pretty big failure.

-----

Second biggest failure was checking in the db connection against a production database into source control. It was internal source control, and the transforms (ASP.Net Web.config transforms) had it anyway, so wasn't an increased risk in that regard. I was troubleshooting a production issue.

What happened is another developer had used that connection configuration to drop/recreate a table. It took a few hours to recover from (DB restore, then replay against API activity logs). And a day of lost time for a few hundred workers.

Re: Ask HN: What is your failure story?

#27

Post by adamredwoods »

In the pre-HTML5 days of the Internet, I wanted to be a Flash Designer. Forever. I loved the platform and process so much, that I only had stars in my eyes for Flash! The idea of being able to blur the lines of animation, programming and design was wonderful. Then it died (April, 2010 https://www.apple.com/hotnews/thoughts-on-flash/) and so did my career.

Fortunately, in most failure stories we can find some ways to recover. Knowing Actionscript developed some programming skills, which were applicable to HTML5.

My key lesson is learning how to pivot from one technology platform to another. It's not always easy, requires some humility, but it's possible.

Re: Ask HN: What is your failure story?

#28

Post by ThrowawayR2 »

deanmoriarty wrote: post #23
tracer4201 wrote: post #20

Not a complete failure story, but I joined a FANG company as an entry level engineer. In about three and a half years, I’ve gone through two promotions. My most recent team where I got promoted to senior engineer is demoralizing to me and gives me anxiety even thinking about work. I’m feeling a mixture of burnout and just a general disinterest in my teams domain area. My current team has also grown in ways I didn’t e…

Thanks for your post. I am currently at a startup and have been here a few years. I came to really hate the environment, mostly because the people this company can hire are not what I would personally consider talented (due to a mix of non-exponential company growth and lower compensations than FAANG companies). The result of hiring low quality professionals is that the engineering standards are so low that it really…

The FAANG companies are not monolithic hiveminds. They are huge places where some parts are good and some are ... not so good. There are talented engineers and opportunities for growth at the good teams but you're not guaranteed to be in a good team at any given time.

I would say go for it but go in with a mission to understand the lay of the land and then transferring to a team that matches your growth goals.

Re: Ask HN: What is your failure story?

#29

Post by kadendogthing »

crimsonalucard wrote: post #21
kadendogthing wrote: post #11

Earlier quoted context omitted.

Huge kudos to your manager for standing up like that. Companies often look for a single point of failure (it's only human), instead of admitting that the organization as a whole systematically failed to prevent the error, let alone discover it "in time." I hate to use platitudes but companies are only really as strong as their weakest link (not to impugn your skills or anything like that). Letting a single failure si…

You also need to be a person worth fighting for. It goes both ways.

Presumably that's what the hiring process is for.

Re: Ask HN: What is your failure story?

#30

Post by souprock »

deanmoriarty wrote: post #23
tracer4201 wrote: post #20

Not a complete failure story, but I joined a FANG company as an entry level engineer. In about three and a half years, I’ve gone through two promotions. My most recent team where I got promoted to senior engineer is demoralizing to me and gives me anxiety even thinking about work. I’m feeling a mixture of burnout and just a general disinterest in my teams domain area. My current team has also grown in ways I didn’t e…

Thanks for your post. I am currently at a startup and have been here a few years. I came to really hate the environment, mostly because the people this company can hire are not what I would personally consider talented (due to a mix of non-exponential company growth and lower compensations than FAANG companies). The result of hiring low quality professionals is that the engineering standards are so low that it really…

FAANG companies are not where you go to find high engineering standards. You need to look for a job featuring one of these:

DO-178C DAL A (if you mess up, aircraft crash)

EAL5, EAL6, EAL7 (if you mess up, national secrets leak)

MISRA C (if you mess up, cars crash and a giant recall is required)

IEC 62304 Class C (if you mess up, medical devices kill people)

NUREG/CR-6463 (if you mess up, you get a nuclear disaster)

ISO 9000 and ISO 9001 (if you mess up, it will at least be documented)

If you didn't really mean high engineering standards, and you just want to work with capable people, you can find that all over the place. Perhaps being within commute distance of a FAANG has an impact, and you are seeing that due to your location. There are many people who refuse to live anywhere near a FAANG and/or wouldn't tolerate the culture.

Post reply on HN