Hey folks - we're back in business. Clearly Dax the Duck had a case of the Mondays!
I'd love to read a post-mortem.
Sometimes ducks just have a bad day.
DuckDuckGo was down
81–90 of 103 posts
Re: DuckDuckGo was down
#82Earlier quoted context omitted.
Do they still use Bing ? I thought DDG had in-house search db.
The Duck Duck Bot crawls the web in some fashion but it is my understanding most of what's in their search result comes from their Bing indices. Correct me if I'm wrong.
Afaik, it's still mostly from Bing.
Re: DuckDuckGo was down
#83All I know is that I would not want to be the one who had last committed to production.
Unlikely it is their logical app code , at this scale you likely will do canary releases or similar , failing for all users should be super rare It perhaps is related to devops or the search index cache servers is broken
Or a bad config. 90% of the time it's a bad config.
Re: DuckDuckGo was down
#84Hey folks - we're back in business. Clearly Dax the Duck had a case of the Mondays!
The Mondays? No, man. I believe you'd get your ass kicked sayin' something like that, man.
Re: DuckDuckGo was down
#85All I know is that I would not want to be the one who had last committed to production.
Good teams don't blame individuals. You can praise individuals, but you take blame as a team.
Hopefully at least two people reviewed the commit before it was put into production. Someone set up the post-launch smoke-tests/api-test-suits. Someone built the CI/CD-system. Etc etc.
My point is, it is rarely a single individuals fault when something at this scale goes down.
Re: DuckDuckGo was down
#86All I know is that I would not want to be the one who had last committed to production.
No, because,
* You probably got a review on your change before it was submitted, so you weren't the only one to think it was an ok change to make
* If you "revert first, ask questions later", you might also be the first one to fix it
* Hopefully the team recognizes that there are higher-order causes that caused your submit to break things than your own personal skill
Re: DuckDuckGo was down
#87All I know is that I would not want to be the one who had last committed to production.
Good teams don't blame individuals. You can praise individuals, but you take blame as a team. Hopefully at least two people reviewed the commit before it was put into production. Someone set up the post-launch smoke-tests/api-test-suits. Someone built the CI/CD-system. Etc etc. My point is, it is rarely a single individuals fault when something at this scale goes down.
>Good teams don't blame individuals. You can praise individuals, but you take blame as a team.
I could not agree more with this statement.
Re: DuckDuckGo was down
#88Earlier quoted context omitted.
The Mondays? No, man. I believe you'd get your ass kicked sayin' something like that, man.
One time I was at a festival where it wouldn't stop raining. It had been a shit show. Come Monday, all the cars were stuck in mud. Then there were tornado warnings. Then they let down a damn up river and the whole area started to flood. Whilst holding down a shade structure, waiting for updates on the tornado path, and standing in slowly rising water, someone made the joke: "Mondays"
Ha. Reminds me of Norm Macdonald’s “Real Jerk” joke.
Re: DuckDuckGo was down
#89All I know is that I would not want to be the one who had last committed to production.
Good teams don't blame individuals. You can praise individuals, but you take blame as a team. Hopefully at least two people reviewed the commit before it was put into production. Someone set up the post-launch smoke-tests/api-test-suits. Someone built the CI/CD-system. Etc etc. My point is, it is rarely a single individuals fault when something at this scale goes down.
I also think a lot of praise should generally be a team thing too (but I also love this: "You can praise individuals, but you take blame as a team.")
When discussing success to an outside group (giving a presentation or something) or higher up (especially here!) you always use "we". Even if it is a section that only you worked on or you did most of the work. You can often pick out good managers by just seeing if they talk like this, if their focus is around what they individually did or what their team accomplished.
My point is, it is rarely a single individual's success when things work.
Re: DuckDuckGo was down
#90I wonder if they have a status page?
Ironically, the best way to find it would be to use a search engine.