501–510 of 526 posts
Two more problems identified solely from the screenshot: * you have two competing subscription id columns. * a uuid is not a string, it is a 128 bit integer. If your database limits to 64 bit integers then use a 64 bit integer for the id instead of a string, or use an array of 128 bytes.
Many ORMs represent UUID as a string.
Earlier quoted context omitted.
Get into the habit of colour-coding your important SSH sessions, red/green/blue can give you a very powerful subconscious check before doing something very silly.
How do you do this so that it lasts? I find mine resets to default
https://bsago.me/tech-notes/change-ssh-background-colour-wit....
No, a lack of monitoring cost you $10K. Your app was throwing a database exception and nobody was alerted that this was not only happening, but happening continuously and in large volumes. Such an alert would have made this a 5-minute investigation rather than 5 days. If you haven't fixed that alerting deficiency, then you haven't really fixed anything.
Maybe ChatGPT didn't tell them that they needed monitoring?
Earlier quoted context omitted.
Looking at this team's project at github.com/reworkd, it clearly tells the maturity of the product as well as the team. Emoji driven development. Emoji's for all commit messages. Monkeys, bananas, rockets, fireworks, you name it, they have it in their commit message.
Commit messages are overrated.
1. Forward thinking
2. Maturity
3. Temperance
All important traits in engineers and founders
Earlier quoted context omitted.
I’ve seen two types of people approach problems: Type 1 tries to find the error message and figure out what it really, really means by breaking down the error message and system. Type 2 does trial and error on random related things until the problem goes away. I hate to say that I've seen way more type 2s engineers than type 1, but maybe I’m working at the wrong companies.
The way you want to build and lead an engineering team is so that the individual engineer’s approach to the problem doesn’t matter. The idea is to establish deployment safety and ops mechanisms so Type 2 isn’t even possible and your Sr. Engineers coach the jr. engineers in how to use these mechanisms effectively through training and runbooks. The idea of COE is to figure out what mechanisms (availability tests, metri…
Earlier quoted context omitted.
I'd rather they admit a mistake and learn a lesson from it even if it isn't a good thing to advertise. That said, I agree that you are identifying a more important issue here but I also think you are being a bit too subtle about even if I agree with what you are saying. The real lesson that they should have learned from this ordeal is to never push code directly into production --- period. The article never mentions…
I don't see how testbed/sandbox would have helped, unless they'd also have a dedicated QA person _and_ configured their sandbox so have dramatically fewer instances. Because I can see "create a new subscription" in the manual test plan, but not "create 5x new subscription".
May be a wise thing to do anyway. I would even advocate for the extreme, one instance in sandbox. I've seen quite a variety of bugs that would be easier to detect and debug with one instance — from values accidentally persisted between requests (OP is one example of this very broad class of bugs), to a thundering herd bug catched in staging due to less instances and more worker threads per instance. But can't remember even one "distributed" bug caught in staging.
> they'd also have a dedicated QA person
This is a new feature, and it's user-facing, business-critical and inherently fragile (external service integration, though it broke for another reason). I hope multiple people manually run such things end to end before deploying to prod, even if none of them are dedicated QA?
Earlier quoted context omitted.
I think deploying and then going to sleep is the red flag here. They should have deployed the change at 9am or something and had the workday to monitor issues.
You can deploy and go to sleep if you have monitoring and alerting and someone getting paged. It shouldn't be a human monitoring for issues anyway, so the only reason to choose 9am over bedtime should be that you don't want to risk a late night page, not that someone will actually be checking up actively on the deployment.
And it's not trivial to set up a correct alert for this one. Simple HTTP 5xx threshold wouldn't work because it's high enough to not wake you up whenever cloud provider restarts Postgres, it's too high to catch this. You need either per-endpoint failure rate alerts or something else more clever.
Earlier quoted context omitted.
"defer tx.Commit() is nonsensical" It's not pure nonsense, it works in the happy path, and it matches the pattern of how people often handle file IO in go. f, err := os.OpenFile(...) defer f.Close() ... which is another place most gophers ignore errors incorrectly. Just like the "defer tx.Commit()" example, it's collocating the idea of setup and cleanup together. Those two patterns are so similar, python handles them…
> it's collocating the idea of setup and cleanup together. Okay, sure, but Rollback is the cleanup function. You always want to rollback – you only sometimes want to commit. I suppose this confirms that someone got their logic mixed up.
This phrase highlights the confusion. If you learned SQL before Go, you want to rollback only on error.
Every time I write `defer tx.Rollback()`, I cringe and have to remind myself that yes, it's actually ok to call a method called `Rollback` after succesfully writing data.
Earlier quoted context omitted.
Commit messages are overrated.
No they aren't. Commit messages show 1. Forward thinking 2. Maturity 3. Temperance All important traits in engineers and founders