41–50 of 83 posts
> To Renaud, Claire, and Eugen of the Mastodon developer team, who went above and beyond all expectations to help us out. You folks were amazing, you took our situation very seriously, and immediately jumped in to help us. I really could not have asked for anything more. Thank you! I don't know if Vivaldi provides financial support to Mastodon (I couldn't find their name on the sponsors page). If not, I hope this sit…
Earlier quoted context omitted.
I log stacks for every error-level log and have never found it that useful. It's better than just logging "EOF" with no context of course, but manually annotating each frame with information not known to the caller is the way to go. Shifting to Go specifics; stack traces miss things like channel recvs and loops. Consider: for _, datum := range data { if err := DoSomethingWithDatum(datum); err != nil { log.Error(...)…
> In that case, the stack trace misses the most important thing: which datum failed. OK we agree that the stacktrace isn't _enough_, but it's still a really useful thing to have to understand what exactly happened (and quite often the single most useful thing). Of course we still expect devs to capture the information that led to the `log.Error`, so that we don't have to play guess games. Rather than manually-annotat…
I prefer to build metrics and distributed traces from logs! But, I think we can agree that they're the same thing. It's a stream of events that a system captures and lets you retrieve. (I wrote our logging/tracing/metrics library at work, and indeed I call any sort of start/finish operation a Span. https://github.com/pachyderm/pachyderm/blob/master/src/inter...)
Post-processing into an efficient retrieval system is the key. You can tail the logs and send span starts/ends to Jaeger, and you can tail the logs and send summarized metrics to Prometheus or InfluxDB. I really like having the too-high-cardinality raw data around, though, so I can analyze specific failing requests. For example, you wouldn't want to have per-request "tx_bytes" metrics in Prometheus; the cardinality is too high and it blows up the whole system. But if you log your metrics, then you find the request (grep for lines that contain the x-request-id), and you can see exact timestamps for when every megabyte of data was sent along with everything else that happened in between. ("It took 10 seconds to send 1MB of this file?" "We did 300 point SQL queries between the 10th and 11th megabyte?" Things like that.) Meanwhile, you still have summarized data for alerts ("across all requests handled by this machine, 90% of requests are taking longer than 10 seconds").
Logs end up being a lot of data, but storing a terabyte of logs to save me a week of debugging is the biggest no-brainer in software engineering. $20 for 1 week of time saved. I'll also add that as a backend developer, logs are your software's UI. If something goes wrong, that's where the operator interfaces with your software to fix the problem. So they are not to be neglected or done without care; the same way you'd probably spell check your HTML frontend.
I'll also add, I have a weird background. When I worked on Google Fiber, we wanted the ability to add new fleet-wide CPE metrics to the system without a software push. So, we logged aggressively and turned relevant log lines into metrics on the log receiver side. (I designed and wrote that system.) That meant we could monitor for issues without taking people's Internet down for 15 minutes while their routers rebooted to apply a software update that captured a metric we wanted to monitor or alert on. At my current job, we don't operate the software we write; our users buy a license and then they do God Knows What on their own infrastructure. What that means is if I want something like Jaeger, it's on me to install it, maintain it, upgrade it, and support it in an environment that I can only access by saying commands to type on a Zoom call. The juice was worth the squeeze exactly once; for logs. Users can run "pachctl debug dump", generate a tar file, send it to us, and then we have logs, metrics, and traces without any operational effort on their end.
While I'm here, here's how I do metrics-to-logs on the producing side: https://github.com/pachyderm/pachyderm/blob/master/src/inter... Apache2 if anyone wants to steal it ;)
Earlier quoted context omitted.
I'm sure all 30 of Nostr users will benefit.
Yeah, because Mastodon is the most happening place online lol. I can't believe people on hacker news talk like this. Embarrassing how far this community has declined. Can't even discuss protocols without these stupid comments.
Compared to other Twitter alternatives? It absolutely is. It's not even a contest, it's in a league of its own. As embarassing as it sounds, Bluesky, Nostr, Post.news, Spoutible etc don't come even close. (Threads does of course, but the two should be compatible in the near future.)
It's also the only one that 1) didn't come to life as a reaction to Twitter changing ownership, 2) federates between a decent amount of interoperable servers and software right now, not in the future, 3) already has years of experience of dealing with bad actors that are gonna come to any decentralised service, 4) grew organically, not with VC money, and 5) is stable enough that you can choose between dozens of third-party clients (the thing we all complained about Twitter and Reddit killing this year).
A couple of millions of MAU is tiny in comparison to centralised social media (like Instagram), but it's huge in comparison to any other decentralised protocol made this century. There's nothing out there that's gonna dethrone it in usage for the next 3-5 years.
I'll never forget the first time I had to restore a massive sql dump and realized that vim actually segfaults trying to read it. That's when I discovered the magic of spit(1) "split a file into pieces". I just split the huge dump into one file per table. Of course a table can also be massive, but at least the file is now more uniform which means you can easier run other tools on it like sed or awk to transform querie…
The workaround involved writing a python script that handled everything in a gradual manner, moving files into subdirectories based on shared prefixes.
One of the better post-mortems I've read in a long while.
Great writeup (including the human cost, e.g. loss / lack of sleep, which in my experience has a huge impact on complicated incident resolution). Here’s what jumped out at me: “The new account was created in our database with a null value in the URI field.” Almost every time I see a database-related postmortem — and I have seen a lot of them — NULL is lurking somewhere in the vicinity of the crime scene. Even if NULL…
IMO the problem (at least in this case) is not NULL in the DB, but NULL at the application level.
If NULL is some sort of Maybe monad and you're forced to deal with it, well, you're forced to deal with it, think about it, etc.
Empty string, whatever NULL string is in your language of choice, or some sort of sigil value you invent... not much of a difference.
> To Renaud, Claire, and Eugen of the Mastodon developer team, who went above and beyond all expectations to help us out. You folks were amazing, you took our situation very seriously, and immediately jumped in to help us. I really could not have asked for anything more. Thank you! I don't know if Vivaldi provides financial support to Mastodon (I couldn't find their name on the sponsors page). If not, I hope this sit…
Great writeup (including the human cost, e.g. loss / lack of sleep, which in my experience has a huge impact on complicated incident resolution). Here’s what jumped out at me: “The new account was created in our database with a null value in the URI field.” Almost every time I see a database-related postmortem — and I have seen a lot of them — NULL is lurking somewhere in the vicinity of the crime scene. Even if NULL…
What's the alternative, an empty string? IMO the problem (at least in this case) is not NULL in the DB, but NULL at the application level. If NULL is some sort of Maybe monad and you're forced to deal with it, well, you're forced to deal with it, think about it, etc. Empty string, whatever NULL string is in your language of choice, or some sort of sigil value you invent... not much of a difference.
Great writeup (including the human cost, e.g. loss / lack of sleep, which in my experience has a huge impact on complicated incident resolution). Here’s what jumped out at me: “The new account was created in our database with a null value in the URI field.” Almost every time I see a database-related postmortem — and I have seen a lot of them — NULL is lurking somewhere in the vicinity of the crime scene. Even if NULL…
What's the alternative, an empty string? IMO the problem (at least in this case) is not NULL in the DB, but NULL at the application level. If NULL is some sort of Maybe monad and you're forced to deal with it, well, you're forced to deal with it, think about it, etc. Empty string, whatever NULL string is in your language of choice, or some sort of sigil value you invent... not much of a difference.
In this case, a `user_uris` table with non-nullable columns and a unique constraint on `user_id` is the first option that comes to mind.
> To Renaud, Claire, and Eugen of the Mastodon developer team, who went above and beyond all expectations to help us out. You folks were amazing, you took our situation very seriously, and immediately jumped in to help us. I really could not have asked for anything more. Thank you! I don't know if Vivaldi provides financial support to Mastodon (I couldn't find their name on the sponsors page). If not, I hope this sit…
But we indeed have sponsorships open, and they really have impact. Having full-time people working on the project is very impactful, but at the moment we only have 1 full-time developer in addition to Eugen (the founder) and a DevOps person on the technical side.