This is a tough and crappy situation but that was an engaging read surprisingly! Nice one
What happened to Vivaldi Social?
31–40 of 83 posts
Re: What happened to Vivaldi Social?
#32> To Renaud, Claire, and Eugen of the Mastodon developer team, who went above and beyond all expectations to help us out. You folks were amazing, you took our situation very seriously, and immediately jumped in to help us. I really could not have asked for anything more. Thank you!
I don't know if Vivaldi provides financial support to Mastodon (I couldn't find their name on the sponsors page). If not, I hope this situation causes them (and other companies using Mastodon) to consider sponsorship or a support contract.
Re: What happened to Vivaldi Social?
#33This make anyone elses eyebrows raise sky high at this? > Claire replied, asking for the full stacktraces for the log entries, which I was able to also extract from the logs. This is either deep voodoo magic, or the code or configuration is turning a Xeon into the equivalent of a 286. House is that not, like, megabytes on every single hit?
Do you mean you do _not_ capture stacktraces of errors in a live system ? How do you go about understanding where the error comes from ?
I log stacks for every error-level log and have never found it that useful. It's better than just logging "EOF" with no context of course, but manually annotating each frame with information not known to the caller is the way to go. Shifting to Go specifics; stack traces miss things like channel recvs and loops. Consider:
for _, datum := range data {
if err := DoSomethingWithDatum(datum); err != nil {
log.Error(...)
}
}
Another common case:
type Thing struct {
Value any
Err error
}
func Produce() {
ch TL;DR stack traces are better than nothing, but a comprehensive way of handling errors and writing the information you need to fix it to the log is going to be more valuable. It is a lot of work, but I've always found it worthwhile.
Re: What happened to Vivaldi Social?
#34Earlier quoted context omitted.
I'm sure all 30 of Nostr users will benefit.
Yeah, because Mastodon is the most happening place online lol. I can't believe people on hacker news talk like this. Embarrassing how far this community has declined. Can't even discuss protocols without these stupid comments.
In the grand scheme of things, even twitter is anecdotic as only a small and marginal fraction of internet users are using twitter.
It makes the remark above even more idiotic.
Re: What happened to Vivaldi Social?
#35Earlier quoted context omitted.
I'm sure all 30 of Nostr users will benefit.
Yeah, because Mastodon is the most happening place online lol. I can't believe people on hacker news talk like this. Embarrassing how far this community has declined. Can't even discuss protocols without these stupid comments.
A protocol is only as useful as the number of people that adopt it. Network effects and preferential attachment are real phenomena. I've been using Nostr almost since the beginning, so it's not that I am biased against it.
Re: What happened to Vivaldi Social?
#36Earlier quoted context omitted.
I'm sure all 30 of Nostr users will benefit.
Yeah, because Mastodon is the most happening place online lol. I can't believe people on hacker news talk like this. Embarrassing how far this community has declined. Can't even discuss protocols without these stupid comments.
What the hell does any of this have to do with protocols or the problem at hand? You're the one who showed mentioning nostr out of nowhere.
Re: What happened to Vivaldi Social?
#37Items two and three not happening atomically feels like an issue, though I assume there's a reason that it's not trivial to do so (I haven't looked at the code; really should at some point.)
One of the linked fixes is: https://github.com/mastodon/mastodon/commit/13ec425b721c9594... It seems like it was trivial to make it happen atomically. There just wasn't a need to before since them not being atomic isn't an issue, unless you have a poor configuration like someone pointing sidekiq at a stale database server (sorry, a replica), which I see as the primary issue here.
> There just wasn't a need to before since them not being atomic isn't an issue
I disagree: there clearly is an issue with a non-local account having a null URI. It’s unlikely but totally possible for the server to crash inbetween query 1 and query 2, irrespective of database replication stuff. This is a textbook example of why you use database transactions.
Re: What happened to Vivaldi Social?
#38Earlier quoted context omitted.
Do you mean you do _not_ capture stacktraces of errors in a live system ? How do you go about understanding where the error comes from ?
I log stacks for every error-level log and have never found it that useful. It's better than just logging "EOF" with no context of course, but manually annotating each frame with information not known to the caller is the way to go. Shifting to Go specifics; stack traces miss things like channel recvs and loops. Consider: for _, datum := range data { if err := DoSomethingWithDatum(datum); err != nil { log.Error(...)…
> In that case, the stack trace misses the most important thing: which datum failed.
OK we agree that the stacktrace isn't _enough_, but it's still a really useful thing to have to understand what exactly happened (and quite often the single most useful thing). Of course we still expect devs to capture the information that led to the `log.Error`, so that we don't have to play guess games.
Rather than manually-annotated logs, I'd prefer getting rid of all logging altogether and use tracing (opentelemetry), which is precisely designed for observability.
Re: What happened to Vivaldi Social?
#39> And it just so happens that all local accounts in a Mastodon instance have a null value in their URI field, so they all matched. How? NULL = NULL evaluates to FALSE, SQL is a three value logic, specifically Kleene's weak three-valued logic, NULL anyoperator NULL is NULL.
Yeah, was wondering. Maybe they filter at the application level? And check equality with their language's null value?
Re: What happened to Vivaldi Social?
#40Earlier quoted context omitted.
I agree with you. The phrase is you don’t have backups unless you test your backups. But in this case I don’t really get what the issue is. Restore everything from the last good backup and people miss some posts made in the meantime, sucks, but it’s an instant solution instead of hand work and uncertainty.
When I worked as a VMS sysadmin full restore checks were one of the things I insisted on doing, sure, it used up a morning every couple of weeks, and tied up one of our microvaxes, but it was worth it. Especially three months after I finished being sysadmin and moved to development, and they had a disk failure. me: 'so you have backups?' the replacement: 'sure, but they didn't restore' me: 'what's the last good backu…
To add some context...
Yes, it was documented in our ISO 9000 docs. But only 'strongly recommended' to perform a regular/routine test restore. I attempted to get it converted to a mandatory step, but since I was only a temporary sysadmin and an intern, it wasn't going to happen.
I was told by my predecessor (who was a direct contractor to my employer) to perform it as routinely as I could. I would guess that he had attempted to get it put as a mandatory step, but his time was billed, mine wasn't, so shrug.
My/the replacement was an external contractor as part of a 'company Y now provides system administration services' deal, who presumably ended up eating the liability of not having working backups that they were contracted to produce.
As horrified as I was, 'it's not really my problem, I wasn't responsible' was the only attitude I could bear to take. Besides, I was busy with fortran.