Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What happened to Vivaldi Social?

thomasp.vivaldi.net

Post reply on HN
Print view View original

21–30 of 83 posts

Re: What happened to Vivaldi Social?

#21

Post by NikkiA »

SV_BubbleTime wrote: post #17
empathy_m wrote: post #16

The part that resonates here is saying "ah yes well we have a full database backup so we can do a full restore", then "the full restore will be tough and involve downtime and has some side effects," then "I bet we could be clever and restore only part of the data that are missing", then doing that by hand, which hits weird errors, then finally shipping the jury-rigged selective restore and cleaning up the last five m…

I agree with you. The phrase is you don’t have backups unless you test your backups. But in this case I don’t really get what the issue is. Restore everything from the last good backup and people miss some posts made in the meantime, sucks, but it’s an instant solution instead of hand work and uncertainty.

When I worked as a VMS sysadmin full restore checks were one of the things I insisted on doing, sure, it used up a morning every couple of weeks, and tied up one of our microvaxes, but it was worth it.

Especially three months after I finished being sysadmin and moved to development, and they had a disk failure.

me: 'so you have backups?'

the replacement: 'sure, but they didn't restore'

me: 'what's the last good backup you have?'

tr: 'august, the last one you did'

me: 'welp'

tr's boss: 'guess £390,000 for third party disk recovery is our only option...'

Re: What happened to Vivaldi Social?

#22

Post by pbjtime »

SV_BubbleTime wrote: post #17
empathy_m wrote: post #16

The part that resonates here is saying "ah yes well we have a full database backup so we can do a full restore", then "the full restore will be tough and involve downtime and has some side effects," then "I bet we could be clever and restore only part of the data that are missing", then doing that by hand, which hits weird errors, then finally shipping the jury-rigged selective restore and cleaning up the last five m…

I agree with you. The phrase is you don’t have backups unless you test your backups. But in this case I don’t really get what the issue is. Restore everything from the last good backup and people miss some posts made in the meantime, sucks, but it’s an instant solution instead of hand work and uncertainty.

"If you've never tested your backup solution, you don't have a backup solution."

Re: What happened to Vivaldi Social?

#23

Post by TheDong »

SV_BubbleTime wrote: post #18
TheDong wrote: post #7

Earlier quoted context omitted.

> HTTP 500 errors when viewing an account > Stacktrace for that 500 This is the default ruby on rails behavior. It prints a stacktrace on any 500 or unknown error, and it's just line numbers and filepaths. > megabytes on every single hit I run a rails app that's very poorly designed. I just checked, and the stack trace for a single 500 is 5KiB. It doesn't even add up to 1MiB a day since there's only a 500 error about…

>Java's default exception behavior is to bubble up a stack trace with every exception, whether you print it or not, Anyone who has spent 5 minutes in Java knows exactly what this looks like. And also how unwelcoming it is to new programmers.

I have spent more than 5 minutes in java, and I don't know why having a stack available on an exception is unwelcoming to new programmers.

It definitely seems better than the approaches some other languages have, like C's "return code 1, check errno", when most new programmers don't even know what an errno is.

Can you explain it to me?

Re: What happened to Vivaldi Social?

#24

Post by INTPenis »

I'll never forget the first time I had to restore a massive sql dump and realized that vim actually segfaults trying to read it.

That's when I discovered the magic of spit(1) "split a file into pieces". I just split the huge dump into one file per table.

Of course a table can also be massive, but at least the file is now more uniform which means you can easier run other tools on it like sed or awk to transform queries.

Re: What happened to Vivaldi Social?

#25

Post by chx »

> And it just so happens that all local accounts in a Mastodon instance have a null value in their URI field, so they all matched.

How? NULL = NULL evaluates to FALSE, SQL is a three value logic, specifically Kleene's weak three-valued logic, NULL anyoperator NULL is NULL.

Re: What happened to Vivaldi Social?

#26

Post by olah_1 »

input_sh wrote: post #12
olah_1 wrote: post #9

Earlier quoted context omitted.

Perhaps better is decentralized twitter (Nostr). Your account doesn't live on a server and you send events to multiple servers if you want to. If one server goes down, it hardly impacts you.

I'm sure all 30 of Nostr users will benefit.

Yeah, because Mastodon is the most happening place online lol. I can't believe people on hacker news talk like this. Embarrassing how far this community has declined. Can't even discuss protocols without these stupid comments.

Re: What happened to Vivaldi Social?

#27

Post by mollems »

Great writeup (including the human cost, e.g. loss / lack of sleep, which in my experience has a huge impact on complicated incident resolution).

Here’s what jumped out at me: “The new account was created in our database with a null value in the URI field.”

Almost every time I see a database-related postmortem — and I have seen a lot of them — NULL is lurking somewhere in the vicinity of the crime scene. Even if NULL sometimes turns out not to be the killer, it should always be brought in for questioning.

My advice is: never rely on NULL as a sentinel value, and if possible, don’t allow it into the database at all. Whatever benefits you think you might gain, they will inevitably be offset by a hard-to-find bug, quite possibly years later, where some innocuous-seeming statement expects either NULL or NOT NULL and the results are unexpected (often due to drift in the semantics of the data model).

Although this was a race condition, if the local accounts and the remote accounts were affirmatively distinguished by type, the order of operations may not have mattered (and the account merge code could have been narrowly scoped).

Re: What happened to Vivaldi Social?

#28

Post by vore »

SV_BubbleTime wrote: post #18
TheDong wrote: post #7

Earlier quoted context omitted.

> HTTP 500 errors when viewing an account > Stacktrace for that 500 This is the default ruby on rails behavior. It prints a stacktrace on any 500 or unknown error, and it's just line numbers and filepaths. > megabytes on every single hit I run a rails app that's very poorly designed. I just checked, and the stack trace for a single 500 is 5KiB. It doesn't even add up to 1MiB a day since there's only a 500 error about…

>Java's default exception behavior is to bubble up a stack trace with every exception, whether you print it or not, Anyone who has spent 5 minutes in Java knows exactly what this looks like. And also how unwelcoming it is to new programmers.

Sure, but imagine how unwelcoming to everyone it would be if you didn't get a stack trace.

Re: What happened to Vivaldi Social?

#29

Post by williamdclt »

TheDong wrote: post #10
rsynnott wrote: post #3

Items two and three not happening atomically feels like an issue, though I assume there's a reason that it's not trivial to do so (I haven't looked at the code; really should at some point.)

One of the linked fixes is: https://github.com/mastodon/mastodon/commit/13ec425b721c9594... It seems like it was trivial to make it happen atomically. There just wasn't a need to before since them not being atomic isn't an issue, unless you have a poor configuration like someone pointing sidekiq at a stale database server (sorry, a replica), which I see as the primary issue here.

Maybe I’m missing something but I f it’s not atomic, it doesn’t matter whether there’s a replica or not: sidekiq (whatever that is) might do a read in-between step 2 and 3.

I see several problems in their setup really

- lack of strong consistency

- using eventually consistent data, the replica, to take business decision

- no concurrency control (pessimistic or optimistic)

I don’t know much about mastodon but, while not trivial, that’s pretty basic systems design concepts

Re: What happened to Vivaldi Social?

#30

Post by williamdclt »

TylerE wrote: post #5

This make anyone elses eyebrows raise sky high at this? > Claire replied, asking for the full stacktraces for the log entries, which I was able to also extract from the logs. This is either deep voodoo magic, or the code or configuration is turning a Xeon into the equivalent of a 286. House is that not, like, megabytes on every single hit?

Do you mean you do _not_ capture stacktraces of errors in a live system ? How do you go about understanding where the error comes from ?
Post reply on HN