921–930 of 953 posts
Earlier quoted context omitted.
> Crashing is not an outage. Are you in the right thread?
Did you skip all the other context about the other systems that failed? The problem was a query producing incorrect data. The crash helped them find it. What do you think happens when a program crashes?
Barely given the initial impression it was a DDoS.
Although you can argue that's bad observability.
Earlier quoted context omitted.
I would love to go further and explicitely forbid unwrap and similar calls using a `no_panic` attribute.
I actually have to do this for programs that runs in bare metal. You can't afford to have nondeterministic panic like this. If things really gone wrong you'd have a watchdog and health checker to verify the state of program.
Earlier quoted context omitted.
This wasn't a runtime property that could not be validated at compile time. And you don't need to fall back on "OS level security and reliability" when your type system is enforcing an application-level invariants. In fact I'd argue that crashing is bad. It means you failed to properly enumerate and express your invariants, hit an unanticipated state, and thus had to fail in a way that requires you to give up and fal…
> The end result was a massive world-wide outage. The world wide outage was actually caused by deploying several incorrect programs in an incorrect system. The root one was actually a bad query as outlined in the article. Let’s get philosophical for a second. Programs WILL be written incorrectly - you will deploy to production something that can’t possibly work. What should you do with a program that can’t work? Pret…
In this particular case, it was the "limit 200" because "performance reasons" so I think there was more space to implement the latter than the former.
Earlier quoted context omitted.
Oh come on, stop spreading FUD. Rust programs are 100% immune to crashes and bugs, they have memory safety (c). Also, exception handling is hard and lame. We don't need exceptions, just add a "match" block after every line in your program.
What's the point of this sarcastic comment? Do you think that some people claim that Rust's memory safety guarantees mean that a Rust program is incapable of crashing or having a bug? This is a dumb thing to claim certainly, but I'm not aware of anyone actually making this claim. I'm also not sure what you're getting at with the comment about exception handling being lame. I think the ML/Haskell inspired model that R…
Undoubtedly yes.
> ...but what does this have to do with match blocks?
You tell me. You're the one advocating for placing one after every single function call.
Earlier quoted context omitted.
> As yourself more the question, is your service that important to need 99.999% uptime? What is the cost of many-9s uptime from Cloudflare? For DDoS protection it is $0/month on their free tier: * https://www.cloudflare.com/en-ca/plans/
Not when you start pushing into the TB's range of monthly data... When you get that dreaded phone call from a CF rep, because the bill that is coming is no joke. Its free as long as you really are small, not worth milking. The moment you can afford to run your own mini dc at your office, you start to enter the "well, hello there" for CF.
As someone who has (and is) runs (running) a DC with all the electrical/UPS, cooling, piping, HVAC+D stuff to deal with: it can be a lot of just time/overhead.
Especially if you don't have a number of folks in-house to deal with all that 'non-IT' equipment (I'm a bit strange in that I have an interest in both IT and HVAC-y stuff).
Earlier quoted context omitted.
Crashes are silent failures but as I mentioned: you can get a lot of your crashes reported via the App Store. This is why I prefer crashes in this situation: it gives me something actionable over silent failures on the client.
But nothing beats catching the problem before the crash. Also, I have found App Store crash reports to be next to useless. TestFlight ones are a bit better. But if I spend a lot of time, doing it right, the first time, we can avoid all kinds of heartbreak.
What have you found useless about the crash reports from the App Store? It would be really nice for it to have something like a breadcrumb capability, but typically the stack trace of the crash is sufficient to see what went wrong.
Earlier quoted context omitted.
I actually have to do this for programs that runs in bare metal. You can't afford to have nondeterministic panic like this. If things really gone wrong you'd have a watchdog and health checker to verify the state of program.
How do you manage to do this?
Earlier quoted context omitted.
`slice[i]` is just sugar for `slice.get(i).unwrap()`. And whether it's a "local" invariant or not is orthogonal. And `unwrap()` does not "require lying about invariants across your API surface." > The blog post doesn’t address the issue, it simply pretends it’s not a real problem. It very explicitly addresses it! It even gives real examples. > Also from the post: “If we were to steelman advocates in favor of this sty…
> `slice[i]` is just sugar for `slice.get(i).unwrap()`. And whether it's a "local" invariant or not is orthogonal. And `unwrap()` does not "require lying about invariants across your API surface." It's not orthogonal. `Result` isn't a local invariant, and yes, `.unwrap()` does require lying. If your code depends on an API that can fail, and you cannot handle that failure locally (`.unwrap()` is not handling it), then…
I don't think you understand what an internal runtime invariant is. Either way, I don't know of any widespread libraries (in any language) that follow this "principled" position. That makes it de facto extreme.
> I've built and worked on ridiculously complex code bases without a single instance of `.unwrap()` or the local language equivalent; it's just not necessary.
Show me. If you're using `slice[i]`, then you're using `unwrap()`. It introduces a panicking branch.
> If your code depends on an API that can fail, and you cannot handle that failure locally (`.unwrap()` is not handling it), then your type signature needs to express that you can fail -- and you need to raise an error on that failure.
You use `unwrap()` when you know the failure cannot happen.
I note you haven't engaged with any of the examples I provided in the blog.
> This showed up to Internet users trying to access our customers' sites as an error page indicating a failure within Cloudflare's network. As a visitor to random web pages, I definitely appreciated this—much better than their completely false “checking the security of your connection” message. > The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind. Instead, it was tri…
Because we initially thought it was an attack. And then when we figured it out we didn’t have a way to insert a good file into the queue. And then we needed to reboot processes on (a lot) of machines worldwide to get them to flush their bad files.
Earlier quoted context omitted.
> The issue here is about the system as a whole not any line of code. Unsoundness in the type system that leads to a systemic failure is about the system as a whole. Not everything can be recovered from restarting a process, and process correctness and recovery is something that also derives from your type system.
In the early 2000s when Google explained how they achieved their (already back then) awesome reliability, ie assuming that any software and hardware will eventually fail, and that they designed everything with the idea that everything was faulty, there were some people who couldn't get it, who would still bring the argument that "yeah but today with modern raid..." People here chatting about unwrap remind me of them…
If you depend on engineers not fucking up, you will fail. Using unwrap is assuming humans won’t get human-enforced invariants wrong. They will. They did here.
As someone that works in formal verification of crypto systems, watching people like yourself advocate for hope-and-prayer development methodology is astonishing.
However, I understand why we’re still having this debate. It’s the same debate that’s been occurring for the same reasons for decades.
Doing things correctly is mentally more difficult, and so people jump through ridiculous rhetorical hoops to justify why they will not — or quite often, mentally cannot — perform that intellectual labor.
It’s a disheartening lack of craftsmanship and industry accountability, but it’s nothing new.