61–70 of 953 posts
Earlier quoted context omitted.
Yes, can't have .unwrap() in production code (it's ok in tests)
Like goto, unwrap is just a tool that has its use cases. No need to make a boogeyman out of it.
Earlier quoted context omitted.
It would have been caught only in stage if there was similar amount of data in the database. If stage has 2x less data it would have never occurred there. Not super clear how easy it would have been to keep stage database exactly as production database in terms of quantity and similarity of data etc. I think it's quite rare for any company to have exact similar scale and size of storage in stage as in prod.
> I think it's quite rare for any company to have exact similar scale and size of storage in stage as in prod. We’re like a millionth the size of cloudflare and we have automated tests for all (sort of) queries to see what would happen with 20x more data. Mostly to catch performance regressions, but it would work to catch these issues too. I guess that doesn’t say anything about how rare it is, because this is also t…
In this case it seems the database table in question seemed modest in size (the features for ML) so naively thinking they could have kept stage features always in sync with prod at the very least, but could be they didn't consider that 55 rows vs 60 rows or similar could be a breaking point given a certain specific bug.
It is much easier to test with 20x data if you don't have the amount of data cloudflare probably handles.
> thread fl2_worker_thread panicked: called Result::unwrap() on an Err value I don't use Rust, but a lot of Rust people say if it compiles it runs. Well Rust won't save you from the usual programming mistake. Not blaming anyone at cloudflare here. I love Cloudflare and the awesome tools they put out. end of day - let's pick languages | tech because of what we love to do. if you love Rust - pick it all day. I actually…
could have been tight deadline, managerial pressure or just the occasional slip up.
Why does cloudflare allow unwraps in their code? I would've assumed they'd have clippy lints stopping that sort of thing. Why not just match with { ok(value) => {}, Err(error) => {} } the function already has a Result type. At the bare minimum they could've used an expect("this should never happen, if it does database schema is incorrect"). The whole point of errors as values is preventing this kind of thing.... It w…
Why call .unwrap() in a function which returns Result ? For something so critical, why aren't you using lints to identify and ideally deny panic inducing code. This is one of the biggest strengths of using Rust in the first place for this problem domain.
Probably because this case was something more akin to an assert than an error check.
This reads to me more like the error type returned by append with names is not (ErrorFlags, i32) and wasn't trivially convertible into that type so someone left an unwrap in place on an "I'll fix it later" basis, but who knows.
Classic combination of errors: Having the feature table pivoted (with 200 feature1, feature2, etc columns) meant they had to do meta queries to system.columns to get all the feature columns which made the query sensitive to permissioning changes (especially duplicate databases). A Crowdstrike style config update that affects all nodes but obviously isn't tested in any QA or staged rollout strategy beforehand (the app…
The solution to that problem wasn't better testing of database permutations or a better staging environment (though in time we did do those things). It was (1) a watchdog system in our proxies to catch arbitrary deadlocks (which caught other stuff later), (2) segmenting our global broadcast domain for changes into regional broadcast domains so prod rollouts are implicitly staged, and (3) a process for operators to quickly restore that system to a known good state in the early stages of an outage.
(Cloudflare's responses will be different than ours, really I'm just sticking up for the idea that the changes you need don't follow obviously from the immediate facts of an outage.)
Why call .unwrap() in a function which returns Result ? For something so critical, why aren't you using lints to identify and ideally deny panic inducing code. This is one of the biggest strengths of using Rust in the first place for this problem domain.
Yes, can't have .unwrap() in production code (it's ok in tests)
> thread fl2_worker_thread panicked: called Result::unwrap() on an Err value I don't use Rust, but a lot of Rust people say if it compiles it runs. Well Rust won't save you from the usual programming mistake. Not blaming anyone at cloudflare here. I love Cloudflare and the awesome tools they put out. end of day - let's pick languages | tech because of what we love to do. if you love Rust - pick it all day. I actually…
> This showed up to Internet users trying to access our customers' sites as an error page indicating a failure within Cloudflare's network. As a visitor to random web pages, I definitely appreciated this—much better than their completely false “checking the security of your connection” message. > The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind. Instead, it was tri…
Because we initially thought it was an attack. And then when we figured it out we didn’t have a way to insert a good file into the queue. And then we needed to reboot processes on (a lot) of machines worldwide to get them to flush their bad files.
- A product depends on frequent configuration updates to defend against attackers.
- A bad data file is pushed into production.
- The system is unable to easily/automatically recover from bad data files.
(The CrowdStrike outages were quite a bit worse though, since it took down the entire computer and remediation required manual intervention on thousands of desktops, whereas parts of Cloudflare were still usable throughout the outage and the issue was 100% resolved in a few hours)