> But the takeaway is clear enough: Just because something is expressed in numbers doesn’t make it right, or even rational. And rationality itself might not be quite what it’s cracked up to be.
I'll grant something the first half of the first statement: sure, the values encoded into algorithms are not necessarily the values we want them to have ('right'). We want and need to do better at modeling our values.
Even the 'or even rational' part has some merit, some of the time. Like when our models don't reflect reality accurately enough to make helpful predictions about what to expect. Sure, the models are still rational -- but flawed.
But the last sentence is self-comforting intellectual garbage. It's a way to say "don't bother with the hard work of knowing things". It's a gateway for denial. It's saying "if the facts don't line up with what you want, that's ok, sometimes logic doesn't work".
Rationality is precisely what it is cracked up to be: the work of aligning your expectations with reality. It is only as useful as its practitioners are good at it. If you want to summarily write it off as 'considered harmful' then that's your choice. You can deal with the consequences of running headlong into your misunderstandings when they happen, but those of us with work to do will continue to try and get to the bottom of things, even if the problems are complicated.