351–360 of 834 posts
Always use checked pre- and post-conditions in a data pipeline. This simple post-condition would have caught this issue: The sheet after merge operation must have a number of rows equal to the sum of number of rows for all merged sheets. Assuming this is a merge of a standardized input, then another post-condition might be: The number of columns in output shall equal the number of columns in the input. Might want to…
I'm not sure they teach that in medical school.
Earlier quoted context omitted.
I think you're overstating both the paper's impact, and the significance of the error. Most such papers are used to support existing policy preferences, not drive them. In addition, the error didn't reverse or erase the correlation, it diminished it, and removed the inflection point from the curve.
It was literally cited by Britain's finance minister George Osborne as justification for UK government austerity: > As Ken Rogoff himself puts it, "there's no question that the most significant vulnerability as we emerge from recession is the soaring government debt. It's very likely that will trigger the next crisis as governments have been stretched so wide." https://web.archive.org/web/20100414205630/http://www.co…
Earlier quoted context omitted.
That would imply non-tech people having some surprisingly deep visibility into technical details. What they really see is Excel vs. some contractor doing "a system". The reality is that Excel is available today and works, and scales up... well, until it doesn't. Still, you have data entry that everybody in the field understands, that's rock solid (so no need to unit test anything), and has a tried and true authentica…
Not to mention the "system" solution will have to go through its own debugging and will undoubtedly have bugs in it for some time after initial release. On top of that, there is STILL room for human error as a user is still the one actually inputting data and designing the experiment/data flow. I'm going to get mauled on this forum given the audience, but parent comment reeks of the technical elitism on this forum an…
That's simply not possible unless you have a whole system around excel running a year harness.
You can maybe do it if you are super dedicated with locked cells and formal double-blind QA passes which maybe exist somewhere but not in the vast majority of operations.
Each test result creating several rows of data seems like a problem too. In clean data, every observation is one row. It makes working with the dataset much easier. In this scenario, I would expect one observation to correspond to one test result. The multiple rows are then better off pivoted into columns.
Earlier quoted context omitted.
But they aren't noticed. The classic blunder is dropping the last item off of a summation, and nobody notices until they add another item and things look janky. Those happen all the time, we just only hear about it when the consequences are outsize.
You are writing about a problem that can be solved if you know what you are doing: Store your data in a table ( https://www.contextures.com/xlExcelTable01.html ), then make a pivot table out of it. You will never have problems with missing data. In fact good practice is to add a check, just to see if your pivot table was refreshed and the data there matches the data in the source table. Just like you make tests in so…
"But you wouldn't use XLS. Nobody would start with that." Oh. "Every" sales and business person probably would. Probably.
Earlier quoted context omitted.
I mean, yeah, but pretend Excel is, idk, sword arms. You get a lot of work done by whirling around the office, spinning your swords, cutting off a variety of limbs and heads in the process. You get to the end, meet the few survivors, staring dismayed at the trail of blood and dismemberment behind you, and you yell, "BUT I GOT THE JOB DONE!" You're not wrong, but... I wish there were some alternative where we instead…
But the negative externalities don't seem all that big, considering. For every problem like this, you get countless millions of person-hours and dollars saved in work that wouldn't have happened at all, or would have been delayed, because of the necessity and cost of creating complex IT systems to support it. If Excel is a sword tornado, it's one happening in an environment where everyone knows to be super vigilant a…
I've found errors in every Excel spreadsheet I've ever looked at, and I'm not some master-excel user; I usually find them because - if I care about the results, I rewrite them as a Python script, so I get to go through everything.
The fact that Excel is effectively not auditable is a huge problem. There are a lot of dollars saved, like you said, but also many wasted or embezzled without anyone noticing in time because past a very low bar of complexity, it's really impossible to figure out what's happening without much, much work.
> If Excel is a sword tornado, it's one happening in an environment where everyone knows to be super vigilant about sharp objects.
... rather, no one is really careful, everyone gets a small cut every now and then, to which the apply a bandage and continue like nothing happened. And occasionally, they lose an eye or a limb or a head - and it is only those cases you read about in the newspapers.
I do not have a better suggestion, I'm afraid, but Excel is causing damage everywhere - e.g. [0], the subtitle - which I'm afraid is not an exaggeration, is "Sometimes it’s easier to rewrite genetics than update Excel"
[0] https://www.theverge.com/2020/8/6/21355674/human-genes-renam...
It’s fun to point and laugh, but is this really just the consequence of an unprecedented health emergency demanding a rush job? As far as bugs go, it doesn’t sound that bad. They didn’t lose data - they just processed it late? And they spotted it within days/weeks, and have a workaround/correction already? And it’s only the reporting that was wrong, not the more important part where they inform people of results? I’d…
Earlier quoted context omitted.
Couldn't you achieve exactly what was needed here with Airtable? Or, easier yet, Google Sheets and a Google Form?
Probably not while complying with privacy laws. I would be much more upset to find out a governemnt employ on a deadline used Google Sheets or Airtable to share my medical data instead of an Excel doc on a secure government file server.
" For customers who are subject to the requirements of the Health Insurance Portability and Accountability Act (HIPAA), G Suite and Cloud Identity can also support HIPAA compliance"
https://support.google.com/a/answer/3407054?hl=en
> secure
"UHS says all U.S. facilities affected by apparent ransomware attack Computer systems at Pennsylvania-based Universal Health Services began to fail over the weekend, leading to a network shutdown at hospitals around the country."
https://www.healthcareitnews.com/news/uhs-says-all-us-facili...