Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Missing Covid-19 test data was caused by the ill-thought-out use of Excel

bbc.com

Post reply on HN
Print view View original

461–470 of 834 posts

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#461

Post by cameronh90 »

icedchai wrote: post #451
konjin wrote: post #407

Earlier quoted context omitted.

Having worked in the financial industry, my first job was writing a system to cover for a $10m fuck up in excel where yahoo exchange rates stopped updating and the same spreadsheet was used for a few weeks until someone noticed "Shit, we're losing a lot of money here.". Excel is the right solution in the same way playdoh is a valid building material.

Seems seems there were at least a couple of problems here. Why were financial professionals using Yahoo as a source of exchange rates?

You'd be surprised at how common that was. I've known two other financial orgs that scraped Yahoo as a critical part of their investment system...

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#462

Post by OJFord »

CamperBob2 wrote: post #439
OJFord wrote: post #110

Earlier quoted context omitted.

Where did you go to school that you didn't do statistics?

Just about anywhere in the US? Remember that HN comments are an exercise in selection bias.

Then that is nuts, and you don't need to remove calculus to fix it.

Presumably there's something you're teaching that we (in the UK) aren't, or more depth somewhere, but I don't know what it is. Our systems are quite different in that mathematics becomes optional after GCSEs (15-16yo) here, but statistics is taught from a far younger age than that, to that, and beyond for those that take A level(s) in mathematics. (As I recall there are six statistics A level modules total, S1-6, I think S1-2 are compulsory for a full A2 (vs. AS) mathematics qualification (which consists of six modules total). In order to do all six statistics modules one would at least take the second A level 'further mathematics', and probably (pun intended; unless statistics was a particular passion and the school allowed it) 'further additional'.

NB I quite liked that structure - there are 18 'modules' total (arranged in 'core', 'further pure', 'decision' (algorithms), 'statistics', and 'mechanics'. Three A levels total available (six modules each) or fewer and an AS (three). Which ones you want to do are almost entirely up to you if the school's big/lenient enough. IIRC you could even decide for yourself how to allocate the modules' grades across the number of A levels you were eligible for, e.g. if AC would be more beneficial to you than BB.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#463

Post by bigiain »

TeMPOraL wrote: post #286
lordnacho wrote: post #247

The problem is that anyone who isn't a decent coder think excel is a sensible choice for this, because they've seen it in their school or work. So when the proposals come in, you're going to see one guy who says it's all common sense and we use familiar old excel for everything, and another lunatic who says something called "pigsqueal" is actually the standard, connected to a "frontend" which for some reason is now s…

That would imply non-tech people having some surprisingly deep visibility into technical details. What they really see is Excel vs. some contractor doing "a system". The reality is that Excel is available today and works, and scales up... well, until it doesn't. Still, you have data entry that everybody in the field understands, that's rock solid (so no need to unit test anything), and has a tried and true authentica…

> Still, you have data entry that everybody in the field understands, that's rock solid (so no need to unit test anything)

Hey @TeMPOraL, there's a guy with a withdrawn PhD thesis on line 1 who'd like a word with you...

http://blogs.nature.com/naturejobs/2017/02/27/escape-gene-na...

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#464

Post by icedchai »

cameronh90 wrote: post #461
icedchai wrote: post #451

Earlier quoted context omitted.

Seems seems there were at least a couple of problems here. Why were financial professionals using Yahoo as a source of exchange rates?

You'd be surprised at how common that was. I've known two other financial orgs that scraped Yahoo as a critical part of their investment system...

I know individuals / hobbyists who do this, which is fine. Financial orgs should know better, especially if it is actually "critical." How did this get through any sort of review? How do you know Yahoo is authoritative?

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#466

Post by icedchai »

konjin wrote: post #453
icedchai wrote: post #451

Earlier quoted context omitted.

Seems seems there were at least a couple of problems here. Why were financial professionals using Yahoo as a source of exchange rates?

Because excel couldn't authenticate into our internal server and it was good enough.

Okay, so you write a script to take the data from your internal server and post it somewhere that Excel can access. You don't use Yahoo.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#467

Post by golemiprague »

lordnacho wrote: post #247

The problem is that anyone who isn't a decent coder think excel is a sensible choice for this, because they've seen it in their school or work. So when the proposals come in, you're going to see one guy who says it's all common sense and we use familiar old excel for everything, and another lunatic who says something called "pigsqueal" is actually the standard, connected to a "frontend" which for some reason is now s…

You didn't read the article, it seems that they tried to import csv files into a proper system but rather than importing them directly they converted it to an old format of excel which couldn't support the amount of record.

I guess it was some microsoft based system and the developers were used to extract data from excel rather than just importing a csv text file as it would be in a more unixy environment.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#468

Post by cameronh90 »

icedchai wrote: post #464
cameronh90 wrote: post #461

Earlier quoted context omitted.

You'd be surprised at how common that was. I've known two other financial orgs that scraped Yahoo as a critical part of their investment system...

I know individuals / hobbyists who do this, which is fine. Financial orgs should know better, especially if it is actually "critical." How did this get through any sort of review? How do you know Yahoo is authoritative?

That assumes there is a review to get through, rather than it being an Excel file with a few hundred macros...

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#469

Post by thrwyoilarticle »

underdeserver wrote: post #18

The real, visceral tragedy that is COVID-19 notwithstanding, I think it brings a marvelous example of why statistics, the importance of accuracy in data collection, and the effects of exponential growth are a must-have in today's highschool curriculums.

This government had a statistics screw-up just two months ago and the only negative reaction was about how they handled the fallout. As a nation we simply do not understand its value - or the cost of it being misused.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#470

Post by makomk »

codeulike wrote: post #142
bencollier49 wrote: post #133

Good lord. Old XLS files. I'd love to have seen how this was concocted.

Perhaps some old integration code from 10+ years ago that was written to spit out XLS rather than XLSX e.g. some old versions of Crystal Reports can only do XLS. There must be loads old systems around that can only do XLS

It's also entirely possible that whatever they're importing this data into only supports XLS and not XLSX. I don't think Excel is the ultimate endpoint - the data is getting imported into one or more databases somehow.
Post reply on HN