Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Missing Covid-19 test data was caused by the ill-thought-out use of Excel

bbc.com

Post reply on HN
Print view View original

101–110 of 834 posts

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#101

Post by gridlockd »

sails wrote: post #60
benaadams wrote: post #27

Earlier quoted context omitted.

https://twitter.com/MaxCRoser/status/1313046638915706880 > The reason was apparently that the database is managed in Excel and the number of columns had reached the maximum.

A question, was it possibly a csv? ie a csv is equivalent to Excel at the Daily Mail, so they printed Excel as the issue due to it being more familiar to their readership. (Can't think of what the technical issue would be in the case of a csv, also doubt the credibility of the source!)

The story I heard was that it was indeed a CSV and that Excel (silently) drops anything beyond 1 million rows.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#102

Post by rvba »

thu2111 wrote: post #88
chii wrote: post #9

Earlier quoted context omitted.

I'm surprised that MS Access didn't take this kind of niche. I know it's a pretty shitty program to use - excel is much more user friendly. If MS had made excel, but have all the features of access in it...

For a very long time, it did. Weaknesses of Access in no particular order: very poor multi-user/multi-device-access features. No real concept of servers. Buggy, had a nasty habit of corrupting its custom file format causing data loss (Excel never does this). Does expect you to understand SQL, data normalisation, etc. VBA just about serviceable for a spreadsheet but not good enough for a 'real' database-backed app. Pr…

IMHO the biggest problem is the 2GB database size limit

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#103

Post by gridlockd »

Mvandenbergh wrote: post #2

It's unlikely this will have had much impact since most reported cases were only a few days late but come on guys, really? Using an Excel sheet as a database? In week 1, that would be "not great" but could be accepted as being a fast solution that everyone could work with. This far into a pandemic I think we can expect a little more professionalism in data handling.

> Using an Excel sheet as a database?

You wouldn't (want to) believe how incredibly common that is.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#104

Post by dang »

pjc50 wrote: post #7

Professionalism has been entirely absent from the top-level handling of the pandemic. It's not entirely clear from the article who's responsible for this, but I note that the so-called "NHS" test and trace is in fact an outsourced organisation: https://en.wikipedia.org/wiki/NHS_Test_and_Trace run by Tory placewoman Dido Harding. > "She is a former chief executive of the TalkTalk Group where she faced calls for her to…

We detached this subthread from https://news.ycombinator.com/item?id=24685998.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#105

Post by callamdelaney »

beagle3 wrote: post #85

This is indeed user incompetence, but so far every news source quoted the official PR and said it was an "Excel Problem". Microsoft PR was caught unprepared - I wonder how they'll re-spin it in the next few days (and for the first time that I can recall, a Microsoft product was wrongly blamed...) Pay attention, how every time there's a Windows virus or worm, it's a "Computer Virus", but in the (extremely rare) occasi…

It's not a user issue, it's a government incompetence issue. Who was the technical genius who thought 'right, we need to store thousands of lines of critical and important data, I know! Excel!'

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#106

Post by OJFord »

spzb wrote: post #15
mytailorisrich wrote: post #12

Earlier quoted context omitted.

For those not in the UK, Dido Harding was already on front pages a couple of weeks ago when she was grilled by a Parliamentary committee about the shambolic Covid testing organisation and her reply was " no-one could predict such a strong demand for Covid tests "... As the NY Times said yesterday, sometimes it feels like " Britain is operating without adult supervision ".

She was also CEO of ISP TalkTalk when they had a massive personal data breach and told an investigation that she didn't know if the data was encrypted or not. https://www.msn.com/en-gb/money/other/people-want-to-know-wh...

Which was the first of its kind and she was new to the role.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#107

Post by dang »

galeos wrote: post #84

I wonder if this will be added to the European Spreadsheet Risk Interest Group's (EuSpRiG) horror stories list: http://www.eusprig.org/horror-stories.htm

Discussed here: https://news.ycombinator.com/item?id=22431500. (Submitted many times, but that's the only real thread, and only from 7 months ago.)

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#109

Post by karmakaze »

TL;DR [I found it hard to find which actual limit was hit.]

> The problems are believed to have arisen when labs sent in their results using CSV files, which have no limits on size. But PHE then imported the results into Excel, where documents have a limit of just over a million lines.

> The technical issue has now been resolved by splitting the Excel files into batches.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#110

Post by OJFord »

CamperBob2 wrote: post #21
underdeserver wrote: post #18

The real, visceral tragedy that is COVID-19 notwithstanding, I think it brings a marvelous example of why statistics, the importance of accuracy in data collection, and the effects of exponential growth are a must-have in today's highschool curriculums.

This. We need to drop calculus, even geometry if we have to, in order to make room in the curriculum for statistics. Statistical illiteracy kills.

Where did you go to school that you didn't do statistics?
Post reply on HN