Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Missing Covid-19 test data was caused by the ill-thought-out use of Excel

bbc.com

Post reply on HN
Print view View original

61–70 of 834 posts

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#61

Post by learnstats2 »

Mvandenbergh wrote: post #2

It's unlikely this will have had much impact since most reported cases were only a few days late but come on guys, really? Using an Excel sheet as a database? In week 1, that would be "not great" but could be accepted as being a fast solution that everyone could work with. This far into a pandemic I think we can expect a little more professionalism in data handling.

It is absolutely standard that everyone doing office work outside of high-tech jobs uses Excel for data handling.

It's not surprising that this reaches into government.

Even if your canonical database is done properly, there will be a clash here as soon as you share (or give anyone the ability to generate) CSV files that are too large for Excel.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#62

Post by makomk »

Mvandenbergh wrote: post #2

It's unlikely this will have had much impact since most reported cases were only a few days late but come on guys, really? Using an Excel sheet as a database? In week 1, that would be "not great" but could be accepted as being a fast solution that everyone could work with. This far into a pandemic I think we can expect a little more professionalism in data handling.

Hey, at least the UK has automated handling of data! Parts of Germany managed to not tell people they'd tested positive for a week or two because their manual, Excel-based process for handling test results couldn't cope with the increased load from testing people returning to the country - and they're meant to be the competent ones we look up to, according to our press. (I get the impression that German experts have a rather different view.)

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#63

Post by FerretFred »

Mvandenbergh wrote: post #2

It's unlikely this will have had much impact since most reported cases were only a few days late but come on guys, really? Using an Excel sheet as a database? In week 1, that would be "not great" but could be accepted as being a fast solution that everyone could work with. This far into a pandemic I think we can expect a little more professionalism in data handling.

> professionalism in data handling

Nope. They appointed Dido Harding to cough manage it, and Matt Hancock oversees it. That should tell you all you need to know. I wonder if they were running Excel 2007?

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#64

Post by vanilla-almond »

pjc50 wrote: post #7

Professionalism has been entirely absent from the top-level handling of the pandemic. It's not entirely clear from the article who's responsible for this, but I note that the so-called "NHS" test and trace is in fact an outsourced organisation: https://en.wikipedia.org/wiki/NHS_Test_and_Trace run by Tory placewoman Dido Harding. > "She is a former chief executive of the TalkTalk Group where she faced calls for her to…

I have not yet watched this programme from the BBC but it claims the NHS Test and Trace system is a chaotic mess. In other words, as shambolic as everything our Conservative government has done over the pandemic.

Panorama - Test and Track Exposed:

> "Panorama hears from whistleblowers working inside the government’s new coronavirus tracking system. They are so concerned about NHS Test and Trace that they are speaking out to reveal chaos, technical problems, confusion, wasted resources and a system that does not appear to them to be working. The programme also hears from local public health teams who say they have largely been ignored by the government in favour of the private companies hired to run the new centralised tracking system. As Panorama investigates, it has left some local authorities questioning whether local lockdowns could have been handled better or avoided altogether."

https://www.bbc.co.uk/iplayer/episode/m000n1xp/panorama-test...

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#65

Post by jlokier »

underdeserver wrote: post #18

The real, visceral tragedy that is COVID-19 notwithstanding, I think it brings a marvelous example of why statistics, the importance of accuracy in data collection, and the effects of exponential growth are a must-have in today's highschool curriculums.

Imho, exponential growth especially, and then clustering.

Some pretty animations in a lesson might be helpful.

Fire grows in a similar way.

I've seen people up in arms because of things like a workplace of 1000 people shut down because "only" 20 people tested positive one day.

What's the problem?

To people who don't understand exponential growth it seems like 1 in 50 people is a fuss about nothing.

To people who do understand, it's an early-warning signal which says 500 people will get it in weeks due to the confined workspace, if not stopped urgently. And they will infect 1000s in the surrounding community in the same timeframe.

(To anyone tempted to point out the published R isn't that high, local R is highly dependent on situation and who mixes with whom. In a confined workspace, especially with people moving around, it's higher than in the general population. This is why it's useful to teach clustering in addition to exponential growth.)

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#66

Post by Closi »

chii wrote: post #9
londons_explore wrote: post #5

Earlier quoted context omitted.

People use excel sheets as databases all the time because it doesn't require any specialist knowledge to build or use. Sure, we techy types prefer a proper database with a web frontend and an API, but that requires significantly more skill to build than an excel file. No database has the flexibility and flat learning curve that excel has. Until a database manages to meet these goals, it remains a good tool for certai…

I'm surprised that MS Access didn't take this kind of niche. I know it's a pretty shitty program to use - excel is much more user friendly. If MS had made excel, but have all the features of access in it...

It depends which bits of functionality you want, but I think you are close to describing PowerQuery (Get & Transform) which is in Excel, which can ingest multiple data sources into effectively a relational database with hard datatypes and data-cleaning.

In PowerQuery & PowerPivot there are also no practical limits on the number of rows of data you can have (other than the impact on processing time, clearly hundreds of millions of records might start to be a problem). It's not quite access - it doesn't actually persist/store the data, just aggregates it from other sources which is probably what is required here.

Access is a little legacy and would have it's own issues. I think in reality it depends on how you are ingesting data (which is probably hospitals submitting excel templates or similar?). Maybe these could be automatically aggregated and put onto an analytics platform? (e.g. PowerBi). Hospitals could maybe report it on a portal, but I'm not convinced that the leadership would have wanted to change the established process (because of operational focus, change reduction & risk).

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#67

Post by riffraff »

jasiek wrote: post #45
pjc50 wrote: post #24

Earlier quoted context omitted.

The main reason Serco is used is their political connections; they really ought to have been banned from consideration after the fraud scandal: https://www.bbc.co.uk/news/uk-50806919 (my one-step programme for improving UK public contracting would be that after a significant failed contract the contracting company and its directors and any other companies they are directors of would be banned from tendering for a per…

Unfortunately the procurement rules currently in place prevent procurement personnel evaluating a tender from taking into account any prior performance . I'm unsure whether this is a result of EU legislation and common market rules, or whether it is just the UK.

italy has legislation to forbid people who engaged in corruption/embezzlement from working with public administrations.

It also has a required certification for companies that work with the public that they weren't condemned for certain crimes (trafficking in illegal waste, extorsion, organized crime etc).

So, it doesn't seem an EU/Single Market rule that you can't check this.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#68

Post by ImaCake »

CamperBob2 wrote: post #21
underdeserver wrote: post #18

The real, visceral tragedy that is COVID-19 notwithstanding, I think it brings a marvelous example of why statistics, the importance of accuracy in data collection, and the effects of exponential growth are a must-have in today's highschool curriculums.

This. We need to drop calculus, even geometry if we have to, in order to make room in the curriculum for statistics. Statistical illiteracy kills.

I mean, statistics relies heavily on calculus. It might be best to try and fit both in. Something at least a little bit fun and educational could be achieved with some dice throwing or simple computer simulations. High school students place dice games, they will immediately appreciate how useful a bit of probability would be in that context.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#69

Post by hdjdbtbgwjsn »

Mvandenbergh wrote: post #46
chimprich wrote: post #26

Earlier quoted context omitted.

> It's unlikely this will have had much impact since most reported cases were only a few days late I'm not sure if you're being sarcastic or not, but in case you're not, this has meant c.16k people missed from having their contacts traced. That's time-critical. Delays of any number of days are very bad. People will likely die because of this error. This is safety-critical software. It should be stress tested and have…

I was being serious. Distributionally, most of the cases were 1-2 days before the error was spotted. To be clear though, when I said "not much impact", I didn't mean that people might not die because of it, only that it wouldn't affect the large scale functioning (such as it is) of the tracing operation.

Sadly it takes about 24 to get tested and 24 hours for the lab result. That gives very little time to trace contacts before they start entering the asymptomatic-but-infectious-to-others stage. A one day delay is a problem.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#70

Post by makomk »

iso1631 wrote: post #35
social_quotient wrote: post #11

Wish we knew more details. Maybe helpful for people that don’t know... Xls excel has a 64k row limit. Newer xlsx has 1mm row limit. Or maybe they were putting people in by columns and hit the 16.3k column limit?

Or maybe it wasn't excel at all. One number the UK government doesn't release is how many people are tested. They have spent the summer championing "testing capacity", which we found out at the start of September was a lie. They release the number of tests done, and the number of people returning positive. They don't release the number of people done -- if you're tested twice, that counts as two on the number of test…

The trouble is, that entire narrative about the testing capacity being a lie because of secret leaked figures about the number of people tested was itself a massive lie.

For context, the Government publishes a stat for the number of people tested for the first time ever each week (used to be daily, but that was dropped in part due to people misusing it). Certain publications and politicians - starting I think with the Times - pushed the narrative that this was the real number of people being tested, and that the reason it was so far below the stated capacity was because the real capacity was far lower than the Government claims: https://archive.is/n3Yku This was bullshit on multiple levels - there are several important uses of testing, like routine screening of NHS frontline staff and hospital patients being admitted and discharged, that result in people being retested who've been tested at some point previously, and those obviously use actual testing capacity on people who haven't been tested that day or week. (Testing two samples from the same person in short succession, on the other hand, is unusual and not general policy.) Not only that, the people tested number is from pillar 1 and 2 testing in England only for the week ending the 2nd of September, whereas the capacity figure seems to be for all pillars in the entire UK at around the 12th or 13th. It makes absolutely no sense to compare them. And the cherry on the top is that I'm pretty sure this number is from before the problems with test shortages and testing delays that they're implying it explains.

The not-so-secret weekly report in question with the number of people tested for the first time is here: https://www.gov.uk/government/publications/nhs-test-and-trac...

Post reply on HN