Missing Covid-19 test data was caused by the ill-thought-out use of Excel
81–90 of 834 posts
Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel
#82It's unlikely this will have had much impact since most reported cases were only a few days late but come on guys, really? Using an Excel sheet as a database? In week 1, that would be "not great" but could be accepted as being a fast solution that everyone could work with. This far into a pandemic I think we can expect a little more professionalism in data handling.
For me, I always feel like I'm living in some sort of parallel universe. Imagine living in a world where _you_, and a select group of others, know that there are better methods to drive a screw into a wall than punching it with your fist. However, 99% of people in the world punch screws into walls by hand, because they know how to use the only tool they have to hand (i.e. their hand). Their friends and neighbours do too. Their colleagues show them neat little tricks about how to fold your hand better to get the screw in with less pain, or more efficiently. The "proper" option -- drilling a pilot hole, using a self-tapping screw, understanding the difference between nails, bolts and screws, etc, or the concept of wall-plugs, is "too technical", "too complex", or requires "a large amount of infrastructure to fix a simple problem" [buying a drill & screwdriver, etc]. Sometimes people buy new walls with screws pre-installed because the thought of driving in those screws seems like an overwhelming technical burden.
You, and a group of similarly-minded nerds, understand just how stupid this is at times. You personally then spend the rest of your life watching people occasionally have horrific hand injuries from punching screws into walls and refusing the offer of screwdrivers. Others profit from selling vertical hard surfaces with a whole array of convenient screws pre-installed; or get employed teaching others the _very_ basics of screw-driving, or sell products containing all sorts of klunky hacks to cope with the fact that screws manually installed are at odd angles and tend to fall over randomly. The supplier of a popular brand of screwdrivers happens to be the world's largest screw manufacturer, but never seems to promote screwdrivers to people who currently punch them in with their fists. Instead, they sell them a subscription to screws, with the option for a very expensive pre-screwed wall-delivery service, built to your specification (that they call Dynamics for some reason…).
You, and a subset of your nerdy and educated friends recognise just how batshit insane the widespread use of punching-as-a-method-to-install-screws is, and occasionally laugh about it, mostly to distract yourself from the grim reality of buildings badly held together by punched-in-screws. Every so often a skyscraper-sized building collapses, [1] and it's realised in retrospect that the manually-driven-in-screws were not the right tool for the job. Still, people carry on banging in screws with their fists, and you increasingly realise that you have to -- somehow -- accept that in order to sleep at night without going insane.
[1] https://www.businessinsider.com/excel-partly-to-blame-for-tr...
Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel
#83Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel
#84Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel
#85Microsoft PR was caught unprepared - I wonder how they'll re-spin it in the next few days (and for the first time that I can recall, a Microsoft product was wrongly blamed...)
Pay attention, how every time there's a Windows virus or worm, it's a "Computer Virus", but in the (extremely rare) occasions where Linux or MacOS is involved, it's attributed to that system. I don't think that's a coincidence, CMV.
Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel
#86IT ineptitude of the highest order, although I'm not surprised having been involved with government IT previously.
Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel
#87Earlier quoted context omitted.
> People use excel sheets as databases all the time because it doesn't require any specialist knowledge to build or use. Test and Trace is a multi billion pound programme of work, involving huge companies (eg, Serco) interfacing with national governmental bodies (Public Health England). We've had months to get something in place. One of the reasons organisations like Serco is used is their apparent expertise with IT.
The main reason Serco is used is their political connections; they really ought to have been banned from consideration after the fraud scandal: https://www.bbc.co.uk/news/uk-50806919 (my one-step programme for improving UK public contracting would be that after a significant failed contract the contracting company and its directors and any other companies they are directors of would be banned from tendering for a per…
Reasons vary but the ones I've seen are generally:
1. Complex procurement regulations that take a lot of effort to comply with by contractors.
2. Low willingness to pay (both price-wise and time-wise).
3. Civil servants are nightmare customers who don't know what they want and are under no pressure to figure it out, so there's a lot of timewasting involved.
4. PR nightmare if/when things go wrong even if it's not your fault, as government is relatively open compared to other types of customers so easier to find out about problems, and lots of people automatically blame the private sector in preference to blaming government employees due to (misguided) assumptions of moral superiority of the public sector.
Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel
#88Earlier quoted context omitted.
People use excel sheets as databases all the time because it doesn't require any specialist knowledge to build or use. Sure, we techy types prefer a proper database with a web frontend and an API, but that requires significantly more skill to build than an excel file. No database has the flexibility and flat learning curve that excel has. Until a database manages to meet these goals, it remains a good tool for certai…
I'm surprised that MS Access didn't take this kind of niche. I know it's a pretty shitty program to use - excel is much more user friendly. If MS had made excel, but have all the features of access in it...
Weaknesses of Access in no particular order: very poor multi-user/multi-device-access features. No real concept of servers. Buggy, had a nasty habit of corrupting its custom file format causing data loss (Excel never does this). Does expect you to understand SQL, data normalisation, etc. VBA just about serviceable for a spreadsheet but not good enough for a 'real' database-backed app.
Probably someone could try and make a better Access. There are surely lots of startups doing that already, along with hosting. All the "no code" tools that are out there, etc. Access had the advantage of coming with Office so lots of people had it already, whereas today's subscription based services that host your data have incremental cost.
Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel
#89The real, visceral tragedy that is COVID-19 notwithstanding, I think it brings a marvelous example of why statistics, the importance of accuracy in data collection, and the effects of exponential growth are a must-have in today's highschool curriculums.
This. We need to drop calculus, even geometry if we have to, in order to make room in the curriculum for statistics. Statistical illiteracy kills.
The pure maths course was mandatory, but for the applied side you had the choice of either mechanics or statistics. At the time I was much more interested in physics, and was considering it as a degree course but, even if I'd done that, statistics would have been far more useful to me.
I certainly wouldn't choose to drop geometry or calculus, but I've had many occasions to regret my choice of mechanics over statistics during the past two decades.
Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel
#90The real, visceral tragedy that is COVID-19 notwithstanding, I think it brings a marvelous example of why statistics, the importance of accuracy in data collection, and the effects of exponential growth are a must-have in today's highschool curriculums.
Oh god not this again.
The problem the world has right now is not a lack of understanding of what exponential growth means. Plenty of people understand that just fine. It's so easy to understand that there is even a simple ancient parable about it (of the Chinese Emperor and the chess board).
The problem is people who are obsessed with the concept of exponential growth even though "grows exponentially until everyone is infected" is not a real thing that happens with viruses, even though COVID-19 no more shows exponential growth than the sine wave does (sin roughly doubles at points), even though Farrs Law is all about how microbial diseases show S-curve type growth.
This leads to crazyness like the UK's chief medical officers going on TV and presenting a graph in which the last few data points are in decline, but with sudden endless exponential growth projected into the future, along with claims that "this isn't a prediction, but clearly, we have to take extreme action now because of exponential growth".
Observing exponential growth for a few days in a row does NOT mean endless growth until the whole world is infected. Growth rates can themselves change over time, and do. That's the thing people don't seem to understand.